Linear Regression

Linear regression is a supervised algorithm or statistical method that learns to model a dependent variable (target) as a function of some independent variables (features) by finding a line (or surface) that best “fits” the data. For linear regression, we assume the target to be continuous (a number), for Logistic regression we assume target to be discrete (has a finite number of classes).

Simple Linear Regression

$y={\beta }_0+{\beta }_{1}x+\epsilon$

Multiple Linear Regression

$y=X\beta +\epsilon$

Where:

• $y$ is the $n\times 1$ vector of dependent variable observations
• $X$ is the $n\times (p+1)$ matrix of independent variables (including a column of 1s for the intercept)
• $\beta$ is the $(p+1)\times 1$ vector of coefficients
• $\epsilon$ is the $n\times 1$ vector of error terms

Learning the coefficients

Ordinary Least Squares (OLS) / Analytics solution

OLS is the most common method for estimating the parameters of a linear regression model.
Objective: Minimize the sum of squared residuals: $min\Sigma (y_i-({\beta }_0+{\beta }_1{x}_{1i}+...+{\beta }_p{x}_{pi}){)}^2$
$\hat{\beta }=(X^{\prime }X{)}^{-1}{X}^{\prime }y$

Where:

• $\hat{\beta }$ is the vector of estimated coefficients
• $X^{\prime }$ is the transpose of $X$
• $(X^{\prime }X{)}^{-1}$ is the inverse of $X^{\prime }X$

Properties of OLS estimators under the classical assumptions:

1. Best Linear Unbiased Estimator (BLUE)
2. Consistent
3. Asymptotically normal

Estimation of σ²

${\hat{\sigma }}^2=\frac{ESS}{n}$
Coefficient validity
In statistics, the coefficients are usually paired with their p-values. These p-values come from null hypothesis statistical tests: t-tests are used to measure whether a given coefficient is significantly different than zero (the null hypothesis that a particular coefficient $\beta i$​ equals zero), while F-tests are used to measure whether any of the terms in a regression model are significantly different from zero.

Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation is an alternative method to OLS for estimating the parameters of a linear regression model. It’s particularly useful when dealing with non-normal error distributions. MLE finds the parameter values that maximize the likelihood of observing the given data.

Likelihood Function for Linear Regression
Assuming normally distributed errors:
$L(\beta ,{\sigma }^2|y,X)={\prod }_i(\frac{1}{\sqrt{2\pi {\sigma }^2}})\ast exp(\frac{-(y_i-x_i^{\prime }\beta {)}^2}{(2{\sigma }^2)})$

Log-likelihood:
$lnL(\beta ,{\sigma }^2|y,X)=-n/2\ast ln(2\pi )-n/2\ast ln({\sigma }^2)-1/(2{\sigma }^2)\ast {\Sigma }_i(y_i-x_i^{\prime }\beta {)}^2$

$lnL(\beta ,{\sigma }^2|y,X)=-\frac{n}{2}\ast ln(2\pi )-\frac{n}{2}\ast ln({\sigma }^2)-\frac{1}{(2{\sigma }^2)}\ast {\Sigma }_i(y_i-x_i^{\prime }\beta {)}^2$

Comparison with OLS

• Under normality assumption, MLE and OLS produce the same estimates for β
• MLE can be extended to non-normal error distributions
• MLE provides a framework for hypothesis testing and model selection (e.g., likelihood ratio tests)

Gradient Descent for Linear Regression

When dealing with large datasets, the analytical solution may be computationally expensive. Gradient descent is an iterative optimization algorithm used to find the minimum of the cost function.

Assumptions

• Linearity: The relationship between $X$ and $Y$ is linear.
• Homoscedasticity: Constant variance of residuals $(Var(\epsilon |X)={\sigma }^2)$. This means that error distribution is consistent or all values of the features, there should be no discernible patterns.
• No Multicollinearity: Independent variables shouldn’t be highly correlated with each other. This can be checked using correlation matrices or Variance Inflation Factor (VIF).
• Normality: Residuals are normally distributed $(\epsilon N(0,{\sigma }^2))$. This can be checked through Q-Q plots of the residuals or by histograms, or through statistical tests such as the Kolmogorov-Smirnov test. Relevant for MLE only
• Independence: Observations are independent of each other
• No Exogeneity: $E(\epsilon |X)=0$, meaning the errors are uncorrelated with the predictors

Consequences of violating assumptions:

• Violating linearity: Biased and inconsistent estimates
• Violating independence: Incorrect standard errors, inefficient estimates
• Violating homoscedasticity: Inefficient estimates, incorrect standard errors
• Violating normality: Hypothesis tests may be invalid for small samples
• Perfect multicollinearity: Unable to estimate unique coefficients

Model Evaluation Metrics

1. R-squared (Coefficient of Determination): $R^2=1-\frac{SSR}{SST}$ Where SSR is the sum of squared residuals and SST is the total sum of squares
2. Adjusted R-squared: $Adj{R}^2=1-[(1-R^2)(n-1)/(n-p-1)]$
3. Mean Squared Error (MSE): $MSE=\Sigma \frac{(y_i-{\hat{y}}_i{)}^2}{n}$
4. Root Mean Squared Error (RMSE): $RMSE=\sqrt{MSE}$
5. Mean Absolute Error (MAE): $MAE=\Sigma \frac{|y_i-{\hat{y}}_i|}{n}$

Hypothesis Testing

1. t-test for individual coefficients:
   $H_0:{\beta }_i=0$
   $t=\frac{{\hat{\beta }}_i}{SE({\hat{\beta }}_i)}$
2. F-test for overall model significance:
   $F=(SSR/p)/(SSE/(n-p-1))$
   $F=\frac{\frac{SSR}{p}}{\frac{SSE}{(n-p-1)}}$

Confidence Intervals

CI for ${\beta }_i$: ${\hat{\beta }}_i\pm t(\alpha /2,n-p-1)\ast SE({\hat{\beta }}_i)$

Extensions and Regularization

1. Ridge Regression (L2): $\hat{\beta }ridge=argmin(||y-X\beta |{|}^2+\lambda ||\beta |{|}^2)$
2. Lasso Regression (L1): $\hat{\beta }lasso=argmin(||y-X\beta |{|}^2+\lambda ||\beta |{|}_1)$
3. Elastic Net: Combination of L1 and L2 penalties

Modeling complex relationships

Polynomial Regression

Extends linear regression to model non-linear relationships:
$y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+...+{\beta }_n{x}^n+\epsilon$

Interaction Terms

Allows for modeling the combined effect of two or more variables:
$y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_3(x_1\ast x_2)+\epsilon$

Code example

import numpy as np
 
class LinearRegression:
    def __init__(self, learning_rate: float=0.01, n_iterations: int=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.weights = None
        self.bias = None
 
    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0
 
        for _ in range(self.n_iterations):
            prediction = np.dot(X, self.weights) + self.bias
 
            dw = (1 / n_samples) * np.dot(X.T, (prediction - y))
            db = (1 / n_samples) * np.sum(prediction - y)
 
            self.weights -= self.learning_rate * dw
            self.bias -= self.learning_rate * db
 
    def predict(self, X):
        return np.dot(X, self.weights) + self.bias
 
    def mse(self, X, y):
        y_predicted = self.predict(X)
        return np.mean((y - y_predicted) ** 2)

Links

• Explained.ai: Linear Regression
• MLcourse.ai lesson