Multiple Linear Regression

Multiple linear regression (MLR) extends simple linear regression to several predictors: the target is modeled as a weighted sum of the inputs plus an intercept. It is the workhorse of applied statistics — interpretable, fast, and the baseline against which fancier regressors are measured.

The model in matrix form

With $p$ predictors and $n$ observations, the model is:

$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1 + \hat{\beta}_2 x_2 + \cdots + \hat{\beta}_p x_p$

Stacking all observations into a matrix $X$ (with a leading column of ones for the intercept) and the targets into a vector $y$, MLR is fit by ordinary least squares: choose $\hat{\beta}$ to minimize the sum of squared errors $\|y - X\hat{\beta}\|^2$. The solution is the normal equations in matrix form:

$\hat{\beta} = (X^T X)^{-1} X^T y$

Each fitted coefficient $\hat{\beta}_j$ is the expected change in $y$ per unit change in $x_j$, holding all other predictors constant — that conditioning is what distinguishes MLR from running $p$ separate simple regressions.

MLR rests on several assumptions: the errors are independent, normally distributed with zero mean, and have constant variance (homoscedasticity). When the assumptions hold, the OLS estimators are unbiased (right on average), efficient (lowest variance among linear unbiased estimators), and consistent (they converge to the truth as $n$ grows).

How good is the model?

$R^2$, the coefficient of determination, is the proportion of variance in the target that the model explains, computed from the sums-of-squares terms:

$R^2 = 1 - \frac{SSE}{SST}, \qquad SST = \sum_{i=1}^{n}(y_i - \bar{y})^2$

If the model is perfect, $SSE = 0$ and $R^2 = 1$; if it is useless, $SSE = SST$ and $R^2 = 0$. Keep in mind that a high $R^2$ says nothing about causation — and that $R^2$ never decreases when you add a predictor, even a useless one. The adjusted $R^2$ corrects for that by penalizing model size:

$\bar{R}^2 = 1 - (1 - R^2)\frac{n - 1}{n - p - 1}$

How significant is the model?

The F-ratio tests the whole model against the null hypothesis that every slope is zero, using the mean-squared terms from the ANOVA decomposition ($SSR = SST - SSE$):

$F = \frac{MSR}{MSE} = \frac{SSR / p}{SSE / (n - p - 1)}$

Unlike $R^2$, the F-ratio accounts for sample size and predictor count, so a model can have a high $R^2$ and still fail this test — the classic small-$n$, many-$p$ trap. For the worked example: $SSR = 3.6$, so $F = (3.6/1)/(2.4/3) = 4.5$ on $(1, 3)$ degrees of freedom.

If the model is significant overall, a t-test on each coefficient, $t = \hat{\beta}_j / SE(\hat{\beta}_j)$, tells you which individual predictors are pulling weight.

Multicollinearity

A high degree of correlation among the predictors makes coefficient estimates unreliable. Warning signs:

• High pairwise correlations between predictors.
• Coefficients whose signs or magnitudes make no physical sense.
• Statistically nonsignificant coefficients on predictors you know are important.
• Coefficients that swing wildly when a predictor is added or removed.

The Variance Inflation Factor (VIF) — from the diagonal of the $(X^T X)^{-1}$ matrix — quantifies how much each coefficient’s variance is inflated by correlation with the other predictors. A VIF above 5 (some texts say 10) signals a problem; drop, combine, or regularize the offending variables.

Model selection

Dropping predictors that do not contribute is almost always wise: it reduces average prediction error, stabilizes the remaining coefficients, and yields a simpler, more interpretable model. The two classic strategies are forward selection (enter the best predictor one at a time until nothing significant remains) and backward elimination (start with everything and remove the worst predictor one at a time). Both are greedy; modern practice often replaces them with regularization (ridge, lasso), which shrinks weak coefficients toward zero continuously.

In practice

sklearn.linear_model.LinearRegression fits MLR directly; statsmodels.OLS gives the full inferential apparatus — t-tests, F-test, VIF via statsmodels.stats.outliers_influence.variance_inflation_factor. When multicollinearity bites, Ridge stabilizes coefficients; when you want selection, Lasso drives weak coefficients to exactly zero. Always compare nested models on adjusted $R^2$ or cross-validated error, never raw $R^2$.

Common pitfalls

• Comparing models by raw $R^2$ — it rewards every added variable; use adjusted $R^2$ or holdout error.
• Ignoring VIF and interpreting unstable coefficients as if they were precise.
• Extrapolating outside the region the data covers, where the linear combination has never been tested.
• Skipping residual diagnostics — non-constant variance and curved patterns invalidate the inference.
• Overfitting small samples: with $n$ barely larger than $p$, the model memorizes rather than learns.

Summary

Multiple linear regression fits $\hat{\beta} = (X^T X)^{-1} X^T y$ by ordinary least squares, giving each coefficient a “holding others constant” interpretation. Judge the model with $R^2$, adjusted $R^2$, and the F-test; judge each predictor with t-tests; and police multicollinearity with VIF. When in doubt, prefer the smaller model.