Regression — Saed Sayad

Regression is the data science task of predicting the value of a numerical target — price, temperature, hours played, blood pressure — from one or more predictors. Where classification assigns a label from a finite set, regression outputs a number from a continuous range.

Every regression method follows the same template: assume a functional form relating the predictors to the target, fit it to historical data by minimizing a measure of error, then use the fitted function to score new cases. The methods in this group differ in the functional form they assume and in the error they minimize.

Simple linear regression

The simplest useful regression model relates a single predictor $x$ to the target $y$ with a straight line:

$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x$

Fitting the model means choosing the intercept $\hat{\beta}_0$ and slope $\hat{\beta}_1$ so the line passes as close as possible to the observed points. Ordinary least squares (OLS) defines “as close as possible” as minimizing the sum of squared errors — the squared vertical distances between each observed $y_i$ and its prediction $\hat{y}_i$ :

$SSE = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \sum_{i=1}^{n} e_i^2$

Setting the derivatives of SSE with respect to $\hat{\beta}_0$ and $\hat{\beta}_1$ to zero yields the normal equations, which have a closed-form solution:

$\hat{\beta}_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}, \qquad \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$

The slope is the covariance of $x$ and $y$ scaled by the variance of $x$ : how much $y$ moves, on average, when $x$ moves by one unit. Squaring the errors makes OLS sensitive to outliers — a point far from the line dominates the fit.

Worked example

Five observations of a predictor $x$ and target $y$ , with $\bar{x} = 3$ and $\bar{y} = 4$ :

$x$	$y$	$x - \bar{x}$	$y - \bar{y}$	$(x-\bar{x})(y-\bar{y})$	$(x-\bar{x})^2$
1	2	−2	−2	4	4
2	4	−1	0	0	1
3	5	0	1	0	0
4	4	1	0	0	1
5	5	2	1	2	4
Sum				6	10

Plugging the sums into the normal equations:

$\hat{\beta}_1 = \frac{6}{10} = 0.6, \qquad \hat{\beta}_0 = 4 - 0.6 \times 3 = 2.2$

The fitted line is $\hat{y} = 2.2 + 0.6x$ . Its residuals and quality:

$x$	$y$	$\hat{y}$	$e$	$e^2$
1	2	2.8	−0.8	0.64
2	4	3.4	0.6	0.36
3	5	4.0	1.0	1.00
4	4	4.6	−0.6	0.36
5	5	5.2	−0.2	0.04

$SSE = 2.4$ , and since the total sum of squares is $SST = \sum (y_i - \bar{y})^2 = 6$ , the model explains $R^2 = 1 - 2.4/6 = 0.6$ of the variance — 60%.

Try it: regression fitter

Drag the line yourself and watch the residuals and SSE respond; then compare your fit against the closed-form optimum:

The regression toolbox

Linear regression is one member of a family. The pages in this group cover the classic alternatives:

Multiple linear regression — OLS with many predictors, in matrix form.
Decision tree regression — splits the data by standard deviation reduction and predicts the leaf average.
k-NN regression — predicts the average target of the k most similar cases.
Support vector regression — fits the flattest line inside an ε-insensitive tube.
Neural networks — learn a non-linear function end to end.

In practice

sklearn.linear_model.LinearRegression is OLS with any number of predictors, and Ridge/Lasso add regularization that shrinks coefficients to fight overfitting. For most tabular problems today, gradient-boosted trees (XGBoost, LightGBM) outperform a single tree or a linear model, but OLS remains the baseline every regressor is judged against and the model you reach for when you need interpretable coefficients. Evaluate regression models with MAE and RMSE rather than raw SSE so error is in target units — see model evaluation for regression.

Common pitfalls

Extrapolating beyond the range of the training data — the line keeps going, but the relationship usually does not.
Ignoring outliers, which OLS amplifies through the squared error; inspect a residual plot before trusting the fit.
Confusing correlation with causation — a high $R^2$ says nothing about why $y$ moves.
Forcing a line through curved data; transform variables or use a non-linear method when residuals show systematic patterns.
Reading $R^2$ alone; a model can explain variance and still predict poorly on new data. Always validate on a holdout set.

Summary

Regression predicts a numerical target from a fitted function. Simple linear regression fits the line $\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x$ by ordinary least squares, with the normal equations giving the slope and intercept in closed form. Residuals and SSE measure fit quality, and the other pages in this group extend the idea to many predictors, trees, neighbors, and margins.