k-NN Regression

K nearest neighbors is a lazy, non-parametric algorithm: it stores the entire training set and answers a query by finding the most similar stored cases. In k-NN classification the neighbors vote on a class label; in k-NN regression their numerical targets are averaged. The machinery — distance functions, the choice of $k$ — is identical; only the aggregation changes.

Algorithm

Given a query point, rank the training cases by distance and take the $k$ closest. The simplest prediction is the plain average of their targets:

$\hat{y} = \frac{1}{k} \sum_{i=1}^{k} y_i$

A refinement weights each neighbor by inverse distance, so closer cases speak louder:

$\hat{y} = \frac{\sum_{i=1}^{k} w_i\, y_i}{\sum_{i=1}^{k} w_i}, \qquad w_i = \frac{1}{d(x, x_i)}$

k-NN regression uses the same distance functions as the classifier. For continuous variables, the Euclidean and Manhattan distances are the $p = 2$ and $p = 1$ cases of the Minkowski family:

$D_{Euclid} = \sqrt{\sum_{i=1}^{p}(x_i - y_i)^2}, \qquad D_{Manhattan} = \sum_{i=1}^{p} |x_i - y_i|, \qquad D_{Minkowski} = \left( \sum_{i=1}^{p} |x_i - y_i|^q \right)^{1/q}$

For categorical variables, use the Hamming distance — the number of positions at which two equal-length attribute vectors differ:

$D_H = \sum_{i=1}^{p} \mathbf{1}[x_i \neq y_i]$

Choosing $k$ is the classic bias–variance dial: small $k$ tracks local noise, large $k$ smooths away real structure. Cross-validation on held-out data is the reliable way to pick it; in practice, $k$ of 10 or more beats 1-NN on most datasets.

Worked example: house price index

The training set relates a house’s Age and Loan amount to its House Price Index (HPI). We predict HPI for a new case with Age = 48 and Loan = $142,000, using Euclidean distance:

Age	Loan ($)	HPI	Distance to query
25	40,000	135	102,000
35	60,000	256	82,000
45	80,000	231	62,000
20	20,000	267	122,000
35	120,000	139	22,000
52	18,000	150	124,000
23	95,000	127	47,000
40	62,000	216	80,000
60	100,000	139	42,000
48	220,000	250	78,000
33	150,000	264	8,000

With $k = 1$, the nearest case is the last row:

$D = \sqrt{(48 - 33)^2 + (142000 - 150000)^2} = 8000.01 \;\Rightarrow\; \widehat{HPI} = 264$

With $k = 3$, the neighbors are the rows at distances 8,000 (HPI 264), 22,000 (HPI 139), and 42,000 (HPI 139), and the prediction is their average:

$\widehat{HPI} = \frac{264 + 139 + 139}{3} = 180.7$

Standardized distance

Notice what dominated that computation: Loan, measured in tens of thousands of dollars, swamped Age, measured in tens of years. The distance was essentially Loan distance alone. When variables live on different scales — or mix numerical and categorical types — standardize first. With min–max scaling, $z = \frac{v - \min}{\max - \min}$, Age spans 20–60 and Loan 18,000–220,000, so the query becomes $(0.70, 0.61)$:

Age (std)	Loan (std)	HPI	Distance
0.13	0.11	135	0.765
0.38	0.21	256	0.494
0.63	0.31	231	0.316
0.00	0.01	267	0.924
0.38	0.50	139	0.336
0.80	0.00	150	0.616
0.08	0.38	127	0.672
0.50	0.22	216	0.426
1.00	0.41	139	0.364
0.70	1.00	250	0.386
0.33	0.66	264	0.371

Now the nearest neighbor is the third row (HPI 231), not the last — a completely different answer from the same data. As with the classifier, that sensitivity is not a good sign of robustness; it means scale choices are modeling decisions.

In practice

sklearn.neighbors.KNeighborsRegressor handles both uniform and inverse-distance weighting (weights='distance'), with StandardScaler in a Pipeline to make standardization automatic. k-NN regression is a solid non-parametric baseline and a natural fit for recommendation-style similarity problems, but prediction cost grows with the training set, so large systems use approximate indexes (ball trees, KD-trees, HNSW). The curse of dimensionality bites hard beyond a dozen or so features — distances concentrate and “nearest” stops meaning much.

Common pitfalls

• Forgetting to scale features, letting the largest-unit variable silently dominate every distance.
• Choosing $k$ by gut — too small and you fit noise; too large and you fit the global mean. Cross-validate it.
• Using plain Euclidean distance on high-dimensional sparse data, where all points become nearly equidistant.
• Ignoring prediction cost: k-NN does zero work at training time and all of it at query time.
• Mixing categorical features into Euclidean distance instead of using Hamming distance or encoding properly.

Summary

k-NN regression predicts the (optionally distance-weighted) average target of the $k$ most similar training cases, using the same distance functions as k-NN classification. On the house-price data, $k = 1$ predicts HPI 264 while $k = 3$ smooths to 180.7 — and standardizing the features changes the nearest neighbor entirely, a reminder that in k-NN, geometry is the model.