Model Evaluation — Saed Sayad

Model evaluation is an integral part of the model development process. It answers two questions: which of your candidate models represents the data best, and how well will the chosen model work on data it has never seen? Everything on this page serves the second question, because the first one alone is a trap.

Evaluating a model on the same data used to train it is not acceptable in data science: a flexible enough model can memorize its training set, producing overoptimistic scores that collapse in production. Both standard evaluation methods — hold-out and cross-validation — exist to measure performance on data the model did not see during training.

Hold-out validation

When the dataset is large, the simplest honest evaluation randomly splits it into three subsets:

Training set — used to fit the model’s parameters.
Validation set — used to assess performance during development: tuning hyperparameters, selecting features, and choosing between candidate algorithms. Not every algorithm needs one.
Test set — unseen examples held back until the very end, used once to estimate the model’s likely future performance.

The discipline matters more than the exact proportions (a common split is 60/20/20). Every time you look at test performance and change something, the test set leaks into your decisions and stops being a fair estimate — that is what the validation set is for.

Cross-validation

When only a limited amount of data is available, holding out a third of it is expensive, and a single random split gives a noisy estimate. $k$ -fold cross-validation uses all the data for both training and testing without ever testing on training data:

Divide the data into $k$ subsets (folds) of equal size.
Build $k$ models; each time leave out one fold from training and use it as the test set.
Average the $k$ performance scores:

\text{CV estimate} = \frac{1}{k} \sum_{i=1}^{k} \text{score}_i

Ten folds ( $k = 10$ ) is the common default. If $k$ equals the sample size, each test set is a single observation — this is leave-one-out cross-validation, which is nearly unbiased but has high variance and costs $n$ model fits. The averaged CV score is a far more stable estimate of future performance than any single split, at the price of training the model $k$ times.

Measuring the score

What goes into “score” depends on the task:

Classification models are judged with the confusion matrix and its derived metrics — accuracy, precision, recall, specificity, F1 — plus threshold-independent measures like ROC and AUC. See Evaluation: Classification.
Regression models are judged by error magnitude — MAE, MSE, RMSE — and by the proportion of variance explained, $R^2$ . See Evaluation: Regression.

Both pages assume you have already produced honest train/test separation using the methods on this page.

In practice

In scikit-learn, sklearn.model_selection.train_test_split performs the hold-out split (use stratify= on the target for classification so class proportions survive), and sklearn.model_selection.cross_val_score runs $k$ -fold CV in one call. GridSearchCV and RandomizedSearchCV tune hyperparameters inside cross-validation so the validation folds never contaminate the final estimate. For time-ordered data, TimeSeriesSplit replaces random folds with forward-chaining splits — random shuffling of time series leaks the future into the past. Gradient-boosting libraries like XGBoost and LightGBM expose an early_stopping callback that uses a validation set to stop training before overfitting sets in.

Common pitfalls

Tuning on the test set. Repeatedly checking test performance while tweaking the model silently turns it into a validation set; your reported score becomes optimistic.
Data leakage. Fitting preprocessors (scalers, encoders, feature selection) on the full dataset before splitting lets test information flow into training. Fit them on the training folds only.
Random splits for grouped or temporal data. If the same patient, customer, or day appears in both train and test, the estimate is inflated.
Trusting a single hold-out split on small data. One unlucky split can swing the score dramatically; use cross-validation instead.
Comparing models on different splits. Every candidate must be evaluated on identical folds or the comparison is meaningless.

Summary

Never evaluate on training data. Hold-out validation separates data into training, validation, and test roles; $k$ -fold cross-validation gets the same honesty out of small datasets by rotating the test fold and averaging. Once the evaluation protocol is sound, pick task-appropriate metrics — classification or regression — and only then compare models.