Classification

Classification is the data science task of predicting the value of a categorical variable — the target or class — by building a model from one or more numerical and/or categorical predictors. Will the customer churn or stay? Is the email spam? Play golf, or not? Whenever the answer is a label rather than a number, you are doing classification. (Predicting a number instead is regression.)

The pages in this group walk through nine classification algorithms, from a one-line baseline to margin-maximizing machines. They fall into four families, distinguished by the mathematical structure each one exploits.

Frequency table methods

These algorithms count co-occurrences of predictor values and classes, then reason from the counts. They are fast, transparent, and handle categorical data natively — numerical inputs are binned first.

• ZeroR — ignores all predictors and always predicts the majority class. No predictive power, but the essential baseline.
• OneR — builds one majority-class rule per value of the single best predictor. Remarkably competitive for three lines of logic.
• Naive Bayes — applies Bayes’ theorem with a conditional-independence assumption to turn frequency tables into class probabilities.
• Decision Tree — recursively splits on the most informative attribute, producing a human-readable tree of rules.

Covariance matrix methods

These methods model how the predictors vary together and draw decision boundaries in that geometric structure. They expect numerical inputs — categorical ones must be encoded.

• Linear Discriminant Analysis — models each class as a Gaussian sharing a common covariance; the optimal boundary is linear.
• Logistic Regression — fits a linear model through the logistic function to output calibrated class probabilities.

Similarity function methods

• k-Nearest Neighbors — stores the training data and classifies each new case by the majority vote of its $k$ closest rows. No training phase at all; the distance function is the model.

Other methods

• Neural Networks — layers of weighted, non-linear units that learn arbitrarily complex boundaries from enough data.
• Support Vector Machine — finds the boundary that maximizes the margin to the nearest points of each class, with kernels for non-linear problems.

When to use what

• Always start with ZeroR and OneR. They cost nothing and tell you how hard the problem actually is.
• Need interpretability? Decision trees and logistic regression explain every prediction.
• Need probabilities? Naive Bayes, logistic regression, and neural networks score as well as classify.
• Small, clean, mostly categorical data? The frequency-table methods shine.
• Numerical data with roughly Gaussian classes? LDA is hard to beat.
• Complex, non-linear boundaries and plenty of data? SVMs with kernels, or neural networks.
• No time to tune? k-NN has almost no knobs — but it slows down and degrades as dimensionality grows.

In practice

Modern tabular classification is often dominated by gradient-boosted trees — XGBoost and LightGBM — which are descendants of the humble decision tree, ensemble-averaged for accuracy. In scikit-learn every algorithm here is one class away: DummyClassifier, GaussianNB, DecisionTreeClassifier, LinearDiscriminantAnalysis, LogisticRegression, KNeighborsClassifier, MLPClassifier, and SVC share one fit/predict interface, so comparing them is cheap. Spend the effort on evaluation instead: pick metrics before modeling, and read them against the ZeroR baseline on held-out data — see Model Evaluation.

Common pitfalls

• Choosing an algorithm before establishing the baseline. A sophisticated model below the majority-class rate is worse than nothing.
• Trusting accuracy on imbalanced classes. A 95%-accurate model can be useless if the class split is 95/5.
• Skipping preparation. Frequency-table methods need binned numbers; covariance and similarity methods need encoded categories and scaled features.
• Evaluating on training data. Every accuracy quoted here is optimistic until measured on data the model has not seen.

Summary

Classification predicts a categorical target from predictors. The nine algorithms in this group exploit four structures — frequency tables, covariance, similarity, and learned non-linearities — and the right choice depends on your data shape, interpretability needs, and whether you need probabilities. Start at the baseline, then climb only as far as the data rewards.