Encoding

Encoding — sometimes called continuization — is the transformation of categorical variables into binary or numerical counterparts: treating male and female as 1 and 0, for instance. Many modeling methods, including linear regression, support vector machines, and neural networks, cannot consume categories directly and require every input to be numeric.

The right encoding depends on the variable. Ordinal categories with a genuine order can be mapped to integers; nominal categories cannot, without inventing a false order. The two main approaches are binary (one-hot) encoding and target-based encoding, with ordinal encoding as the natural choice for ordered data.

Ordinal (label) encoding

When categories have an intrinsic order — low, medium, high — map each category to an integer that preserves it: low → 0, medium → 1, high → 2. One column in, one column out, and the model can exploit the ordering. Applying this to nominal data is a mistake: an arbitrary numbering like Blue = 0, Green = 1, Red = 2 implies Green is “between” Blue and Red, which means nothing.

Binary (one-hot) encoding

One-hot encoding numerizes a categorical variable with 0/1 indicators for the absence or presence of each category. If the variable has $k$ categories, create $k$ binary variables — technically $k - 1$ suffice, since one category is implied when all indicators are 0. Here is the legacy example: the categorical variable Trend, with three values, becomes three binary variables.

Trend	Trend_Up	Trend_Down	Trend_Flat
Up	1	0	0
Up	1	0	0
Down	0	1	0
Flat	0	0	1
Down	0	1	0
Up	1	0	0
Down	0	1	0
Flat	0	0	1
Flat	0	0	1
Flat	0	0	1

The main drawback appears when the variable has many categories (a city column, for example): one-hot encoding can tremendously increase the dimension of the data.

Target-based encoding

Target-based encoding replaces the categorical variable with a single numerical one: each category is replaced by its corresponding probability of the target (if the target is categorical) or the average of the target (if it is numerical). For category $c$,

$$\text{encode}(c) = \frac{1}{n_c} \sum_{i:\, x_i = c} y_i$$

where $n_c$ is the number of rows with category $c$. Using the same Trend data with a binary target:

Trend	Target = 0	Target = 1	$P(\text{Target} = 1 \mid \text{Trend})$
Up	1	2	2/3 ≈ 0.67
Down	2	1	1/3 ≈ 0.33
Flat	2	2	2/4 = 0.50

So every Up row encodes to 0.67, Down to 0.33, and Flat to 0.50. With a numerical target the same idea uses means instead — the legacy page’s Trend values (Up: 21, 24, 26; Down: 8, 11, 12; Flat: 15, 16, 14, 13) encode to 23.7, 10.3, and 14.5 respectively. The drawbacks: the encoding depends on the target’s distribution, it carries less predictive power than full one-hot encoding, and — computed naively on the training set — it leaks target information into the predictors.

Try it: encoding lab

Compare one-hot and ordinal encoding on a small table with the interactive widget:

Static summary: five rows with a nominal Color column (Red, Green, Blue) and a numeric Size — (Red, 12), (Green, 7), (Blue, 15), (Red, 9), (Green, 11). One-hot encoding keeps Size and adds three indicator columns:

Size	Color_Red	Color_Green	Color_Blue
12	1	0	0
7	0	1	0
15	0	0	1
9	1	0	0
11	0	1	0

Ordinal encoding instead replaces Color with a single integer column (alphabetical: Blue = 0, Green = 1, Red = 2): 2, 1, 0, 2, 1. Compact, but it invents an ordering that nominal categories do not have.

In practice

scikit-learn covers the basics: sklearn.preprocessing.OneHotEncoder (with drop="first" for the $k-1$ variant) and OrdinalEncoder for ordered categories, while pandas.get_dummies is the quick in-notebook option. For high-cardinality columns, the category_encoders library adds the serious tools — TargetEncoder with built-in smoothing, HashingEncoder, which hashes categories into a fixed number of columns to bound dimensionality, and leakage-resistant variants like LeaveOneOutEncoder. Whatever you choose, fit encoders inside a Pipeline on training data only — target-based encoding fit on the full dataset is a classic leakage bug.

Common pitfalls

• Ordinal-encoding nominal data. Arbitrary integers invent order and distances that distance-based models will treat as real.
• The dummy variable trap. Keeping all $k$ one-hot columns makes linear-model coefficients non-identifiable; drop one column.
• High-cardinality explosion. One-hot on a column with thousands of categories creates thousands of sparse features; consider target or hashing encoding instead.
• Target leakage. Target-based encodings must be computed with cross-fitting or leave-one-out schemes, never on the whole training set.
• Train/serve skew. A category unseen in training needs an explicit handling strategy (handle_unknown), not a crash.

Summary

Encoding turns categories into numbers: ordinal encoding for genuinely ordered categories, one-hot for nominal ones at the cost of $k$ (or $k-1$) new columns, and target-based encoding as a compact alternative that must be guarded against leakage. Choose by cardinality, ordering, and model type — and always fit the encoding on training data alone.