Naive Bayes

The Naive Bayes classifier applies Bayes’ theorem with one bold simplification: it assumes every predictor is independent of every other predictor, given the class. That assumption is rarely true, yet the resulting model is fast to train, needs little data, and often outperforms far more sophisticated methods — especially on large datasets.

Where ZeroR uses no predictor and OneR picks the single best one, Naive Bayes uses all predictors at once, combining their evidence through probability.

Bayes’ rule

Bayes’ theorem computes the posterior probability of a class $c$ given the evidence $x$ from three ingredients: the prior $P(c)$, the likelihood $P(x \mid c)$, and the evidence probability $P(x)$:

$P(c \mid x) = \frac{P(x \mid c)\, P(c)}{P(x)}$

• $P(c \mid x)$ — posterior: probability of the class given the predictors
• $P(c)$ — prior: probability of the class before seeing any evidence
• $P(x \mid c)$ — likelihood: probability of observing the predictors within that class
• $P(x)$ — marginal probability of the evidence (a normalizing constant)

With several predictors $x_1, x_2, \dots, x_n$, the naive assumption of class-conditional independence lets the joint likelihood factor into a product:

$P(x_1, \dots, x_n \mid c) = \prod_{i=1}^{n} P(x_i \mid c)$

The classifier then predicts the class with the highest posterior:

$\hat{c} = \arg\max_{c} \; P(c) \prod_{i=1}^{n} P(x_i \mid c)$

Because $P(x)$ is the same for every class, you can compare the unnormalized scores $P(c)\prod_i P(x_i \mid c)$ and normalize at the end.

Worked example: Play Golf

We use the same 14-row weather dataset as the decision tree page. The target is Play Golf (9 Yes, 5 No), so the priors are $P(\text{Yes}) = 9/14$ and $P(\text{No}) = 5/14$.

#	Outlook	Temp	Humidity	Windy	Play Golf
1	Rainy	Hot	High	False	No
2	Rainy	Hot	High	True	No
3	Overcast	Hot	High	False	Yes
4	Sunny	Mild	High	False	Yes
5	Sunny	Cool	Normal	False	Yes
6	Sunny	Cool	Normal	True	No
7	Overcast	Cool	Normal	True	Yes
8	Rainy	Mild	High	False	No
9	Rainy	Cool	Normal	False	Yes
10	Sunny	Mild	Normal	False	Yes
11	Rainy	Mild	Normal	True	Yes
12	Overcast	Mild	High	True	Yes
13	Overcast	Hot	Normal	False	Yes
14	Sunny	Mild	High	True	No

Step 1 — frequency tables

Count each attribute value against the target:

Outlook	Yes	No		Temp	Yes	No
Sunny	3	2		Hot	2	2
Overcast	4	0		Mild	4	2
Rainy	2	3		Cool	3	1

Humidity	Yes	No		Windy	Yes	No
High	3	4		False	6	2
Normal	6	1		True	3	3

Step 2 — likelihood tables

Divide by the class counts (9 Yes, 5 No) to get $P(x_i \mid c)$:

Outlook	P(·|Yes)	P(·|No)		Temp	P(·|Yes)	P(·|No)
Sunny	3/9	2/5		Hot	2/9	2/5
Overcast	4/9	0/5		Mild	4/9	2/5
Rainy	2/9	3/5		Cool	3/9	1/5

Humidity	P(·|Yes)	P(·|No)		Windy	P(·|Yes)	P(·|No)
High	3/9	4/5		False	6/9	2/5
Normal	6/9	1/5		True	3/9	3/5

Step 3 — predict a new day

Should you play when Outlook = Sunny, Temp = Cool, Humidity = High, Windy = True? Multiply the matching likelihoods by each prior:

$P(\text{Yes} \mid x) \propto \tfrac{9}{14} \times \tfrac{3}{9} \times \tfrac{3}{9} \times \tfrac{3}{9} \times \tfrac{3}{9} = 0.0079$

$P(\text{No} \mid x) \propto \tfrac{5}{14} \times \tfrac{2}{5} \times \tfrac{1}{5} \times \tfrac{4}{5} \times \tfrac{3}{5} = 0.0137$

Normalizing both scores ($0.0079 + 0.0137 = 0.0216$):

$P(\text{Yes} \mid x) = \frac{0.0079}{0.0216} = 0.37 \qquad P(\text{No} \mid x) = \frac{0.0137}{0.0216} = 0.63$

No has the higher posterior — the prediction is don’t play.

Interactive calculator: toggle the evidence attributes and watch the posterior for each class update. Static result for the default evidence (Outlook = Sunny, Temp = Cool, Humidity = High, Windy = True):

Class	Unnormalized score	Posterior
Yes	0.0079	0.37
No	0.0137	0.63 ← prediction

The zero-frequency problem

The fix is Laplace smoothing — add 1 to every count before dividing:

$P(\text{Overcast} \mid \text{No}) = \frac{0 + 1}{5 + 3} = \frac{1}{8}$

where 3 is the number of Outlook values. Every probability stays positive, and no class is ever silently eliminated.

Numerical predictors

Categorical frequency tables don’t apply directly to numbers. One option is binning the variable into categories first. The other is to assume a distribution — typically Gaussian — and use its density as the likelihood:

$f(x) = \frac{1}{\sqrt{2\pi}\,\sigma}\, e^{-\frac{(x - \mu)^2}{2\sigma^2}}$

For example, if Humidity were numeric with mean 79.1 and standard deviation 10.2 for Yes days versus 86.2 and 9.7 for No days, the likelihood of Humidity = 80 is $f(80 \mid \text{Yes}) \approx 0.039$ and $f(80 \mid \text{No}) \approx 0.034$.

In practice

sklearn.naive_bayes offers GaussianNB for numeric features, MultinomialNB for counts (the classic spam-filter workhorse), and BernoulliNB for binary features. Naive Bayes trains in a single pass over the data, so it remains a strong baseline for text classification and a useful real-time model when data arrives continuously. Its posterior rankings are usually reliable even when the calibrated probabilities are not — for calibrated outputs, pair it with CalibratedClassifierCV.

Common pitfalls

• Correlated predictors double-count evidence. Two nearly identical features each multiply the score, inflating confidence. Remove redundant features first.
• Forgetting Laplace smoothing. Without it, one unseen attribute value zeroes out a class.
• Trusting the probabilities absolutely. The independence assumption biases posteriors toward 0 and 1; treat them as scores, not calibrated risks.
• Applying Gaussian NB to skewed data. If a numeric feature is far from normal, bin it or transform it instead.

Summary

Naive Bayes combines the prior probability of each class with per-attribute likelihoods through Bayes’ rule, assuming predictors are independent given the class. Training is just counting: build frequency tables, convert them to likelihood tables, and multiply. Add Laplace smoothing to survive unseen values, and use Gaussian densities or binning for numeric predictors.