Brier Score

If a model says “there’s a 70% chance this email is spam,” you want that number to mean something in the real world: out of many such emails, about 70% should truly be spam. The Brier Score is a simple way to check how good those probability predictions are.

What it measures

The Brier Score measures the average squared difference between a predicted probability and the actual outcome. For binary classification, the outcome is coded as 1 (event happened) or 0 (didn’t happen). If your model predicts probability p and the true label is y, the per-example error is (p − y)², and the Brier Score is the mean of that across the dataset. Lower is better, with 0 being perfect. Because it’s a squared error on probabilities, it rewards predictions that are both correct and appropriately confident, and it punishes confident wrong predictions heavily.

Why it matters for supervised learning

Accuracy only cares about the final class decision; the Brier Score cares about the quality of probabilities. That’s crucial when probabilities drive decisions:

Credit risk: pricing a loan depends on the predicted default probability, not just “default vs not.”
Medical triage: a 5% vs 40% risk changes what tests you order.
Churn prevention: you target customers above a probability threshold tied to budget.

How it’s used in practice

You’ll see it in calibration workflows: compare an uncalibrated model (e.g., a boosted tree) to a calibrated one (Platt scaling or isotonic regression) and check whether the Brier Score drops. In scikit-learn, it’s available as sklearn.metrics.brier_score_loss:

from sklearn.metrics import brier_score_loss
brier = brier_score_loss(y_true, y_prob)

The Brier Score is a proper scoring rule that measures the mean squared error between predicted probabilities and the actual binary outcomes (0/1), with lower values indicating better probabilistic accuracy. It captures both calibration and discrimination in a single number (e.g., predicting 0.9 for events that rarely occur yields a poor score). It matters because it directly evaluates whether a classifier’s probability outputs are trustworthy for decision-making.

Think of a weather app that says there’s a 70% chance of rain. If it makes lots of “70%” predictions, you’d hope it actually rains about 7 out of 10 times. The Brier Score is a way to grade predictions like that when the model outputs probabilities.

It measures how close a model’s predicted probability is to what really happened (rain or no rain, fraud or not, illness or not). Lower is better: a low Brier Score means the model is not only picking the right outcome more often, but also being honest about its confidence, which matters when decisions depend on risk.