Cohen's Kappa

Two classifiers can “agree” a lot just because one class is very common. Cohen’s Kappa is a way to measure agreement that subtracts out the agreement you’d expect to happen by chance, so the number means something even when classes are imbalanced.

What it measures

Cohen’s Kappa compares two labelings of the same items: in supervised learning, that’s usually your model’s predicted class labels versus the ground-truth labels. It starts with the observed agreement (like accuracy), then corrects it by estimating how much agreement would occur if predictions and truths matched only because of their overall class frequencies.

The standard formula is: κ = (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e is chance agreement from the marginal label distributions.

How to interpret κ

κ = 1: perfect agreement.
κ = 0: no better than chance given the class frequencies.
κ < 0: worse than chance (systematic disagreement).

This makes κ especially informative when accuracy is inflated by a dominant class. For example, in fraud detection where 99% of transactions are legitimate, a model that predicts “not fraud” for everything gets 99% accuracy but κ near 0, correctly signaling “this model learned nothing useful.”

Why it matters in practice

κ helps you choose between classifiers when class imbalance or skewed predictions make accuracy misleading (spam detection, disease screening, churn). In scikit-learn, you’ll see it as sklearn.metrics.cohen_kappa_score:

from sklearn.metrics import cohen_kappa_score
kappa = cohen_kappa_score(y_true, y_pred)

Cohen’s Kappa is a chance-corrected agreement metric for categorical labels, defined as the agreement between predicted and true classes beyond what would be expected from their marginal class frequencies. It ranges from −1 (systematic disagreement) through 0 (chance-level agreement) to 1 (perfect agreement). It matters because it provides a more reliable performance signal than raw accuracy under class imbalance or skewed label distributions.

Imagine two movie critics rating films as “good” or “bad.” If almost every movie is obviously good, they’ll often agree just by luck. Cohen’s Kappa is a way to measure how much two labelers truly agree after subtracting the agreement you’d expect from chance.

In supervised learning, it’s used to judge how well a classifier’s predictions match the real labels, especially when one class is much more common than the other (like “not fraud” vs. “fraud”). A kappa near 1 means strong real agreement, near 0 means no better than guessing.