McNemar's Test
When two classifiers look close in accuracy, the real question is: are they genuinely different, or did you just get lucky with this particular test set? McNemar's Test is a simple way to answer that question when both models were evaluated on the same examples.
What it tests (and what data it uses)
McNemar's Test is a paired statistical test for binary classification that compares two models based on their disagreements. It ignores cases where both models are correct or both are wrong, and focuses only on the off-diagonal counts in a 2x2 table:
- b: Model A correct, Model B wrong
- c: Model A wrong, Model B correct
The null hypothesis is that both models have the same error rate, which implies b and c should be similar. A large imbalance suggests one model is consistently beating the other on the same items.
Intuition and a concrete example
Think of it as counting “head-to-head wins.” In spam detection, suppose Model A correctly flags 40 emails that Model B misses (b=40), while Model B correctly flags only 15 that Model A misses (c=15). Even if their overall accuracies are close, that asymmetry is evidence A is better on this dataset.
Why it matters in supervised learning practice
Model comparisons are usually paired (same test fold, same customers, same patients). Treating results as independent inflates confidence and can lead to picking the wrong model. McNemar's test is a good fit when:
- You have two classifiers evaluated on the same labeled set
- The outcome is binary (or you reduce multiclass to one-vs-rest)
- You care about a statistically grounded “is A better than B?” decision
In Python, you'll commonly see it via statsmodels.stats.contingency_tables.mcnemar.
McNemar’s test is a paired, nonparametric hypothesis test for comparing two classifiers on the same labeled instances using a 2×2 table of their disagreements (cases where one is correct and the other is wrong). It tests whether the classifiers have equal error rates by focusing on discordant pairs. It matters because it provides a statistically grounded way to decide whether an observed accuracy difference reflects a real improvement rather than sampling noise.
Imagine you and a friend take the same true/false quiz, and you want to know who’s actually better—not just who got a slightly higher score by luck. McNemar’s Test is a simple way to do that kind of “fair head-to-head” check.
In AI, it’s often used to compare two classification models (like two spam filters) on the exact same set of emails. It focuses on the cases where the models disagree: emails one model gets right and the other gets wrong, and vice versa. If one model wins those “disagreement” cases by a lot, McNemar’s Test suggests the difference is probably real, not random chance.