Confidence Interval for Performance Metrics
A single number like “accuracy = 0.91” feels reassuring, but it hides an important fact: that score came from a finite sample of data, so it has uncertainty. A confidence interval puts honest bounds around a performance metric, showing a plausible range for the model’s true performance.
What it is (and what it isn’t)
A confidence interval for a performance metric is an interval estimate for quantities like accuracy, F1, AUROC, RMSE, or log loss. A 95% interval is constructed by a procedure that, if repeated across many new samples drawn the same way, would contain the true metric about 95% of the time. It does not mean “there’s a 95% probability the true value lies in this specific interval” (that’s a Bayesian statement), but it does communicate how noisy your estimate is.
How people compute it in practice
- Bootstrap: repeatedly resample the test set with replacement, recompute the metric, and take percentiles (common for AUROC/F1 where formulas are messy).
- Binomial/Wilson intervals for accuracy (treat correct/incorrect as Bernoulli outcomes; better than the naive normal approximation).
- Cross-validation-based intervals: summarize fold-to-fold variation, but be careful—fold scores are correlated, so “mean ± 1.96·sd/√k” can be misleading.
Why it matters for supervised learning decisions
Confidence intervals change how you interpret improvements. In spam detection, an accuracy jump from 0.910 to 0.915 might be meaningless if both models’ 95% intervals overlap heavily. In credit scoring or disease diagnosis, intervals help quantify risk: a model with slightly lower mean AUROC but a tighter interval can be the safer choice. They also guide data collection—wide intervals usually mean you need a larger, more representative test set.
A confidence interval for performance metrics is a statistically derived range (at a stated confidence level) that quantifies uncertainty around an estimated evaluation metric such as accuracy, AUC, or RMSE on finite data. It matters because point estimates alone can be misleading; intervals support defensible model comparison, detect when apparent gains fall within noise, and communicate reliability to stakeholders (e.g., AUC 0.84, 95% CI [0.81, 0.87]).
Think of tasting a spoonful of soup to judge the whole pot. You can say, “It’s salty,” but you also know you might have hit an extra-salty spot. A confidence interval is the “wiggle room” around your best guess.
In AI, we measure a model with performance metrics like accuracy, precision, or AUC (scores that summarize how well it predicts). A confidence interval for performance metrics gives a range where the model’s true performance likely falls, not just a single number. This matters because results can change depending on which data you tested on, and the interval shows how reliable that score really is.