Notes
🔍

Confidence Interval

A single number estimate can feel reassuring, but it hides an important truth: if you collected a different sample, you’d likely get a different estimate. A confidence interval is a practical way to show that uncertainty as a range of plausible values.

What it means (without the common confusion)
A confidence interval (CI) is an interval computed from sample data that is designed to capture an unknown population quantity (like a mean, proportion, or regression coefficient) with a chosen long-run success rate, such as 95%. The key idea is about the method, not the specific interval you got: if you repeated the same sampling process many times and built a 95% CI each time, about 95% of those intervals would contain the true value.

How it’s built
Most CIs follow the pattern:

  • estimate ± (critical value × standard error)

The standard error reflects sampling variability; the critical value comes from a reference distribution (often Normal or t). Wider intervals mean more uncertainty; larger samples usually shrink the interval.

Concrete examples

  • House prices: “Average sale price is $420k, 95% CI [$400k, $440k]” communicates precision better than $420k alone.
  • Medical measurements: A CI for the difference in blood pressure between treatments shows both effect size and uncertainty.
  • A/B tests: A CI for conversion-rate lift helps decide if an observed improvement is likely real or just noise.

Why it matters in AI/ML
In ML, CIs help quantify uncertainty around model parameters (e.g., linear/logistic regression coefficients), performance metrics (often via bootstrapping), and treatment effects in experimentation. Without intervals, teams may overreact to tiny, unstable changes in accuracy, loss, or business KPIs.

A Confidence Interval is a range of plausible values for an unknown population parameter (e.g., a mean or regression coefficient) computed from sample data, paired with a confidence level (e.g., 95%). Over repeated sampling, that procedure would produce intervals containing the true parameter about that fraction of the time. In AI/ML, it quantifies uncertainty in estimated effects and model metrics; e.g., a 95% CI for accuracy from bootstrap resampling.

Imagine you taste one spoonful of soup to guess how salty the whole pot is. You might say, “It’s probably this salty, give or take a bit.” A confidence interval is that “give or take” range for a number you estimate from data.

In statistics and machine learning, we often measure something from a sample—like an average, an accuracy score, or the effect of a change—and we know it won’t be perfectly exact. A confidence interval gives a lower and upper bound that likely contains the true value. For example, a 95% confidence interval means that if you repeated the same process many times, about 95% of those intervals would capture the true value.