p-value
A p-value is a way to quantify how surprising your data would be if “nothing interesting is going on.” It’s often used as a reality check when you’re deciding whether an observed effect might just be random noise.
What it means (without the jargon)
In hypothesis testing, you start with a null hypothesis (usually “no difference” or “no effect”). The p-value is the probability of getting results at least as extreme as what you observed, assuming the null hypothesis is true. Smaller p-values mean your data would be rarer under the null, so the null looks less believable.
What it is not
A common trap: the p-value is not the probability that the null hypothesis is true. It also doesn’t tell you how big or important an effect is—only how incompatible the data are with the null model.
A concrete example
Suppose you test whether a new study app improves exam scores. The null says “average improvement is 0.” If you compute a p-value of 0.03, that means: if the app truly had no effect, there’s a 3% chance you’d see an improvement as large (or larger) than the one in your sample just by random variation.
Why it matters in AI/ML
In machine learning, p-values show up when you want statistical evidence, not just predictive accuracy:
- A/B tests for model changes (does the new ranking model increase click-through rate?)
- Feature significance in models like linear/logistic regression (common in statsmodels)
- Model comparison in classical settings (e.g., likelihood ratio tests)
Misusing p-values can lead to “discovering” improvements that are just luck—especially when testing many features or many variants—so they’re often paired with multiple-testing corrections and effect sizes.
A p-value is the probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one obtained. Smaller p-values indicate stronger evidence against the null, but they do not measure effect size or the probability the hypothesis is true. In AI/ML, p-values support feature significance tests and A/B evaluations; e.g., testing whether a new model’s conversion rate differs from a baseline.
Imagine you flip a coin 10 times and get 9 heads. You might wonder: is the coin weird, or did you just get a surprising streak by chance? A p-value is a number that answers that kind of question.
In statistics (and in AI/ML experiments), the p-value means: “If nothing special is going on” (the usual assumption, called the null hypothesis), how likely is it to see results at least as extreme as the ones you got? A small p-value suggests your result would be rare under ‘just chance,’ so it’s evidence something real may be happening.