Kolmogorov-Smirnov Test
The Kolmogorov–Smirnov (KS) test is a way to ask a simple question: do these data look like they come from the same distribution, or not? It’s especially handy when you don’t want to assume your data are “nicely” bell-shaped.
What it measures
The KS test compares cumulative distribution functions (CDFs). A CDF tells you, for any value x, what fraction of observations are ≤ x. The test statistic, usually called D, is the maximum vertical distance between two CDFs:
- One-sample KS: compares your sample’s empirical CDF to a reference distribution (like Normal(0,1)).
- Two-sample KS: compares two empirical CDFs (two datasets).
A larger D means the distributions differ more. The test converts D into a p-value to judge whether the difference is bigger than you’d expect from random sampling.
Practical examples
Common uses include:
- Model checking: Do regression residuals look plausibly normal?
- Data drift: Did the distribution of customer ages change between last month’s training data and this month’s live data?
- A/B comparisons: Are delivery times under two shipping methods distributed differently, not just with different averages?
Why it matters in AI/ML
Many ML failures come from silent distribution shifts. The two-sample KS test is a classic, lightweight tool for monitoring covariate drift (feature distributions changing) and sometimes prediction drift. It’s also useful in simulation and generative modeling to compare generated vs. real samples.
Important caveats
The KS test is most sensitive near the center of the distribution and less so in the tails. With very large datasets, tiny, unimportant differences can become “statistically significant.” For discrete data (many ties), the standard KS p-values can be conservative; alternatives or adjustments may be better.
The Kolmogorov-Smirnov Test is a nonparametric hypothesis test that compares cumulative distribution functions to quantify the maximum vertical difference between them. It can test whether a sample matches a reference distribution (one-sample KS) or whether two samples come from the same distribution (two-sample KS), without assuming normality. In AI/ML, it is used to detect dataset shift or validate simulation outputs; e.g., comparing training vs. production feature distributions.
Imagine you have two jars of candy, and you want to know whether they contain the same mix of colors. Instead of assuming anything about how the candies “should” be distributed, you simply compare what you actually see as you count them.
The Kolmogorov-Smirnov Test does something similar for data. It checks whether two sets of numbers look like they come from the same pattern (or whether one set matches a known reference pattern). It focuses on the biggest gap between the two “running totals” of values. In AI/ML, it’s often used to spot when new data has shifted away from training data.