Overfitting

It’s easy to build a model that looks brilliant on the data you already have. The trap is when that “brilliance” comes from memorizing quirks and noise—so the model falls apart on new, real-world cases.

What overfitting really means

Overfitting happens when a supervised model learns patterns that are too specific to the training set, including random fluctuations, rare coincidences, or leakage-like cues. Technically, the model’s training error becomes very low, but its generalization error (error on unseen data) stays high. This is closely tied to the bias–variance tradeoff: overfit models typically have low bias but high variance, meaning small changes in the training data can lead to big changes in the learned function.

How it shows up in practice

You notice overfitting when validation performance lags behind training performance. Common scenarios:

• Spam detection: the model keys on a few rare words that happened to appear in spam in your dataset, then misses new spam styles.
• Credit scoring: a complex model fits idiosyncrasies of last year’s applicants and misjudges this year’s population shift.
• House price prediction: a high-degree polynomial matches training prices perfectly but predicts absurd values for new neighborhoods.

Why it matters and how it’s controlled

Overfitting leads to brittle decisions, unstable metrics, and costly deployment failures. Pipelines prevent it with:

• Proper train/validation/test splits or cross-validation (e.g., scikit-learn’s cross_val_score).
• Regularization (e.g., L2 in LogisticRegression, pruning/constraints in trees, early stopping in boosting).
• Reducing model complexity, adding more data, and using learning/validation curves to spot the gap.