Notes
🔍

Overfitting

Overfitting is what happens when a model becomes a little too good at “studying” its training data. It learns not just the real patterns, but also the quirks and noise—so when it sees new data, it performs worse than you’d expect.

What overfitting really means

A model overfits when it matches the training set extremely well but has poor generalization to unseen examples. Technically, it’s often linked to a model having too much capacity (too many parameters, too much flexibility) relative to the amount of data or the true complexity of the relationship. You’ll typically see:

  • Very low training error (or high training accuracy)
  • Noticeably higher validation/test error

This is closely related to the bias–variance tradeoff: overfitting usually means low bias but high variance (predictions swing a lot depending on the sample).

Intuition and a concrete example

Imagine predicting house prices. A sensible model learns that size, location, and condition matter. An overfit model also “learns” that houses listed on Tuesdays in your dataset were slightly cheaper—because of random chance—then treats that as a real rule. It’s like memorizing practice questions instead of learning the underlying topic: great on the practice sheet, shaky on the exam.

Why it matters in AI/ML

Overfitting is a core failure mode in machine learning because the goal is reliable performance on future data. It can lead to brittle models, misleadingly optimistic metrics, and poor real-world decisions (credit scoring, medical risk prediction, demand forecasting).

Common ways to prevent it

  • Use train/validation/test splits and cross-validation
  • Add regularization (L1/L2, weight decay)
  • Simplify the model (fewer features, shallower trees)
  • Early stopping and dropout (neural networks)
  • More data or data augmentation

Overfitting is when a model learns patterns specific to the training data—including noise—so it achieves low training error but performs poorly on new, unseen data. It matters in AI/ML because it undermines generalization and leads to unreliable predictions. For example, a high-degree polynomial regression may fit training points extremely well yet produce large errors on a validation set; regularization or cross-validation helps detect and reduce it.

Imagine you’re studying for a test by memorizing the exact answers from last year’s exam. You might score perfectly on those old questions, but struggle when the new test asks things a little differently. That’s what overfitting is like in machine learning.

A model with overfitting learns the training data “too well,” including random noise and quirks that don’t really matter. It then makes great predictions on the data it saw before, but poor predictions on new, unseen data. In other words, it memorizes instead of learning the general pattern.