Notes
🔍

Capacity Control

When a supervised model “learns,” it can either pick up the real signal in your data or get distracted by noise and quirks. Capacity control is the set of choices that keep a model’s flexibility at the right level so it generalizes to new examples, not just the training set.

What capacity means

A model’s capacity is its ability to fit a wide range of functions. High-capacity models can represent very complex patterns (and can memorize), while low-capacity models are more constrained (and can miss real structure). Capacity is shaped by things like:

  • Number of parameters (e.g., deep neural networks vs. linear models)
  • Model structure (tree depth, polynomial degree, number of features used)
  • Strength of constraints such as regularization

How capacity is controlled in practice

Capacity control is implemented through knobs that deliberately limit complexity or stop learning before memorization kicks in:

  • L2/L1 regularization (weight decay, sparsity): e.g., scikit-learn’s C in LogisticRegression (smaller C = stronger regularization)
  • Early stopping: halt training when validation loss stops improving
  • Tree constraints: max depth, min samples per leaf in decision trees / random forests
  • Margin control: in SVMs, the regularization setting trades off margin size vs. training errors

Why it matters for real supervised tasks

In spam detection or credit scoring, uncontrolled capacity can yield impressive training accuracy while failing on new emails or applicants—classic overfitting. Too little capacity causes underfitting, like a house-price model that can’t capture neighborhood effects. Good capacity control narrows the train–test gap, stabilizes predictions, and makes model selection (via validation) meaningful rather than a contest of who can memorize the training set best.

Capacity control is the set of design choices and constraints that limit a supervised model’s effective complexity—its ability to fit arbitrary patterns in the training data—through architecture, feature space, and regularization (e.g., L1/L2 penalties, early stopping, dropout). It matters because generalization depends on matching capacity to data: too much capacity drives overfitting and unstable predictions, while too little causes underfitting and high bias.

Think of studying for an exam. You can either memorize every practice question word-for-word, or learn the underlying ideas so you can handle new questions. Capacity control in machine learning is the same idea: it’s about keeping a model from being “too powerful” in a way that makes it memorize the training examples instead of learning general patterns.

Why does this matter? In supervised learning (where the model learns from labeled examples), a model with too much freedom can look amazing on old data but fail on new data—like a spam filter that nails yesterday’s spam but misses today’s tricks. Capacity control helps models stay flexible enough to learn, but not so flexible that they overfit.