L2 Regularization

When a supervised model starts “memorizing” the training data, it usually does so by pushing some weights to extreme values. L2 regularization is a simple way to discourage that behavior, nudging the model toward smoother, more stable solutions that generalize better.

What it is (mechanism)

L2 regularization adds a penalty to the training objective proportional to the squared size of the model’s weights. If your original loss is loss(data), the regularized objective becomes:

loss(data) + λ * Σ w_j^2

Here, λ (lambda) controls the strength of the penalty. Squaring matters: large weights get penalized much more than small ones, so the optimizer prefers spreading influence across many modest weights rather than relying on a few huge ones. In gradient-based training, this shows up as weight decay: each update gently pulls weights toward zero.

Why it matters in supervised learning

L2 regularization directly manages model capacity and the bias–variance tradeoff:

• Reduces overfitting, improving test performance when features are noisy or high-dimensional.
• Stabilizes models under multicollinearity (highly correlated features), a classic issue in linear regression.
• Produces better-calibrated, less extreme predictions in models like logistic regression.

If λ is too large, the model underfits (weights shrink too much); too small, you’re back to overfitting.

Where you’ll see it in practice

• Ridge regression is linear regression with L2 regularization—useful for house price prediction with many overlapping signals.
• LogisticRegression in scikit-learn uses L2 by default; the C parameter is the inverse of regularization strength.
• Neural networks commonly apply L2 as weight_decay in optimizers (e.g., AdamW) for tasks like fraud detection or churn prediction.