L1 Regularization

When a supervised model has lots of features, it can start “explaining” the training data a little too well—by leaning on many tiny, brittle patterns. L1 regularization is a way to push the model toward simpler explanations by encouraging it to use fewer features.

What it is (mechanism)

L1 regularization adds a penalty to the training objective equal to the sum of the absolute values of the model’s weights: ∑|wᵢ|. So instead of minimizing just the loss (like squared error or logistic loss), you minimize:

loss(data, w) + λ ∑|wᵢ|

The hyperparameter λ controls how strongly large weights are discouraged. The key behavior is that the absolute-value penalty creates a “sharp corner” at zero, which makes it optimal for many weights to become exactly 0.

Why it matters in supervised learning

This “weights hit zero” effect turns L1 into a practical form of feature selection. It can improve generalization when you have many correlated or weak predictors, and it makes models easier to interpret because only a subset of features remain active. If you ignore regularization in high-dimensional settings, models can overfit, producing unstable coefficients and disappointing test performance.

Where you’ll see it (examples)

Spam detection: from thousands of word features, L1 keeps only the most informative terms.
Credit scoring: selects a smaller set of stable signals, reducing reliance on noisy variables.
In scikit-learn, L1 is common in linear models like Lasso (regression) and logistic regression with penalty="l1" (via solvers that support it).

L1 regularization adds an L1 penalty (the sum of absolute parameter values) to a supervised model’s loss, encouraging many weights to become exactly zero. This yields sparse models that effectively perform feature selection while controlling model complexity. It matters because it reduces overfitting and improves interpretability, especially in high-dimensional settings (e.g., Lasso regression for selecting a small subset of predictive features).

Imagine packing for a trip with a strict luggage limit. You’re forced to leave behind “nice-to-have” items and keep only what you truly need. L1 Regularization plays a similar role in supervised learning: it adds a small “cost” for using lots of input signals, nudging the model to rely on fewer of them.

Why that matters: models can get distracted by noise in the training data and look great in practice runs but fail in the real world. L1 Regularization encourages simpler, more focused models that often generalize better. It can even push some inputs to be ignored entirely, like a spam filter deciding only a handful of words really matter.