Bias-Variance Tradeoff

When a model makes mistakes, it’s usually for one of two reasons: it’s too “stiff” to capture the real pattern, or it’s so “flexible” that it starts chasing noise. The bias-variance tradeoff is the idea that you typically can’t minimize both problems at the same time.

What bias and variance mean

Bias is error from overly simple assumptions. A high-bias model consistently misses in the same direction because it can’t represent the true relationship (classic underfitting). Variance is error from sensitivity to the particular training sample. A high-variance model changes a lot if you retrain it on a new dataset, because it has learned quirks of the training data (classic overfitting).

The tradeoff (and the key equation)

For many supervised learning settings, the expected prediction error can be thought of as:

Error ≈ Bias² + Variance + Irreducible noise

Making a model more complex often lowers bias (it can fit richer patterns) but raises variance (it becomes more sample-sensitive). The “best” complexity is where their sum is smallest.

Concrete example

Predicting house prices:

A straight-line model using only square footage may have high bias: it ignores neighborhood effects and renovations.
A very deep decision tree using dozens of features may have high variance: it fits oddities of the training homes and fails on new listings.

Why it matters in ML practice

This tradeoff drives model selection and regularization. Tools like cross-validation, Ridge/Lasso, limiting tree depth, early stopping, and dropout aim to control variance without inflating bias too much. In libraries like scikit-learn, many hyperparameters (e.g., max_depth, C, alpha) are essentially bias-variance knobs.

Bias-Variance Tradeoff is the tension between error from overly simple models (high bias, underfitting) and error from overly sensitive models (high variance, overfitting). Expected prediction error can be viewed as bias² + variance + irreducible noise, so reducing one often increases the other. It guides model selection and regularization in ML. Example: increasing polynomial degree lowers bias but can raise variance unless controlled (e.g., via L2 regularization).

Imagine throwing darts at a target. If your darts always land far from the bullseye in the same direction, that’s like bias: your aim is consistently off because your “rule” is too simple. If your darts land all over the place, that’s like variance: your aim changes too much from throw to throw.

In machine learning, the bias-variance tradeoff is the balancing act between these two problems. Very simple models often have high bias (they miss important patterns). Very complex models often have high variance (they “memorize” the training data and do poorly on new data). The goal is a sweet spot that predicts well on unseen examples.