Bagging

When a single model feels a bit “fragile”—small changes in the training data lead to noticeably different predictions—bagging is a practical way to make it steadier. The idea is to train many versions of the same kind of model and let them vote (or average) so that individual quirks cancel out.

How bagging works

Bagging (short for bootstrap aggregating) builds an ensemble by repeatedly sampling the training set with replacement to create multiple bootstrap samples. Each sample is the same size as the original dataset, but contains duplicates and leaves out some rows. You train a separate base learner on each bootstrap sample, then combine their predictions:

Regression: average the predicted values.
Classification: take a majority vote (or average predicted probabilities).

This reduces variance: the “wobbliness” of high-flexibility models. It does not primarily fix bias (systematic error from an overly simple model).

Why it matters in supervised learning

Bagging is especially valuable when your base model is unstable, meaning it changes a lot if the data changes a little—classic examples are decision trees. A single tree can overfit; a bagged collection of trees is far more reliable. This is the core intuition behind Random Forests: they use bagging plus extra randomness in feature selection to increase diversity among trees.

Practical examples and tools

Fraud detection: bagged trees reduce false alarms caused by noisy, rare patterns.
House price prediction: averaging many models smooths out extreme predictions from outlier neighborhoods.
Churn prediction: voting across bootstrapped models stabilizes probability estimates.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

model = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=200,
    bootstrap=True,
    n_jobs=-1,
    random_state=0
)

Bagging (bootstrap aggregating) is an ensemble technique that trains multiple copies of a base model on different bootstrap-resampled versions of the training set, then combines their predictions (averaging for regression, majority vote for classification). It matters because it reduces variance and improves stability and generalization, especially for high-variance learners like decision trees, and provides a practical foundation for methods such as Random Forests.

Bagging is like asking a crowd for opinions instead of trusting one person. Imagine you want to guess how much a house should sell for. Rather than relying on a single friend’s estimate, you ask many friends—each looking at a slightly different set of recent sales—and then you average their guesses.

In supervised learning, bagging trains many separate models on slightly different “resampled” versions of the same labeled data, then combines their predictions (often by voting or averaging). The point is to get a steadier, more reliable answer that’s less likely to be thrown off by quirks or noise in the training data.