Notes
🔍

AdaBoost

AdaBoost is a way to turn a bunch of “barely decent” models into one strong predictor by training them in sequence and making each new model pay extra attention to the examples the previous ones got wrong.

How AdaBoost works (the mechanism)

AdaBoost (short for Adaptive Boosting) builds an ensemble of weak learners, most commonly very small decision trees called decision stumps (one split). It maintains a weight for every training example. The loop looks like this:

  • Start with equal weights on all training points.
  • Train a weak learner to minimize weighted error (mistakes on high-weight points “count more”).
  • Give that learner a vote weight (often called alpha) based on how accurate it was.
  • Increase weights on the points it misclassified and decrease weights on the ones it got right.
  • Repeat, then predict by a weighted vote (classification) or weighted sum (regression variants exist).

This “adaptive” reweighting is the key: the ensemble keeps steering toward the hard cases.

Why it matters in supervised learning

AdaBoost is a practical bias-reduction tool: simple learners that underfit on their own can become highly accurate together. But it’s also sensitive to noisy labels and outliers—because those points keep getting up-weighted, the model can chase them and overfit. That’s why choices like the number of estimators and the learning_rate matter.

Where you’ll see it in practice

  • Fraud detection or spam filtering: focus progressively on tricky borderline cases.
  • Credit scoring: combine many simple rules into a stronger risk signal.
  • In scikit-learn: AdaBoostClassifier typically with a shallow tree base estimator.

AdaBoost (Adaptive Boosting) is a boosting ensemble algorithm that builds a strong predictor by training a sequence of weak learners—commonly decision stumps—and combining them with weights. Each round increases the weight of previously misclassified training examples so later learners focus on harder cases. It matters because it can substantially improve classification accuracy with simple base models and provides a clear, tunable trade-off between fit and generalization.

Think of a coach training a team by running lots of short drills. After each drill, the coach pays extra attention to the mistakes players keep making, so the next drill focuses on fixing those weak spots. AdaBoost (short for “Adaptive Boosting”) works in a similar spirit for AI.

Instead of relying on one big, complicated model, it builds a strong predictor by combining many simple “weak” ones. Each new simple model is encouraged to focus more on the examples the earlier models got wrong. By the end, their combined vote is usually much more accurate—useful for tasks like spam detection, fraud spotting, or medical risk prediction.