Post-Pruning
A decision tree can keep splitting until it perfectly matches the training data—like memorizing every quirk in a practice set. That usually looks impressive on training accuracy, but it’s a classic way to end up with a model that performs worse on new, unseen cases.
What post-pruning is
Post-pruning is a way to simplify a decision tree after it has been fully (or nearly fully) grown. The idea is to remove branches that don’t earn their keep—splits that reduce training error but don’t improve, or even hurt, generalization. Technically, pruning replaces a subtree with a single leaf (or a smaller subtree), trading a small increase in training error for a bigger decrease in variance and better test performance.
How it’s done in practice
A common approach is cost-complexity pruning (also called weakest-link pruning): it scores each candidate tree by combining fit and size, then chooses the best trade-off.
- Define an objective like: error(tree) + α × (# leaves)
- Generate a sequence of pruned trees for different α values
- Pick α using a validation set or cross-validation
In scikit-learn, this shows up as ccp_alpha in DecisionTreeClassifier/DecisionTreeRegressor.
Why it matters (with examples)
Post-pruning is one of the main defenses against overfitting in trees. In spam detection, an unpruned tree might split on rare words that appear in only a handful of emails; pruning removes those brittle rules. In credit scoring, it can prevent the model from creating tiny “exception” leaves for a few applicants, improving stability and fairness of decisions. The result is usually a smaller, more interpretable tree that predicts better on real-world data.
Post-pruning is a decision-tree regularization step applied after growing a full tree, where subtrees are removed or replaced by leaves when they do not improve performance on a validation set or under a complexity-penalized criterion (e.g., cost-complexity pruning). It matters because it reduces overfitting, improves generalization, and yields smaller, more interpretable trees with more stable predictions.
Think of writing a long set of instructions, then going back and crossing out the fussy, overly specific steps that only helped in one weird situation. That cleanup is like Post-Pruning.
In a decision tree, the model can grow lots of tiny branches to perfectly match the training examples it learned from. The problem is that some of those branches are “noise”—they reflect quirks of the past rather than patterns that will repeat. Post-Pruning means you first grow the full tree, then trim back branches that don’t help on new, unseen data. The result is a simpler tree that usually makes more reliable predictions, like for spam detection or medical risk checks.