Notes
🔍

Tree Depth

When you look at a decision tree, it’s tempting to think “more splits means a smarter model.” Tree depth is the simple knob that controls how far that splitting can go—and it has a huge impact on whether the tree learns real patterns or just memorizes the training data.

What tree depth means

Tree depth is the length of the longest path from the root node (the first split) down to a leaf (a final prediction). A depth-1 tree is a “stump”: one split, then predictions. Each extra level lets the model carve the feature space into smaller regions by stacking rules like “if income > 50k” then “if age < 30” then “if tenure > 2 years,” and so on.

Why it matters: bias, variance, and overfitting

Depth controls model complexity:

  • Shallow trees (small depth) have higher bias: they miss interactions and non-linear structure.
  • Deep trees have higher variance: they fit quirks and noise, creating tiny leaves with very few samples.

In practice, an unconstrained tree can drive training error near zero while test performance gets worse—classic overfitting.

Practical examples and how you set it

  • Spam detection: a very deep tree might learn rare word combinations that appear once, hurting generalization.
  • Credit scoring: limiting depth helps keep rules stable and more interpretable for risk teams.
  • House prices: moderate depth captures interactions (neighborhood × size) without fragmenting into single-house leaves.

In scikit-learn, depth is controlled directly with max_depth in DecisionTreeClassifier/DecisionTreeRegressor, and tuned via cross-validation.

Tree Depth is the maximum number of splits (edges) from the root node to any leaf in a decision tree, equivalently the number of decision levels the model can apply before producing a prediction. It controls model capacity: greater tree depth captures more complex interactions but increases overfitting risk, while shallower trees improve generalization and interpretability. Depth is a key hyperparameter for pruning and for limiting complexity in ensembles like random forests and gradient boosting.

Think of a “choose-your-own-adventure” book. Each time you make a choice, you go one step deeper into the story. In a decision tree, Tree Depth is the same idea: it’s how many decision steps you can take from the start to reach an answer.

A shallow tree (small depth) asks only a few questions, like “Is the email from a known contact?” It’s simpler and less likely to get distracted by tiny quirks in the training examples. A deep tree (large depth) can ask many questions and fit the training data very closely, but it may start “memorizing” noise and make worse predictions on new cases.