Entropy
When a decision tree decides where to split the data, it needs a quick way to tell whether a group of examples is “mostly one class” or a messy mix. Entropy is the score it uses to measure that messiness (uncertainty) in a set of labeled training examples.
What entropy measures
In classification trees, entropy is computed from the class proportions in a node. If a node is perfectly pure (all spam or all not-spam), entropy is 0. If it’s a 50/50 mix of two classes, entropy is high. For classes with probabilities p1, …, pK, the common definition is:
H = - Σ p_k * log2(p_k)The log makes uncertainty grow in a smooth, principled way: rare classes contribute less, and the maximum happens when classes are evenly balanced.
How trees use it: information gain
A split is good if it reduces entropy. Decision trees compare candidate splits by information gain: the parent node’s entropy minus the weighted average entropy of the child nodes. The split that creates the biggest drop produces children that are more class-consistent, which makes later predictions simpler and more reliable.
- Spam detection: splitting on “contains ‘free’” might create a low-entropy “mostly spam” branch.
- Fraud detection: a split on transaction amount could separate a mixed node into a mostly-legit and a mostly-fraud region.
Why it matters in practice
Entropy is one of the main impurity criteria (alongside Gini impurity) that controls how a tree grows. In scikit-learn you’ll see it as criterion="entropy" (or the closely related "log_loss") in DecisionTreeClassifier. Choosing and understanding this criterion affects split choices, tree shape, and how quickly the model finds clean, predictive rules.
Entropy is an information-theoretic measure of uncertainty in a class label distribution, defined as −∑c p(c) log p(c). In decision-tree classification, node entropy quantifies impurity: it is 0 when all samples belong to one class and maximal when classes are evenly mixed. It matters because splits are chosen to reduce entropy (maximize information gain), directly shaping the tree’s structure and predictive accuracy.
Think of a messy sock drawer. If every sock is mixed together, it’s hard to quickly find a matching pair. If the drawer is neatly sorted, it’s easy. Entropy is a way to describe that same idea of “messiness” or uncertainty in data.
In AI, especially in decision trees, entropy measures how mixed the examples are at a point in the tree. If a group contains mostly one outcome (like “spam” emails), entropy is low. If it’s a near-even mix of outcomes (“spam” and “not spam”), entropy is high. Decision trees try to split the data to reduce entropy, making each group clearer and easier to label.