Notes
🔍

Kullback-Leibler (KL) Divergence Loss

When a classifier outputs probabilities, you don’t just care whether it picked the right class—you care whether its whole probability distribution “looks like” the truth. Kullback-Leibler (KL) Divergence Loss is a way to measure how far the model’s predicted distribution is from a target distribution.

What it measures (and what it isn’t)

KL divergence compares two probability distributions: a target distribution P (what you want) and a predicted distribution Q (what the model outputs). It’s defined as:

KL(P || Q) = Σ_i P(i) * log( P(i) / Q(i) )

It’s not a true distance: it’s asymmetric (KL(P||Q) ≠ KL(Q||P)) and it heavily penalizes putting very low probability on outcomes that the target says are likely.

How it shows up as a classification loss

In supervised classification, the target P is often a one-hot label (100% on the correct class). In that common case, minimizing KL divergence is equivalent to minimizing cross-entropy loss (up to a constant that doesn’t affect training). KL becomes especially natural when P is a “soft” label distribution:

  • Label smoothing: targets are slightly spread out to reduce overconfidence.
  • Knowledge distillation: a student model learns to match a teacher’s full probability output.
  • Imperfect or ambiguous labels: multiple classes can be plausibly correct.

Why it matters in practice

KL-based losses push models toward well-calibrated probabilities, not just correct argmax decisions—important in credit scoring, medical triage, and fraud review where confidence affects actions. In PyTorch you’ll see torch.nn.KLDivLoss (used with log-probabilities), and in TensorFlow/Keras you’ll encounter tf.keras.losses.KLDivergence.

Kullback-Leibler (KL) Divergence Loss measures how far a model’s predicted probability distribution is from a target distribution, computed as the expected log difference between them. In supervised classification it is commonly used with soft targets (e.g., label smoothing or knowledge distillation), and reduces to cross-entropy when the target is one-hot. It matters because it directly trains probabilistic outputs to match desired class probabilities, supporting calibrated predictions and stable learning.

Imagine you asked 100 people to vote for their favorite ice cream flavor, and you predicted the results. If your prediction says “mostly vanilla” but the real crowd is “mostly chocolate,” you’re not just a little off—you’ve misunderstood the whole pattern. Kullback-Leibler (KL) Divergence Loss is a way to measure that kind of mismatch between two sets of probabilities.

In supervised learning, it’s used when a model outputs probabilities (like “70% spam, 30% not spam”) and you want those probabilities to match the true target distribution. Lower KL loss means the model’s probability guesses are closer to reality.