Multi-Label Stratification

When each example can have several correct labels at once, splitting data into train/validation/test sets gets trickier than it looks. Multi-label stratification is the idea of making those splits “fair” so the label patterns you care about don’t accidentally vanish from one split.

What it is and what it preserves

Multi-label stratification is a dataset-splitting method for multi-label classification that tries to keep the distribution of labels similar across splits. In single-label classification, stratification just matches class proportions. In multi-label settings, you need to preserve:

• Per-label prevalence (each label appears at similar rates in each split).
• Label co-occurrence structure (common label pairs/sets don’t end up concentrated in only train or only test).

A practical way to do this is iterative assignment: repeatedly place examples so that, for each label (and sometimes label pairs), the remaining “needed counts” in each split stay balanced.

Why it matters in supervised learning

If you ignore multi-label stratification, you can get misleading evaluation: a rare label might appear only in training (inflating confidence) or only in test (making performance look terrible), and label combinations critical to the task might never be seen during training. This directly affects threshold tuning, calibration, and metrics like micro/macro F1 and precision-recall curves, which are sensitive to label imbalance.

Concrete example and tooling

In medical coding, a note might have labels like “diabetes,” “hypertension,” and “kidney disease.” Multi-label stratification helps ensure each condition—and their frequent co-occurrence—shows up in both training and evaluation. In Python, you’ll commonly see this via the iterative stratification approach (e.g., the MultilabelStratifiedKFold splitter in the iterstrat package), which mirrors scikit-learn’s splitters but respects multi-label structure.