Linear Discriminant Analysis (LDA)
When you have labeled data and lots of features, it’s easy to feel like the classes are “mixed together” in a high-dimensional space. Linear Discriminant Analysis (LDA) is a way to rotate and compress that space so the classes separate as cleanly as possible.
What LDA is doing under the hood
LDA is a supervised linear projection: it uses the class labels to find a new set of axes (a lower-dimensional subspace) where different classes are far apart while points within the same class stay tight. It does this by comparing:
- Within-class scatter: how spread out each class is around its own mean.
- Between-class scatter: how far apart the class means are from each other.
Mathematically, LDA chooses projection directions that maximize the ratio of between-class scatter to within-class scatter. With K classes, you can get at most K−1 LDA components—one reason it’s a natural dimensionality reducer for classification.
Why it matters in supervised learning
Because LDA is label-aware, it can produce features that are more directly useful for classification than unsupervised methods like PCA. If you ignore class structure and only preserve variance (PCA), you can keep directions that look “important” but don’t separate classes. LDA can improve:
- Generalization by reducing noise dimensions
- Efficiency by shrinking feature space before a classifier
- Interpretability by showing which directions best discriminate classes
Practical use and examples
In spam detection or disease diagnosis, you might project thousands of text or biomarker features down to a few discriminative components, then classify. In scikit-learn, you’ll see both roles: dimensionality reduction via LinearDiscriminantAnalysis(n_components=...) and a built-in classifier (with a linear decision boundary under Gaussian-with-shared-covariance assumptions).
Linear Discriminant Analysis (LDA) is a supervised method that finds a linear projection of features that maximizes separation between labeled classes by increasing between-class variance while reducing within-class variance. It is used for both classification (via class-discriminant scores) and dimensionality reduction into a low-dimensional space (up to C−1 dimensions for C classes). It matters because it can improve class separability, reduce noise and computation, and strengthen generalization with fewer features.
Imagine you have a mixed box of red and blue marbles, and you want a single “best view” of the box that makes the colors look as separated as possible. Linear Discriminant Analysis (LDA) is like choosing that viewing angle. It takes examples where the correct group is already known (like “spam” vs “not spam,” or “healthy” vs “sick”) and finds a way to combine the measurements into a simpler set of numbers that keeps the groups far apart.
This helps because fewer, clearer measurements can make a classifier more reliable, faster, and easier to interpret.