Chi-Square Feature Selection

When you have thousands of possible inputs (words in emails, products a customer viewed, symptoms in a chart), it’s comforting to have a quick way to ask: “Which of these features actually relates to the label I’m trying to predict?” Chi-square feature selection is one of the simplest, most widely used tools for that job—especially for text and other count-like data.

What it is and what it measures

Chi-Square Feature Selection is a supervised filter method that scores each feature by how strongly it is associated with the target class. It uses the chi-square (χ²) test of independence: if a feature and the class label were truly unrelated, the observed co-occurrence counts should look like what you’d “expect by chance.” A large χ² score means the feature’s presence/absence (or counts) differs a lot across classes, so it’s informative for classification.

How it’s applied in practice

You typically build a contingency table per feature (e.g., word appears vs. doesn’t appear, crossed with spam vs. not spam), compute χ², then keep the top-k features. Common use cases:

Spam detection: words like “free” or “winner” get high χ² if they concentrate in spam.
Disease diagnosis: a symptom that appears disproportionately in positive cases scores high.
Churn prediction: certain complaint categories might strongly align with churners.

Why it matters (and key caveats)

It reduces dimensionality, speeds training, and can improve generalization by removing noisy features before fitting models like Logistic Regression or linear SVMs. In scikit-learn you’ll see it as:

from sklearn.feature_selection import SelectKBest, chi2

Important constraints: χ² assumes non-negative feature values (counts, binary indicators, TF-IDF works if non-negative) and is mainly for classification, not regression. Also, it evaluates features one-by-one, so it can miss interactions between features.

Chi-Square Feature Selection is a filter method that ranks features by the chi-square (χ²) test of independence between each feature and the target label, selecting those with the strongest statistical association. It is commonly applied to non-negative, count-based, or one-hot encoded categorical features (e.g., word counts in text classification). It matters because it quickly removes weakly related features, reducing dimensionality, training time, and overfitting risk while preserving predictive signal.

Imagine you’re trying to figure out which clues in a pile of evidence actually point to the right suspect. Some clues show up a lot with one suspect and rarely with others, so they’re useful. Chi-Square Feature Selection does a similar job for AI: it helps pick the input signals (“features,” like words in an email or symptoms in a medical form) that are most strongly linked to the correct label (like “spam” vs “not spam”).

It’s a quick way to drop features that don’t seem connected to the outcome, so the model can focus on the most informative patterns and ignore noise.