Feature Importance
When a supervised model makes predictions, it’s using many input signals at once—income, age, past purchases, lab values, and so on. Feature importance is about figuring out which of those signals the model relied on the most.
What “feature importance” means
Feature importance is a score (or ranking) that describes how strongly each input feature influences a model’s predictions. Most importance methods are global: they summarize the model’s behavior across a dataset, not just one individual case. The exact meaning depends on the method:
- In tree-based models (e.g., Random Forest, XGBoost), importance can come from how much a feature reduces error when the model splits on it (often called gain or impurity decrease).
- Permutation importance measures how much performance drops when you shuffle one feature column, breaking its relationship with the target.
- For linear models, standardized coefficients are sometimes treated as importance, but they depend heavily on scaling and correlation.
How it shows up in real projects
In spam detection, feature importance might reveal that certain keywords or sender reputation dominate decisions. In credit scoring, it can show whether utilization ratio matters more than number of open accounts. In churn prediction, it might highlight recent support tickets or declining usage. In scikit-learn, you’ll commonly see model.feature_importances_ for trees, or sklearn.inspection.permutation_importance for model-agnostic scoring.
Why it matters (and common traps)
Feature importance helps with debugging (catching leakage), communicating with stakeholders, and guiding feature engineering. But it’s not the same as causality: a feature can be “important” because it’s correlated with the real driver. Also watch for bias when features are correlated—importance can be split across them or assigned inconsistently, especially with impurity-based tree importances.
Feature importance quantifies how much each input feature contributes to a supervised model’s predictions, either globally across the dataset or for specific instances. It is computed with model-specific scores (e.g., tree split gain) or model-agnostic methods (e.g., permutation importance, SHAP). Feature importance matters because it supports interpretability, debugging, and feature selection, and helps detect spurious reliance on proxies or leakage that can undermine generalization.
Imagine you’re trying to guess a house’s price. You might look at the number of bedrooms, the neighborhood, and how close it is to a train station. Some of those clues matter a lot more than others. Feature Importance is the AI version of asking, “Which clues did the model rely on most?”
In supervised learning, a model makes predictions using many inputs (called “features,” meaning pieces of information). Feature Importance helps you see which features had the biggest influence on the model’s decisions. This is useful for trust and debugging—like checking whether a medical model is focusing on symptoms rather than irrelevant details.