Recursive Feature Elimination (RFE)

When you have dozens (or thousands) of input columns, it’s hard to know which ones actually help a supervised model and which ones just add noise. Recursive Feature Elimination (RFE) is a practical way to “trim the fat” by repeatedly training a model and throwing away the least useful features.

How RFE works (the mechanism)

RFE is a wrapper feature selection method: it uses a real predictive model to judge feature usefulness. The loop looks like this:

• Train an estimator that can score features (for example via coefficients in linear models or feature importances in tree-based models).
• Rank features by importance.
• Remove the weakest feature(s).
• Repeat until you reach a target number of features.

Because it retrains after each removal, it can catch situations where a feature looks useful at first but becomes redundant once another feature is present.

Where you’ll see it in practice

In scikit-learn, you’ll encounter sklearn.feature_selection.RFE and RFECV (which chooses the number of features using cross-validation). Typical uses include:

• Spam detection: reduce a huge bag-of-words down to the most predictive terms.
• Credit scoring: keep a smaller set of drivers (e.g., utilization, delinquencies) for interpretability and compliance.
• Disease diagnosis: select a compact biomarker panel that still predicts outcomes well.

Why it matters (and what can go wrong)

RFE can improve generalization, speed up training, and make models easier to explain. The trade-off is cost: it trains many models, so it can be slow on large datasets. It also depends on the estimator’s importance signal; for example, correlated features can cause the model to “split credit,” making rankings unstable. Done correctly (with feature selection inside cross-validation), RFE gives you a disciplined way to pick features based on real predictive performance.