Sequential Feature Selection

When you have dozens (or thousands) of input columns, it’s rarely true that every single one helps your model. Sequential Feature Selection is a practical way to “try out” features step by step and keep only the ones that actually improve predictive performance.

What it is and how it works

Sequential Feature Selection (SFS) is a wrapper feature-selection method: it chooses features by repeatedly training and evaluating a real model, using a scoring rule (like accuracy, AUC, or RMSE). Two common variants are:

• Forward selection: start with zero features, add the single feature that improves the validation score the most, repeat.
• Backward elimination: start with all features, remove the single feature whose removal hurts the score the least, repeat.

This “greedy” search is much cheaper than testing every possible subset, but it can miss the globally best combination because it commits to choices early.

Why it matters in supervised learning

SFS directly optimizes what you care about: generalization on unseen data. It can:

• Reduce overfitting by removing noisy or redundant inputs.
• Improve interpretability (fewer drivers to explain in credit scoring or diagnosis).
• Cut training/inference cost, especially with expensive models.

Because it trains many models, you typically run it with cross-validation and a clear stopping rule (target number of features or no meaningful score improvement).

Practical examples and tooling

• Spam detection: keep the handful of text/metadata features that raise AUC, drop the rest.
• House price prediction: select a compact set (size, location proxies, age) that lowers RMSE.
• Churn modeling: remove correlated usage metrics that don’t add incremental lift.

# scikit-learn provides SequentialFeatureSelector
from sklearn.feature_selection import SequentialFeatureSelector