Hyperparameter Search Space

When you tune a model, you’re really making a series of “design choices” the training algorithm won’t figure out by itself—like how strong regularization should be or how deep a tree can grow. The hyperparameter search space is the menu of choices you’re willing to consider.

What it is (and what it contains)

A hyperparameter search space is the set of candidate values (or ranges and distributions) for a model’s hyperparameters—settings that shape learning but aren’t learned directly from data like weights are. It can include:

• Continuous ranges (e.g., learning rate from 1e-5 to 1e-1)
• Discrete choices (e.g., max_depth in {3, 5, 7})
• Categorical options (e.g., penalty in {l1, l2})
• Conditional structure (e.g., “if kernel=rbf, then tune gamma” in an SVM)

How it’s used in practice

Search methods—grid search, random search, or Bayesian optimization—sample points from this space and evaluate them, typically with cross-validation. In scikit-learn, you express the space via parameter grids/distributions:

from sklearn.model_selection import RandomizedSearchCV
from sklearn.linear_model import LogisticRegression

param_dist = {"C": [0.01, 0.1, 1, 10], "penalty": ["l2"]}
search = RandomizedSearchCV(LogisticRegression(solver="lbfgs"), param_dist, n_iter=4)

Why it matters

The search space quietly determines what “good” models are even possible to find. If it’s too narrow, you miss strong settings (e.g., a churn model stuck under-regularized). If it’s too wide or poorly scaled (e.g., linear spacing for a parameter that behaves logarithmically), you waste budget and get noisy comparisons. A well-designed space reflects how the model behaves, the data size, and the real cost of mistakes—like prioritizing recall in fraud detection by including decision-threshold or class-weight options.