Notes
🔍

Grid Search

Training a model is only half the job. The other half is choosing the “settings” that control how the model learns—settings you can’t directly learn from data, but that strongly shape performance.

What grid search is doing
Grid search is a systematic way to tune hyperparameters by trying a predefined set of values for each one and evaluating every combination. Think of it like testing every point on a checkerboard: you decide the rows and columns (the candidate values), then you measure which square gives the best result. For each combination, you train the model on training folds and score it on validation folds, typically using cross-validation, so you’re not fooled by a lucky split.

Concrete example in supervised learning
Suppose you’re building a spam detector with scikit-learn’s LogisticRegression. You might grid-search over the regularization strength C and penalty type, because these control under/overfitting. Or for an SVM predicting customer churn, you might search over C and gamma. A typical workflow looks like:

  • Choose a model and a scoring metric (e.g., AUC for churn, RMSE for house prices).
  • Define a parameter grid (e.g., C ∈ {0.1, 1, 10}).
  • Run cross-validated evaluation for every combination.
  • Pick the best combination and refit on all training data.

Why it matters (and the trade-offs)
Grid search matters because hyperparameters determine model capacity and generalization; ignore them and you can ship a model that looks good in training but fails in production. The trade-off is cost: the number of fits grows multiplicatively with grid size and CV folds. In practice you’ll see this as scikit-learn’s GridSearchCV, which automates the loop and tracks the best settings.

Grid Search is a hyperparameter optimization method that exhaustively evaluates a predefined set of hyperparameter values by training and scoring a model for every combination, typically using cross-validation. It matters because supervised models can be highly sensitive to hyperparameter choices (e.g., C and kernel parameters in SVMs, depth and learning rate in boosted trees); grid search provides a systematic, reproducible way to select settings that improve generalization.

Imagine you’re trying to bake the “best” cookie. You don’t just guess the perfect oven temperature and baking time—you try a bunch of combinations: 170°C for 10 minutes, 170°C for 12 minutes, 180°C for 10 minutes, and so on, then pick the tastiest result.

Grid Search is that same idea for AI models. A model has a few important “settings” you choose before training (like how complex it’s allowed to be). Grid search systematically tries a planned list of setting combinations, checks which one performs best on held-out data, and keeps the winner. It’s a simple, reliable way to tune a model instead of relying on luck.