Notes
🔍

Learning Rate

Training a model is like walking downhill in fog toward the lowest point: you take steps, check whether you’re lower, and adjust. The learning rate is the knob that controls how big each step is when an optimizer updates the model’s parameters.

What it is mechanically

In gradient-based training (like gradient descent), the model computes the gradient of the loss with respect to each parameter—basically, “which direction increases the error fastest.” The update moves the opposite way:

parameter = parameter - learning_rate * gradient

A larger learning rate makes bigger jumps in parameter space; a smaller one makes tiny, cautious moves. This single number can decide whether training converges smoothly, crawls, or blows up.

Why it matters in supervised learning

If the learning rate is too high, the optimizer can overshoot the best solution and bounce around, making the loss fluctuate or diverge. If it’s too low, training becomes painfully slow and can get stuck looking “flat” even when improvement is possible. This affects real tasks directly:

  • Credit scoring with logistic regression: too high can prevent the classifier from settling; too low can take many epochs to reach a useful decision boundary.
  • House price prediction with a neural network: too high can cause unstable training; too low can underfit within your time budget.

How it’s handled in practice

You’ll see it as a hyperparameter like learning_rate in XGBoost or as part of an optimizer setup in deep learning (e.g., Adam). Many training pipelines use learning rate schedules (decay over time) or adaptive optimizers that effectively scale step sizes per parameter, but the base learning rate still strongly controls training behavior.

Learning rate is a hyperparameter that sets the step size an optimizer uses to update model parameters in response to the loss gradient during training. A higher learning rate speeds progress but can overshoot and destabilize convergence; a lower value improves stability but can slow learning or stall. It matters because it largely determines whether supervised training converges efficiently to a good solution or fails to train at all.

Think of teaching a kid to throw a ball into a basket. If you tell them to “adjust a lot” after each miss, they might overcorrect and keep missing in new ways. If you tell them to “adjust just a tiny bit,” they’ll improve steadily but it may take forever. In machine learning, the learning rate is that “how big of an adjustment” setting during training.

It controls how quickly a model changes its behavior as it learns from examples, like spotting spam or predicting house prices. Too high can make learning unstable; too low can make learning painfully slow.