Gradient Descent

Training a supervised model is basically a search problem: you’re trying to find parameter values that make predictions match the labeled data as closely as possible. Gradient descent is the workhorse method for doing that search efficiently, even when there are millions of parameters.

What gradient descent is doing

Every supervised model chooses parameters (weights) to minimize a loss function—for example, mean squared error for house-price prediction or cross-entropy for spam detection. The gradient is the vector of partial derivatives of the loss with respect to each parameter; it points in the direction where the loss increases fastest. Gradient descent updates parameters by stepping in the opposite direction:

θ ← θ − η ∇L(θ)

Here, η is the learning rate, controlling step size. Intuitively, it’s like walking downhill in fog: you feel the slope (the gradient) and take a step downward, repeating until you reach a low point.

Variants you’ll see in practice

Batch gradient descent: uses the full training set to compute each gradient step (stable, but slow on large data).
Stochastic gradient descent (SGD): uses one example at a time (fast, noisy updates).
Mini-batch gradient descent: uses small batches (the standard choice in neural networks).

Why it matters in supervised learning

Optimization quality directly affects accuracy and reliability. If the learning rate is too large, training can diverge; too small, and it crawls or gets stuck. Gradient descent also interacts with feature scaling (poorly scaled features create “ravines” that slow progress) and with regularization terms in the loss. You’ll encounter it in tools like scikit-learn’s SGDClassifier and deep-learning libraries where backpropagation computes gradients automatically.

Gradient Descent is an iterative optimization algorithm that updates model parameters by moving them in the direction of the negative gradient of a chosen loss function, reducing training error step by step. It matters because it is the core procedure for fitting many supervised models—especially neural networks—so learning quality, speed, and stability depend on choices like learning rate and stopping criteria.

Imagine you’re hiking in thick fog and want to reach the lowest point in a valley. You can’t see far, so you feel the slope under your feet and take a small step downhill, then repeat. That “keep stepping downhill” idea is gradient descent.

In supervised learning, a model makes predictions (like spotting spam or estimating house prices) and we score how wrong it is with a single “badness” number. Gradient descent is the routine that nudges the model’s settings a little at a time to make that badness smaller. If the steps are too big, it can overshoot; too small, it learns slowly.