Stochastic Gradient Descent (SGD)

Training a supervised model is basically a long search for parameter values that make its predictions match the labeled data. Stochastic Gradient Descent (SGD) is a practical way to run that search when your dataset is too big (or your model too complex) to update using the entire dataset every step.

How SGD works

Most models are trained by minimizing a loss function (like squared error for regression or cross-entropy for classification). Gradient descent updates parameters in the direction that reduces loss, using the gradient (the slope of the loss with respect to the parameters). SGD makes this cheaper by estimating that gradient from a single example (or, more commonly, a small mini-batch) instead of the full training set. That estimate is noisy, but the noise is useful: it helps the optimizer move quickly and can shake it out of shallow bad spots.

Why it matters in supervised learning

SGD is the workhorse behind training linear models at scale and essentially all neural networks. Key choices determine whether it converges or diverges:

• Learning rate: too high overshoots; too low crawls.
• Batch size: smaller batches add noise (often better generalization), larger batches are steadier (often faster on GPUs).
• Momentum and variants (e.g., Nesterov): reduce zig-zagging and speed up progress.
• Learning-rate schedules: decay the step size to “settle” near a good solution.

Concrete examples and tools

In spam detection with millions of emails, SGD can train a linear classifier by streaming through data without holding everything in memory. In demand forecasting, it can update a model daily as new labeled sales arrive. In scikit-learn, you’ll meet it as SGDClassifier and SGDRegressor, which optimize linear models using mini-batches and regularization.