Gradient Descent

What is gradient descent?

Gradient descent is an iterative optimization algorithm that finds parameter values minimizing a loss function by repeatedly moving in the direction of steepest descent. At each step it computes the gradient of the loss with respect to the parameters, then nudges the parameters a small amount in the opposite direction. Repeat enough times and the parameters settle into a region where the loss is low.

The gradient is the vector of partial derivatives of the loss with respect to every parameter. It points in the direction of steepest increase, so its negative points downhill. Following the negative gradient is the local greedy strategy for reducing loss as fast as possible from the current position.

Almost every modern neural network, from a small classifier to a frontier large language model, is trained with some variant of gradient descent. The gradients themselves come from backpropagation, which applies the chain rule to compute every partial derivative efficiently in one backward pass through the network.

• Iterative: take a step, recompute the gradient, repeat until the loss stops improving.
• Direction comes from the gradient; magnitude comes from the learning rate.
• Pairs with backpropagation, which supplies the gradients for a neural network.

The update rule and how it works

The core of gradient descent is one equation applied every step. Given parameters theta, a loss function L, and a learning rate eta, the new parameters are the old parameters minus eta times the gradient of L. The gradient is evaluated at the current parameter values, so the direction changes as the parameters move.

Think of the loss surface as a landscape where height is the loss and position is the parameter setting. Gradient descent drops a ball at the starting point and lets it roll downhill in small, fixed-size hops. The slope under the ball is the gradient; the hop size is governed by the learning rate. When the slope flattens to near zero, the updates shrink and the algorithm has converged to a minimum.

For a convex loss with a single bowl shape, gradient descent provably reaches the global minimum given a small enough learning rate. Neural network losses are non-convex and riddled with many local minima and saddle points, yet in high dimensions the method still finds parameter settings that generalize well, a result that took the field years to understand empirically.

• Update: subtract the scaled gradient from the current parameters every step.
• Convex losses converge to the global minimum; neural network losses are non-convex but still trainable.
• Near a minimum the gradient approaches zero, so steps naturally get smaller.

The learning rate: the most important hyperparameter

The learning rate eta controls how big each step is, and it is the single hyperparameter that most often makes or breaks training. Too small and training crawls, taking far more steps than necessary and sometimes stalling in a poor region. Too large and the steps overshoot the minimum, causing the loss to oscillate, plateau, or diverge to infinity.

Because a single fixed value rarely works for the whole run, practitioners use learning rate schedules that change eta over time. A common pattern warms the rate up over the first few hundred steps, then decays it (linearly, by cosine, or in steps) so early training explores quickly and late training fine-tunes carefully. Transformer training almost always uses warmup plus decay.

Adaptive optimizers reduce, but do not remove, the need to tune the learning rate by hand. Adam, AdaGrad, and RMSProp scale the step per parameter using running statistics of past gradients, so parameters with large or noisy gradients get smaller effective steps. You still pick a base learning rate, but the method is far more forgiving than plain gradient descent.

• Too high: loss oscillates or diverges. Too low: training is slow and may stall.
• Schedules (warmup then cosine or step decay) beat a single fixed value for deep models.
• Adam and friends adapt the per-parameter step but still need a base learning rate.

Batch, stochastic, and mini-batch gradient descent

The three variants differ only in how much data each step uses to estimate the gradient. Batch gradient descent computes the gradient over the entire training set before every update. The estimate is exact and the path is smooth, but one step requires a full pass over all data, which is impossibly slow for large datasets.

Stochastic gradient descent (SGD) goes to the other extreme: it estimates the gradient from a single training example per step. Updates are fast and frequent, and the noise in the estimate can actually help the optimizer escape shallow local minima and saddle points. The cost is a jittery, high-variance path toward the minimum.

Mini-batch gradient descent is the practical middle ground and what nearly everyone means by SGD in practice. It computes the gradient over a small batch, commonly 32 to 512 examples, balancing the stability of full-batch with the speed of single-example updates. Mini-batches also map cleanly onto GPU parallelism, which is why they dominate deep learning.

• Batch: whole dataset per step. Exact gradient, slow, smooth.
• Stochastic (SGD): one example per step. Fast, noisy, escapes shallow minima.
• Mini-batch: small batches (often 32 to 512). The default, and GPU-friendly.

Momentum, Adam, and modern optimizers

Plain gradient descent treats each step independently, which makes it slow through long, narrow valleys where the gradient keeps flipping direction. Momentum fixes this by accumulating an exponentially decaying average of past gradients, so the optimizer builds up velocity along consistent directions and damps out oscillations across them. The standard momentum coefficient is around 0.9.

Adam (Adaptive Moment Estimation), introduced by Diederik Kingma and Jimmy Ba in a 2014 arXiv preprint and published at ICLR 2015, combines momentum with per-parameter adaptive learning rates. It tracks a running mean of the gradients (first moment) and a running mean of their squares (second moment), then scales each parameter's step by both. Adam and its weight-decay-corrected variant AdamW are the default optimizers for training transformers and most large models as of 2026.

These optimizers are still gradient descent at heart: every one of them steps parameters opposite the gradient. What they change is how the raw gradient is filtered and scaled before the step, trading a little extra memory (to store the running statistics) for much faster and more stable convergence.

• Momentum averages past gradients to accelerate consistent directions and damp oscillation.
• Adam combines momentum with per-parameter adaptive step sizes (first and second moments).
• AdamW (Adam with decoupled weight decay) is the common default for transformers in 2026.

Gradient descent in code

The algorithm is short enough to write from scratch. The loop below fits a simple linear model y = w·x + b to data using mini-batch gradient descent: compute predictions, compute the gradient of the mean-squared-error loss, then step the parameters opposite that gradient. Real frameworks like PyTorch and TensorFlow automate the gradient step with autodiff and an optimizer object, but the mechanics are identical.

import numpy as np

w, b, lr = 0.0, 0.0, 0.01
for step in range(1000):
    y_pred = w * x + b              # forward pass
    error = y_pred - y             # residual
    grad_w = 2 * np.mean(error * x)  # dL/dw for MSE
    grad_b = 2 * np.mean(error)      # dL/db for MSE
    w -= lr * grad_w               # gradient descent step
    b -= lr * grad_b

• The loop is: forward pass, compute gradient, update parameters, repeat.
• Frameworks replace the manual gradient with backpropagation (autodiff).
• The learning rate lr and the number of steps are the knobs you tune.

Common problems: local minima, saddle points, and exploding gradients

Several failure modes recur when training with gradient descent. The classic worry is getting stuck in a local minimum, a low point that is not the lowest. In practice, high-dimensional neural network losses have relatively few bad local minima; saddle points, where the surface curves up in some directions and down in others, are the more common obstacle, and momentum-based methods usually push through them.

Vanishing and exploding gradients plague deep networks. When gradients are repeatedly multiplied through many layers, they can shrink toward zero (so early layers barely learn) or blow up (so updates become unstable). Careful weight initialization, normalization layers, residual connections, and gradient clipping all exist to keep gradient magnitudes in a healthy range.

There is also the question of when to stop. Training too long drives the training loss down while the model starts memorizing noise, which is overfitting. Watching a validation metric and stopping when it stops improving (early stopping) is the standard guard, alongside regularization.

• Saddle points, not local minima, are the more common obstacle in high dimensions.
• Vanishing or exploding gradients are managed by initialization, normalization, residuals, and gradient clipping.
• Stop using a validation metric (early stopping) to avoid overfitting.

Why gradient descent matters for AI

Gradient descent is the engine that turns data into a trained model. Every weight in a neural network, including the billions of parameters in a large language model, was set by gradient descent steps driven by backpropagation. Without an efficient way to follow the gradient, training networks of any meaningful size would be intractable.

Its scalability is what makes modern deep learning possible. Mini-batch updates parallelize across GPU cores, adaptive optimizers like AdamW keep training stable at scale, and learning rate schedules let runs that span weeks and trillions of tokens converge reliably. The same algorithm that fits a two-parameter line also trains frontier models, differing only in scale and tooling.

Understanding gradient descent clarifies why training choices matter: the learning rate sets how aggressively a model learns, the batch size trades noise against speed, and the loss function defines what the model is optimizing toward. These are the levers behind every fine-tuning run and every from-scratch pretraining job.

• Every parameter in a trained neural network, including LLMs, came from gradient descent.
• It scales from two-parameter toy models to trillion-token pretraining runs.
• Learning rate, batch size, and loss function are the practical levers it exposes.