Backpropagation

What is backpropagation?

Backpropagation (short for backward propagation of errors) is the algorithm that computes the gradient of a neural network's loss function with respect to every weight and bias in the network. That gradient tells an optimizer how to change each parameter to reduce the loss, which is the entire mechanism by which a network learns from data.

The procedure has two phases. The forward pass feeds an input through the layers to produce a prediction and a scalar loss, caching the intermediate values along the way. The backward pass then starts from the loss and applies the chain rule of calculus layer by layer, moving from the output back toward the input, to assemble the gradient with respect to each parameter. A single forward and backward pass yields the full gradient for one batch of examples.

Backpropagation is not the learning rule itself; it is the efficient way to obtain the gradients that a learning rule needs. It is almost always paired with gradient descent or one of its variants, which take the gradients and update the weights. Together they form the inner loop of training for nearly every modern deep learning system, from convolutional vision models to large language models.

• Computes the gradient of the loss with respect to all weights and biases.
• Forward pass produces the loss; backward pass propagates its gradient using the chain rule.
• Supplies the gradients that gradient descent uses to update parameters.

The chain rule: how the gradient flows backward

Backpropagation is the chain rule applied systematically to a composed function. A neural network is a deep composition: the output of one layer is the input to the next, so the loss is a function of the final layer, which is a function of the layer before it, and so on back to the weights. The derivative of a composition is the product of the derivatives of its parts, and backpropagation evaluates that product from the outside in.

Concretely, each layer computes a linear step z = Wx + b followed by a nonlinear activation a = f(z). To find how the loss L changes with a weight, backpropagation multiplies the gradient of the loss with respect to the layer's output by the derivative of the activation, then by the derivative of the linear step. The key quantity passed between layers is the gradient of the loss with respect to that layer's pre-activation, often written as delta. Each layer receives delta from the layer above, uses it to compute its own weight gradients, and sends a transformed delta to the layer below.

Because the same delta values feed both the weight-gradient computation and the message sent to the previous layer, the algorithm computes every partial derivative exactly once. This reuse is what makes the backward pass roughly as cheap as the forward pass, regardless of how many parameters the network has.

• A network is a function composition; its gradient is a product of per-layer derivatives.
• delta = gradient of the loss with respect to a layer's pre-activation, passed backward layer to layer.
• Each delta is reused for both weight gradients and the message to the previous layer.

The forward pass: predictions and stored activations

The forward pass runs the input through the network to produce a prediction and a loss, and it stores the intermediate values that the backward pass will need. For each layer it computes the pre-activation z = Wx + b and the activation a = f(z), passing a to the next layer until the final layer emits the output. A loss function then compares that output to the target and reduces it to a single scalar.

Caching matters. The backward pass needs each layer's pre-activation z (to evaluate the activation derivative f'(z)) and each layer's input activation a (to form the weight gradient). Storing these during the forward pass is why training a network uses far more memory than running it for inference. The peak memory of training scales with the number of activations held in the computation graph, which is one reason long sequences and large batches are memory-hungry.

This is also where the choice of activation function and loss function shapes learning. Saturating activations such as sigmoid produce near-zero derivatives in their flat regions, which shrinks the deltas that flow backward and can stall early layers, the vanishing-gradient problem. Pairings such as softmax with cross-entropy are popular partly because their combined gradient is simple and well-behaved.

• Forward pass computes z and a for every layer, then the final loss.
• It caches z and a values because the backward pass consumes them.
• Training memory grows with the stored activation graph, unlike inference.

The backward pass: assembling every weight's gradient

The backward pass begins at the loss and computes one delta per layer, output side first. It starts by differentiating the loss with respect to the network's output to get the output-layer delta, then repeatedly applies the per-layer rule: take the delta arriving from the layer above, multiply by the local activation derivative, and pull it through the layer's weight matrix to produce the delta for the layer below.

At each layer, before passing delta backward, the algorithm uses it to compute that layer's gradients. The weight gradient is the outer product of the layer's delta with the activation that entered the layer on the forward pass; the bias gradient is the delta itself. After sweeping through every layer, the network holds a complete gradient: one number for every weight and bias, all consistent with the same batch of data.

This single-sweep efficiency is the historical breakthrough. A naive approach that perturbed each weight separately to estimate its effect would cost one forward pass per parameter, which is hopeless for networks with millions or billions of weights. Backpropagation gets the entire gradient for a cost comparable to two forward passes, independent of parameter count.

• Start from the loss, compute the output delta, then propagate deltas backward.
• Each layer's weight gradient is delta times its incoming activation; bias gradient is delta.
• Full gradient costs about two forward passes, not one pass per weight.

How backpropagation pairs with gradient descent

Backpropagation supplies gradients; gradient descent uses them. After the backward pass produces dL/dW for every parameter, the optimizer takes a step in the opposite direction of the gradient, scaled by a learning rate, because the gradient points toward steepest increase of the loss and the goal is to decrease it. One such update per batch is the core training step.

In practice, networks are trained with mini-batch stochastic gradient descent: the forward and backward passes run on a small batch of examples, the gradients are averaged, and one update is applied. This estimates the true gradient from a sample, which is both faster and often a better optimizer than using the whole dataset at once. Modern training rarely uses plain SGD; adaptive optimizers such as Adam and AdamW keep running estimates of the gradient's first and second moments to set a per-parameter step size, which speeds convergence on the loss surfaces typical of deep networks.

The division of labor is clean and worth holding onto: backpropagation is exact differentiation of the loss, while the optimizer is a heuristic for using those derivatives. You can swap optimizers (SGD, momentum, Adam) without changing how gradients are computed, and you can change the architecture without changing the optimizer.

• Gradient descent moves each weight opposite its gradient, scaled by the learning rate.
• Mini-batch SGD averages gradients over a small batch per update.
• Adam and AdamW adapt per-parameter step sizes; the gradient source stays the same.

A minimal worked example in code

Here is one training step for a two-layer network with a sigmoid output and mean-squared-error loss, written without a framework so the forward and backward passes are explicit. It shows the caching of activations, the output delta, the backpropagated hidden delta, and the gradient descent update.

In real systems this hand-written logic lives inside an automatic differentiation engine. You write only the forward pass; the framework records each operation in a computation graph and runs the backward pass for you, which is why the algorithm is also called reverse-mode automatic differentiation.

import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))

# Forward pass (cache activations)
z1 = X @ W1 + b1;   h = np.maximum(0, z1)        # ReLU hidden
z2 = h @ W2 + b2;   y_hat = sigmoid(z2)          # sigmoid output
loss = np.mean((y_hat - y) ** 2)

# Backward pass (chain rule, output side first)
d_yhat   = 2 * (y_hat - y) / y.shape[0]
delta_out = d_yhat * y_hat * (1 - y_hat)         # through sigmoid
dW2 = h.T @ delta_out;            db2 = delta_out.sum(0)
delta_hidden = (delta_out @ W2.T) * (z1 > 0)     # through ReLU
dW1 = X.T @ delta_hidden;        db1 = delta_hidden.sum(0)

# Gradient descent update
for p, g in [(W1, dW1), (b1, db1), (W2, dW2), (b2, db2)]:
    p -= lr * g

• The forward pass stores h and y_hat for reuse in the backward pass.
• delta_out carries the loss gradient through the output activation; delta_hidden pulls it back through W2.
• Frameworks generate this backward pass automatically from the forward code.

A short history: who invented backpropagation

The modern algorithm has deeper roots than the famous 1986 paper. The reverse mode of automatic differentiation, which is mathematically what backpropagation is, was first published in 1970 by the Finnish researcher Seppo Linnainmaa in his master's thesis, with an earlier related method from Henry J. Kelley in 1960. Paul Werbos discussed the idea in his 1974 PhD thesis; his first explicit application of backpropagation to neural networks came later, described around 1982.

What brought backpropagation into the mainstream was the 1986 Nature paper 'Learning representations by back-propagating errors' by David Rumelhart, Geoffrey Hinton, and Ronald Williams. It showed clearly that the method lets hidden units learn useful internal representations rather than relying on hand-designed features, which is the property that separated it from the single-layer perceptron rule.

Backpropagation then waited on hardware and data. It became the engine of the deep learning era once GPUs made large networks trainable and large labeled datasets became available, roughly from 2012 onward. The algorithm itself has changed little; what changed is the scale at which it runs.

• Reverse-mode autodiff: Linnainmaa, 1970; Werbos discussed the idea in 1974 and applied it to neural networks around 1982.
• Popularized by Rumelhart, Hinton, and Williams in Nature, 1986 (323:533-536).
• Unchanged in essence; deep learning scaled it up with GPUs and large datasets.

Limitations and practical failure modes

Backpropagation computes exact gradients, but those gradients can still be hard to use. In deep networks, repeatedly multiplying small activation derivatives causes the vanishing-gradient problem, where early layers receive almost no learning signal. The opposite, exploding gradients, makes updates blow up. Practitioners counter these with non-saturating activations such as ReLU, residual (skip) connections, normalization layers, careful weight initialization, and gradient clipping.

There are also structural constraints. Standard backpropagation needs the whole network to be differentiable, which complicates discrete or non-differentiable operations and motivates tricks like the straight-through estimator. It requires storing forward-pass activations, so memory grows with depth and sequence length; gradient checkpointing trades extra compute for lower memory by recomputing some activations during the backward pass. And because gradients are exact only for the current parameters, the optimizer can still land in poor minima or saddle regions, which is the optimizer's problem, not backpropagation's.

Researchers continue to study biologically plausible or memory-cheaper alternatives, such as feedback alignment and forward-only methods, but as of 2026 backpropagation paired with an adaptive optimizer remains the standard way to train deep networks, including the transformers behind large language models.

• Vanishing and exploding gradients are mitigated by ReLU, residual connections, normalization, and gradient clipping.
• Requires differentiability and stored activations; checkpointing trades compute for memory.
• Exact gradients do not guarantee good minima; that is the optimizer's job.