Backpropagation

Definition: The algorithm that efficiently computes the gradient of a neural network’s loss with respect to every weight, by propagating error backward from the output layer to the input layer using the chain rule.

How It Works

Forward pass: input flows through the network to produce a prediction
Compute loss by comparing prediction to the true label
Backward pass: apply the chain rule layer by layer, from output back to input, to get each weight’s gradient
Gradient descent then uses these gradients to update the weights
Each layer caches the intermediate values computed during the forward pass (pre-activations, activations)
The backward pass needs those cached values to compute local derivatives
This caching is why training consumes far more memory than inference — inference can discard intermediate activations immediately, training can’t
The process repeats for every batch, every epoch, incrementally nudging weights toward lower loss
Frameworks build a computation graph during the forward pass specifically so the backward pass knows which operations to differentiate and in what order

Under the Hood

Backprop is a disciplined, efficient application of the chain rule across a computation graph. For a simple 2-layer network with loss L, weights W1, W2, pre-activations z1, z2, and activations a1 = f(z1), a2 = f(z2):

dL/dW2 = dL/da2 * da2/dz2 * dz2/dW2
dL/dW1 = dL/da2 * da2/dz2 * dz2/da1 * da1/dz1 * dz1/dW1

Notice dL/da2 * da2/dz2 is shared between both gradient computations. Backprop computes this term once — call it delta2, the “error signal” at layer 2 — and reuses it rather than recomputing shared subexpressions from scratch. This reuse is what makes backprop O(n) in the number of edges in the network, instead of the exponential blowup of computing each weight’s gradient independently.

Generalizing, the error signal at layer l is:

delta_l = (W_(l+1)^T . delta_(l+1)) elementwise* f'(z_l)

This recursive definition is the entire algorithm: each layer’s error depends only on the next layer’s error and local derivatives. Once you have delta_l for every layer, the gradient with respect to that layer’s weights is delta_l . a_(l-1)^T.

Forward and Backward Pass, Visualized

The recursive delta_l relationship above is easier to hold in your head as a picture than as an equation. For a small network with 2 inputs, 2 hidden neurons, and 1 output, the forward pass (solid arrows, left to right) and the backward pass (dashed arrows, right to left) look like this:

Every solid edge carries an activation forward; every dashed edge carries an error signal (delta) backward along that exact same connection, scaled by the same weight the forward pass used to travel the other way. Nothing on the backward pass travels a path the forward pass didn’t already establish — this is precisely why backprop is cheap: it reuses the network’s existing wiring rather than computing each weight’s gradient from an independent perturbation.

The same two passes, viewed as a timeline of messages between layers instead of a static graph:

Both diagrams show the same fact from different angles: the backward pass is not a separate computation bolted onto the network, it’s the forward computation graph traversed in reverse, with each step’s local derivative multiplied in as it goes.

Worked numeric example

A tiny network: one input x = 1.0, one weight w = 0.5, no bias, activation f(z) = z^2 (illustrative, not a real activation), target y = 4.0, loss L = (f(z) - y)^2.

Forward:  z = w*x = 0.5        a = f(z) = z^2 = 0.25       L = (0.25 - 4)^2 = 14.0625
Backward: dL/da = 2*(a - y) = 2*(0.25 - 4) = -7.5
          da/dz = 2*z = 1.0
          dz/dw = x = 1.0
          dL/dw = dL/da * da/dz * dz/dw = -7.5 * 1.0 * 1.0 = -7.5
Update:   w_new = w - lr * dL/dw = 0.5 - 0.01 * (-7.5) = 0.575

Every layer of a real network repeats this same three-step pattern (local derivative, multiply into the incoming signal, pass the product further back) — depth just means more links in the chain.

History

Backpropagation as applied to multi-layer neural networks was popularized by Rumelhart, Hinton, and Williams in their 1986 paper “Learning representations by back-propagating errors.” The core mathematical technique (reverse-mode automatic differentiation) dates back further in control theory and was independently derived by several researchers, including Paul Werbos in his 1974 PhD thesis. It took until the 2000s-2010s — with GPUs, larger datasets, and tricks like ReLU and Batch Normalization to combat vanishing gradients — for backprop-trained deep networks to become practically dominant.

The reverse-mode traversal underlying backprop was described even earlier than Werbos: Finnish mathematician Seppo Linnainmaa’s 1970 master’s thesis at the University of Helsinki laid out the technique now called reverse-mode automatic differentiation and implemented it in Fortran, with no connection to neural networks at all — that link came later. Werbos’s 1974 Harvard PhD thesis was the first to propose applying this kind of reverse accumulation to train neural-network-like models specifically, but it went largely unnoticed for years, a casualty of both the AI winter’s funding collapse and not being published in a widely-read venue until 1981.

Backpropagation was then independently rediscovered more than once before Rumelhart, Hinton, and Williams’s 1986 paper made it stick: David Parker described an equivalent method in 1985, and Yann LeCun published his own independent derivation the same year, in France. It was LeCun who gave backprop its first influential practical demonstration, training a small convolutional network with it in 1989 to read handwritten ZIP codes for the U.S. Postal Service — one of the earliest examples of a multi-layer network solving a real, economically useful task via gradient-based learning, years before deep learning’s mainstream breakout and a direct precursor to the LeNet-5 architecture LeCun published in 1998.

The gap between 1986 and the 2000s-2010s resurgence mentioned above wasn’t idle — it was largely spent understanding why deep and recurrent networks trained by backprop kept failing to propagate useful signal across many layers or time steps. Sepp Hochreiter’s 1991 diploma thesis, written in German and consequently under-recognized outside German-speaking research circles for years, gave the first formal account of exactly this failure mode: the vanishing/exploding gradient problem. It directly motivated LSTM’s 1997 gating mechanism and, later, the ReLU activations and normalization techniques credited above with finally making deep backprop-trained networks practical at scale.

Why It Matters

The algorithm that made training deep (many-layer) neural networks computationally feasible
Without it, gradient descent would require impractically expensive numerical estimation per weight
Naive numerical differentiation (perturb each weight slightly, measure the change in loss) requires one forward pass per weight
For a network with millions of parameters, that’s millions of forward passes per single update
Backprop gets the exact gradient for all weights in a single backward pass, roughly the same cost as one forward pass
Modern frameworks (PyTorch, TensorFlow) automate this via “autograd” — you never hand-derive these formulas in practice
Understanding the mechanics is still essential for debugging vanishing/exploding gradients, custom loss functions, and custom layers
Composes cleanly with any differentiable operation, which is why modern frameworks can backprop through control flow, custom loss functions, and even other networks (as in a GAN’s generator-discriminator pair) without any change to the core algorithm
The same reverse-mode traversal that computes weight gradients can compute gradients with respect to the input itself, which is what powers adversarial example generation, saliency maps, and neural style transfer — none of it requires a different algorithm, just a different variable to differentiate with respect to

Code Example

import torch

# Autograd performs backprop automatically once you call .backward()
x = torch.tensor([1.0, 2.0], requires_grad=False)
W1 = torch.randn(2, 3, requires_grad=True)
W2 = torch.randn(3, 1, requires_grad=True)

z1 = x @ W1
a1 = torch.relu(z1)
z2 = a1 @ W2
loss = (z2 - torch.tensor([1.0]))**2

loss.backward()          # this IS backpropagation
print(W1.grad)           # dL/dW1, computed via chain rule
print(W2.grad)           # dL/dW2

# Manual gradient descent step (what an optimizer does under the hood)
with torch.no_grad():
    W1 -= 0.01 * W1.grad
    W2 -= 0.01 * W2.grad
    W1.grad.zero_()       # must clear, or next backward() accumulates onto this
    W2.grad.zero_()

The same computation, done by hand in plain JavaScript — no autograd, just the forward pass followed by the manual chain-rule steps that autograd normally hides:

Real-World Example

Training an image classifier on ImageNet with a ResNet-50: a single forward pass pushes a batch of images through roughly 50 layers to produce class predictions, and the loss (cross-entropy against the true labels) is a single number. Backprop then walks that number backward through all 50 layers in one pass, computing gradients for around 25 million parameters simultaneously.

Without backprop’s reuse of shared intermediate terms, computing those 25 million gradients via naive numerical differentiation would require roughly 25 million separate forward passes per training step — turning a training run that takes hours into one that would take years on the same hardware.

Fine-tuning a pretrained language model like BERT for a downstream task (sentiment classification, named entity recognition) runs the identical algorithm at a much smaller scale: a forward pass through 12 or 24 transformer blocks produces logits, cross-entropy loss compares them to labels, and backprop computes gradients through every attention and feed-forward sublayer back to the embedding layer — typically across the entire network unless earlier layers are explicitly frozen, even when only a small classification head is meant to change substantially.

Backprop’s first influential practical demonstration predates modern deep learning by decades: Yann LeCun’s 1989 system for reading handwritten ZIP codes for the U.S. Postal Service trained a small convolutional network with backpropagation, at a time when most of the field still doubted multi-layer networks were trainable at any useful scale — the same core backward pass described above, running on hardware many orders of magnitude weaker than what trains today’s models.

Common Pitfalls

Vanishing/exploding gradients in very deep networks, where signal shrinks or blows up as it propagates backward
Treating backprop and gradient descent as the same thing — backprop computes gradients, gradient descent uses them to update weights
Forgetting to zero out accumulated gradients between batches (optimizer.zero_grad() in PyTorch) — gradients accumulate by default, silently corrupting training
Detaching a tensor from the computation graph (e.g., converting to NumPy and back) partway through a custom forward pass, which silently breaks gradient flow upstream
Assuming backprop gives a globally optimal gradient direction — it gives the exact local gradient, not a guarantee about the overall loss surface
Loss landscapes in deep networks are non-convex, so gradient descent can still land in a poor local minimum or saddle point even with perfectly correct gradients
Running backward() twice on the same graph without retain_graph=True, which raises an error because intermediate buffers are freed after the first backward pass by default
Freezing a layer by setting requires_grad=False on its parameters but forgetting that an upstream operation still routes through it, so gradients silently fail to reach parameters that do need updating
Applying an in-place operation (e.g., x += 1 on a tensor x the backward pass still needs) that overwrites a value autograd had cached, producing a cryptic “one of the variables needed for gradient computation has been modified by an inplace operation” error
Assuming a deeper network always learns a better function — without residual connections or normalization, added depth can make gradients vanish before they ever reach early layers, so extra depth can make training worse despite adding representational capacity

Best Practices

Use gradient clipping (capping the norm of the gradient vector) when training RNNs or very deep networks prone to exploding gradients
Check for NaN or Inf in gradients early in a training run — it’s almost always a numerical stability issue (bad initialization, too-high learning rate, or a log(0) in the loss), and it’s easier to catch immediately than after hours of training
Use torch.autograd.gradcheck or manual finite-difference checks when implementing a custom layer’s backward pass, to confirm your hand-derived gradient matches numerical differentiation
Profile memory usage on deep networks — if you don’t need gradients for a subgraph (e.g., a frozen pretrained backbone), wrap it in torch.no_grad() to skip caching activations you’ll never backpropagate through
Initialize weights with a scheme matched to the activation function (He initialization for ReLU-family activations, Xavier/Glorot for tanh/sigmoid) — mismatched initialization compounds the vanishing/exploding gradient problem before training even starts
When debugging a custom autograd function, test it in isolation on a tiny input first (a 2x2 tensor, not a full batch) so a shape or sign error in the backward formula is obvious rather than buried inside a full-size run

Comparison: Backprop vs. Forward-Mode Autodiff

	Reverse-mode (backprop)	Forward-mode autodiff
Efficient when	Few outputs, many inputs (typical neural net: 1 loss, millions of weights)	Many outputs, few inputs
Cost per pass	One backward pass computes gradients for all weights	One forward pass computes derivatives for one input direction only
Memory	Must cache intermediate activations for the backward pass	No need to cache the full forward computation
Typical use	Deep learning training	Sensitivity analysis, some scientific computing
Compute cost scaling	One pass gives gradients for all weights, regardless of weight count	Cost scales with the number of input directions being differentiated
Used by	PyTorch and TensorFlow as the default/primary mode; JAX’s `grad`	JAX’s `jvp`; specialized scientific-computing and engineering tools

Common Interview Questions

What’s the computational complexity of backprop relative to the forward pass? Roughly the same order — both are O(number of edges in the computation graph).
Why do we need to cache activations from the forward pass? Because the local derivatives used in the chain rule (like f'(z)) depend on the actual values computed during the forward pass, not just the network’s structure.
What’s the difference between backprop and backpropagation through time (BPTT)? BPTT is backprop applied to a recurrent network unrolled across time steps — same chain rule, applied along the temporal dimension in addition to the depth dimension.
Why is backprop (reverse-mode autodiff) preferred over forward-mode for training neural networks? Because a network has one scalar loss but millions of weights — reverse mode computes gradients for all of them in a single backward pass, while forward mode would need a separate pass per weight.
What causes exploding gradients, mechanically? Repeated multiplication by weight matrices (or derivatives) with magnitude greater than 1 across many layers compounds multiplicatively, the mirror image of the vanishing gradient problem caused by factors less than 1.
Why is the chain rule specifically necessary here, rather than some other technique? A network’s loss is a composition of many nested functions, one per layer — the chain rule is the calculus rule for differentiating a composition of functions, and there’s no way to get an exact gradient through that composition without applying it, short of a per-weight numerical approximation that doesn’t scale.
Derive dL/dz for a single sigmoid output unit with squared-error loss. With a = sigmoid(z) and L = 1/2 (a - y)^2: dL/da = a - y, and the sigmoid’s own derivative is da/dz = a(1-a), so by the chain rule dL/dz = (a - y) * a * (1 - a). Note that when a is near 0 or 1 this product collapses toward zero regardless of how wrong the prediction is — a concrete, derivable instance of sigmoid saturation causing vanishing gradients.

Debugging Backprop in Practice

A short checklist engineers actually use when a network isn’t training:

Print gradient norms per layer after the first backward pass — near-zero norms in early layers point to vanishing gradients, huge norms point to exploding gradients
Verify the loss actually decreases on a tiny toy dataset (even 10 examples) before trusting results on the full dataset — if a model can’t overfit 10 examples, something in the forward/backward pipeline is broken
Check that requires_grad=True is set on all trainable parameters and that no unintended .detach() or .data access breaks the graph
Confirm the loss function’s input/output shapes match what it expects (e.g., raw logits vs. probabilities) — a mismatched shape often trains “successfully” while learning nothing meaningful
Watch for a loss that goes to NaN a few steps in — almost always a learning rate too high or a numerical instability like log(0) inside the loss
Compare gradients computed by autograd against a manual finite-difference estimate on a small subset of parameters, as a last-resort sanity check when a custom operation is suspected
If loss decreases initially then plateaus far above zero even on a tiny dataset, suspect a capacity or architecture issue rather than an optimization bug — backprop only computes correct gradients for the function the network defines, it can’t fix a network that’s structurally unable to represent the target relationship
Log the ratio of gradient norm to weight norm per layer, not just the raw gradient norm — a raw norm can look reasonable while still being far too large or small relative to the weights it’s updating, which the ratio exposes directly

Backprop Through Different Architectures

Feedforward networks: the case described above — chain rule applied once per layer, straightforwardly back through the stack
Convolutional networks: the same chain rule applies, but the local derivative at a conv layer involves a correlation operation with the flipped kernel, which is why frameworks implement a dedicated “conv2d backward” operation rather than reusing the dense-layer backward pass
Recurrent networks: backprop through time (BPTT) unrolls the recurrent layer across time steps and applies the chain rule across both depth and time simultaneously, which is why RNNs are especially prone to vanishing/exploding gradients over long sequences
Transformers: backprop flows through attention weights as well as the usual linear layers — the softmax inside attention has its own local derivative that must be chained through just like any other activation
Graph neural networks: the chain rule is applied through however many rounds of message-passing (neighbor aggregation) the network performs, with each round behaving like an additional depth-wise layer, which is why very deep message-passing schedules face the same vanishing-gradient pressure as very deep feedforward networks

Example

In a 5-layer network, an error at the output layer is mathematically traced back through layers 4, 3, 2, and 1 to determine how much each layer’s weights contributed to the mistake. Concretely: if layer 5’s output was too high, backprop computes exactly how much of that error is attributable to layer 4’s output via the chain rule, then how much of layer 4’s error is attributable to layer 3, and so on.

Each layer only needs the error signal handed to it from the layer after it, plus its own local derivative — never the full end-to-end formula spelled out across all five layers at once. That locality is precisely what keeps the algorithm computationally tractable at depth.

Referenced by

Dig deeper

Spot an error in this note?