Vanishing-Exploding Gradient

Vanishing/Exploding Gradient

Definition: A training problem in deep networks where gradients become extremely small (vanishing) or extremely large (exploding) as they propagate backward through many layers, stalling or destabilizing training.

How It Works

Backpropagation multiplies gradients layer by layer via the chain rule
If each layer’s gradient contribution is consistently < 1, the product shrinks toward zero across many layers (vanishing)
If consistently > 1, the product grows explosively (exploding)
The problem is specific to depth of the computation graph the gradient must traverse, not to the total parameter count — a wide but shallow network is far less prone to it than a narrow but very deep one
The gradient of the loss with respect to an early layer’s weights is a product of many terms — one per layer between that weight and the output — each term itself a product of a weight matrix and an activation function’s derivative
With n layers, that product has roughly n multiplicative factors; even a mild consistent bias away from 1.0 per factor compounds exponentially with depth, which is why the problem gets dramatically worse as networks get deeper
The same mechanism affects the forward pass too: activations can shrink toward zero or blow up as they propagate forward through many layers, which is part of why normalization layers address both forward and backward stability simultaneously
Both vanishing and exploding gradients ultimately manifest as the same practical symptom — the model fails to learn — which is why distinguishing them requires actually inspecting gradient magnitudes rather than guessing from the loss curve alone
Recurrent networks feel this most acutely because the same weight matrix is reapplied at every timestep; LSTMs were invented specifically to solve this recurrent version of the problem, not as a coincidental overlap between the two topics

Under the Hood

Consider the derivative of the sigmoid activation: its maximum value is 0.25, reached only at x=0, and it approaches 0 for large positive or negative inputs. Chaining even a modest number of sigmoid layers means multiplying together many numbers that are at best 0.25 and often much smaller — after 10 layers, the gradient can already be scaled down by a factor near 0.25^10, roughly one ten-thousandth
Weight initialization compounds this: if weight matrices have eigenvalues consistently above 1, the forward activations and backward gradients both grow layer over layer (exploding); if consistently below 1, both shrink (vanishing) — this is why principled initialization schemes (Xavier/Glorot, He initialization) explicitly try to keep the variance of activations and gradients roughly constant across layers
In recurrent networks, the same weight matrix is applied at every timestep, so the chain-rule product for a sequence of length T involves the same matrix raised to roughly the T-th power — a small deviation of its largest eigenvalue from 1 compounds far more severely than in a feedforward net of comparable depth, which is why plain RNNs struggle even more with long sequences than deep feedforward nets do with many layers
Batch normalization mitigates vanishing/exploding gradients indirectly by re-centering and re-scaling layer inputs at every step, which keeps activations (and therefore the local gradient magnitudes flowing through them) in a well-behaved range regardless of how earlier layers’ weights evolve — see Batch Normalization
Residual connections change the chain-rule product itself: instead of the gradient at layer L depending purely on a product of factors down to layer l, the identity shortcut adds a term of exactly 1 at each block, so the total gradient is a sum that includes an unattenuated path back to every earlier layer — this is why 100+ layer ResNets train at all
LSTMs solve the recurrent version of the problem with a “cell state” highway regulated by gates: the forget gate can be close to 1, letting gradients pass through a timestep almost unchanged instead of being multiplied by a small recurrent weight matrix at every step
Layer normalization (used pervasively in transformers) normalizes across the feature dimension per example rather than across the batch, which keeps it effective at any batch size, including batch size 1 at inference — unlike batch normalization, which needs batch statistics
The LSTM’s fix, concretely: its cell state is updated through addition and elementwise gating rather than through a repeated matrix multiplication and nonlinearity, so the backward path through the cell state alone doesn’t accumulate the same exponential shrinkage — Hochreiter and Schmidhuber’s original paper named this self-connected, identity-activated pathway the constant error carousel; see LSTM (Long Short-Term Memory) for the full gate-by-gate mechanics

The Math, Concretely

For a network with L layers, the gradient of the loss with respect to layer 1’s weights involves a product of roughly L Jacobian terms: dLoss/dW1 = dLoss/dA_L * dA_L/dA_(L-1) * ... * dA_2/dA_1 * dA_1/dW1
Each dA_k/dA_(k-1) term is approximately W_k * f'(z_k) — a weight matrix times an activation derivative. If the typical magnitude of these terms is r, the accumulated gradient scales roughly like r^L
r < 1 (e.g., saturated sigmoid regions, small weight init): gradient shrinks toward 0 exponentially in L — vanishing
r > 1 (e.g., large weight init, unbounded activations): gradient grows toward infinity exponentially in L — exploding
This is exactly why 10-layer and 100-layer networks behave so differently under the same activation/init choices — the exponent, not just the base, is what changes
For recurrent networks, the equivalent quantity is the spectral radius (largest absolute eigenvalue) of the recurrent weight matrix — values consistently below 1 predict vanishing gradients over long sequences, values above 1 predict exploding gradients, and this is the theoretical basis for the LSTM/GRU gating fix
Concretely, for r = 0.9 — a mild, easy-to-miss-in-review shrinkage — the accumulated factor r^L is already about 0.35 at 10 layers, roughly 0.005 at 50 layers, and roughly 0.00003 at 100 layers: depth alone turns a barely-noticeable per-layer effect into a training-breaking one
The same arithmetic runs in reverse for r = 1.1: roughly 2.6 at 10 layers, roughly 117 at 50 layers, and roughly 13,780 at 100 layers — which is why even a small, easy-to-dismiss overshoot in initialization scale reliably turns into NaN losses once a network gets deep enough

Gradient Flow, Visualized

Seeing two networks of identical depth diverge under only a small change in the per-layer factor r makes the exponential effect concrete. Backward pass through 4 layers, with r = 0.5 on one path and r = 1.5 on the other, starting from the same gradient magnitude at the output:

Both paths start from the same gradient magnitude at the output and take the same number of steps backward; only the per-layer factor differs, and after just 4 layers the two paths are already almost 100x apart (0.063 versus 5.063). Extrapolate either path out to the 50 or 100 layers common in modern networks and the gap isn’t measured in multiples anymore — it’s measured in orders of magnitude, which is exactly why unmitigated deep networks either stall (the earliest layers’ gradients underflow toward numerical zero) or diverge to NaN (the earliest layers’ gradients overflow) long before training finishes even a single epoch.

Mitigations by Failure Mode

Not every standard fix targets both failure modes equally — some are specifically anti-vanishing, some specifically anti-exploding, and a few genuinely help both because they attack the shared root cause: a chain-rule product not held near magnitude 1.

Mitigation	Addresses vanishing	Addresses exploding	Why
ReLU-family activations	Yes — primary target	Indirectly	Derivative is exactly 1 for positive inputs, so it doesn’t itself shrink a gradient the way a saturating sigmoid/tanh does; unbounded positive output means it does little to cap growth
Careful weight initialization (Xavier/Glorot, He)	Yes	Yes	Explicitly derived to keep activation and gradient variance close to constant layer over layer, targeting both directions of the problem at once
Batch normalization	Yes	Yes	Re-centers and re-scales layer inputs at every step, keeping activations (and local gradients) in a well-behaved range regardless of how earlier layers’ weights evolve
Residual / skip connections	Yes — primary target	Partially	Adds an unattenuated additive gradient path around each block; doesn’t cap the multiplicative path, so it doesn’t prevent explosion on its own
Gradient clipping	No	Yes — primary target	Rescales an already-computed gradient after the fact; does nothing to amplify a gradient that’s already near zero, only to cap one that’s too large
LSTM / GRU gating	Yes — primary target	Partially	The constant error carousel gives gradients an unattenuated path through time; the gates themselves and any stacked depth can still explode, which is why clipping is still standard practice on top of gating

This is also why the fixes are so often layered together in practice rather than used one at a time — He initialization plus ReLU plus residual connections plus gradient clipping is a common combination precisely because between them they cover both failure modes from multiple angles instead of relying on any single technique to do all the work.

Comparison

Aspect	Vanishing Gradient	Exploding Gradient
Gradient magnitude	Shrinks toward 0 across layers	Grows toward infinity/NaN across layers
Visible symptom	Loss plateaus, early layers barely update	Loss oscillates wildly or becomes NaN
Common cause	Saturating activations (Sigmoid, Tanh), deep/long recurrent chains	Poor weight init, high learning rate, deep/long recurrent chains
Typical fix	ReLU-family activations, residual connections, better init, LSTM/GRU gating	Gradient clipping, lower learning rate, better init
Most affected architectures	Deep feedforward nets, plain RNNs	Deep feedforward nets, plain RNNs
Detection method	Per-layer gradient norm shrinks toward zero when logged	Per-layer gradient norm grows rapidly, or loss turns to NaN
Typical onset during training	Gradual — loss plateaus over many steps before it’s obvious	Often sudden — a single bad batch can trigger it

Both problems stem from the same root cause — a chain-rule product that isn’t kept near magnitude 1 — which is why fixes like normalization and careful initialization tend to address both simultaneously, while clipping only patches the exploding side.

Why It Matters

Was the main obstacle to training very deep networks before solutions like ReLU, batch normalization, residual connections, and LSTMs/attention were developed
Explains why plain RNNs struggle with long sequences and why “deep” learning wasn’t practical until these fixes existed
Understanding this failure mode is essential for debugging training runs — a loss curve that flatlines immediately or explodes to NaN within a few steps is almost always one of these two problems, not a data or architecture-design issue in the usual sense
Directly motivated some of the most influential architectural innovations in deep learning history: residual/skip connections (ResNet), gating mechanisms (LSTM, GRU), and normalization layers all exist largely to keep gradients well-behaved across depth
A recurring theme across deep learning history: nearly every major architectural breakthrough since 2012 can be read, at least partly, as a new way of keeping gradients flowing cleanly across more layers or longer sequences
Understanding it prevents wasted debugging time — engineers unfamiliar with the failure mode often spend days tuning data pipelines or hyperparameters unrelated to the actual cause
The two failure modes are asymmetric in urgency: exploding gradients usually announce themselves immediately (NaN losses, training crashes), while vanishing gradients fail silently — training merely looks slow or stuck rather than broken, which is why the latter is often misdiagnosed for far longer

Common Interview Questions

Why does ReLU help with vanishing gradients? Its derivative is exactly 1 for any positive input (versus a sigmoid’s derivative, which is at most 0.25 and shrinks toward 0 away from the origin), so it doesn’t itself multiplicatively shrink the gradient as it passes backward through an active unit
What is “dying ReLU,” and is it related? Dying ReLU is a separate failure mode where a unit’s input is always negative, so its gradient is always exactly 0 and it stops learning entirely; Leaky ReLU and GELU address this by allowing a small or smooth gradient for negative inputs
Why does weight initialization matter so much in deep networks? Because the chain-rule product amplifies any consistent bias in per-layer gradient magnitude exponentially with depth, so even a modest initialization mismatch compounds into severe vanishing or exploding behavior in very deep networks
How do residual connections mathematically prevent vanishing gradients? The identity shortcut adds a constant-1 term to the local derivative at each block, so the gradient with respect to an early layer includes an unattenuated additive path back through every skip connection, in addition to the usual multiplicative path
Why is gradient clipping applied to the norm rather than individual values in most modern setups? Norm clipping rescales the entire gradient vector when its magnitude exceeds a threshold, preserving its direction; value clipping truncates each coordinate independently, which can distort the direction of the update and hurt convergence
Why do residual networks still use batch or layer normalization if skip connections already address vanishing gradients? Skip connections address the gradient path specifically; normalization addresses the forward-pass activation distributions feeding into that path. The two problems can occur somewhat independently, so modern architectures address them with separate, complementary mechanisms rather than treating either as a full substitute for the other

History

Sepp Hochreiter’s 1991 diploma thesis first formally identified the vanishing gradient problem in deep and recurrent networks, though it wasn’t widely recognized until Bengio et al.’s 1994 paper analyzed it in the context of RNNs
Glorot and Bengio’s 2010 paper on Xavier initialization gave a principled derivation for weight initialization scale as a function of layer fan-in/fan-out, directly targeting the vanishing/exploding problem at the start of training
LSTMs (Hochreiter & Schmidhuber, 1997) were explicitly designed with a gated “cell state” that allows gradients to flow across many timesteps largely unimpeded, directly targeting the vanishing gradient problem in recurrent nets
ReLU activations (popularized by AlexNet, 2012) largely replaced sigmoid/tanh in feedforward nets because their derivative is exactly 1 for all positive inputs, avoiding the multiplicative shrinkage that saturating activations cause
Residual connections (ResNet, 2015) let gradients skip directly across blocks via identity shortcuts, enabling stable training of networks with over 100 layers — something that was essentially impossible with plain deep stacks before this
Hochreiter’s 1991 thesis was written in German at TU Munich under advisor Jurgen Schmidhuber — the same pairing that would go on to write the 1997 LSTM paper — which is a large part of why the vanishing gradient problem and its most famous architectural fix trace back to the same two researchers
Bengio, Simard, and Frasconi’s 1994 paper, “Learning Long-Term Dependencies with Gradient Descent is Difficult,” appeared in IEEE Transactions on Neural Networks and gave the problem its first widely-cited formal treatment in the recurrent setting, proving that storing information robustly over long time lags and remaining sensitive to gradient-based updates are, in a precise sense, in tension with each other
He et al.’s 2015 ResNet paper documented the “degradation problem” directly: plain (non-residual) networks with more layers showed higher training error than shallower versions, not just higher test error — evidence the failure was an optimization difficulty tied to depth, not simply overfitting, though the paper itself is cautious about attributing the effect purely to vanishing gradients rather than to the general difficulty of optimizing very deep stacks of nonlinear transformations

Common Pitfalls

Diagnosing training that “just doesn’t improve” as a data problem when it’s actually a vanishing gradient issue
Not using gradient clipping for exploding gradients, letting a few bad batches destroy an otherwise-good training run
Stacking many Sigmoid or Tanh layers in a deep feedforward network without considering that each one caps its own gradient contribution below 1
Using a very high learning rate on a deep or recurrent architecture, which makes an already-unstable gradient signal even more likely to explode into NaN losses
Copy-pasting a learning rate and initialization scheme that worked for a shallow network onto a much deeper one without re-validating that it’s still appropriate for the new depth
Ignoring gradient norms during training — logging the L2 norm of gradients per layer is a cheap diagnostic that catches vanishing/exploding behavior long before the loss curve makes it obvious
Assuming residual connections or normalization alone fully “solve” the problem for arbitrarily deep networks — very deep or very long-sequence models can still need clipping, careful init, and learning rate warmup
Confusing exploding gradients with a bad loss function — a spike in loss right before it becomes NaN is a classic exploding-gradient signature, not necessarily evidence the objective is mis-specified
Overlooking that mixed-precision training (float16) narrows the representable numeric range, making both underflow (compounding vanishing gradients to literal zero) and overflow (compounding exploding gradients to infinity) more likely unless loss scaling is used
Applying gradient clipping by value instead of by norm when the intent was to preserve gradient direction — clipping each element independently distorts the direction of the update, while norm clipping rescales the whole vector and preserves it
Treating the fix for one failure mode as a fix for both — gradient clipping, for instance, does nothing for a vanishing gradient, and residual connections alone don’t cap an exploding one (see Mitigations by Failure Mode above)
Debugging a suspected vanishing/exploding gradient issue by staring at the loss curve alone instead of logging actual per-layer gradient norms, which is the one diagnostic that directly distinguishes this failure mode from every other kind of training problem
Assuming a model that trained fine at 12 layers will scale cleanly to 48 layers with the same initialization, activation, and learning rate — the exponential relationship between depth and accumulated gradient factor means small, previously-invisible per-layer biases become dominant well before 4x the depth is reached

Real-World Example

Plain RNNs on long documents: an RNN-based language model trained on paragraphs longer than a few dozen tokens historically lost the ability to use information from early in the sequence, motivating the switch to LSTMs and later attention-based transformers
Very deep CNNs before ResNet: early attempts to stack 20+ plain convolutional layers often trained worse than shallower networks, not because of overfitting but because gradients couldn’t reach the earliest layers — this “degradation problem” was the direct motivation for residual connections
Transformer pretraining at scale: large transformer training runs routinely apply gradient clipping and learning-rate warmup as standard configuration, since a single exploding-gradient spike deep into a multi-week training run can waste enormous compute cost if unrecovered
Speech and time-series models: recurrent models processing long audio or sensor sequences face an especially severe version of the problem, since sequence length can run into the thousands of timesteps, which is part of why attention-based and convolutional alternatives have displaced plain RNNs in much of this space too
Reinforcement learning policy networks: deep policy and value networks used in Reinforcement Learning are susceptible to the same instability, which is one reason PPO and similar algorithms constrain how far a policy update is allowed to move in a single step
OPT-175B (Meta, 2022): Meta’s public training logbook for its 175-billion-parameter OPT model documented dozens of manual restarts, some triggered by loss divergence rather than hardware failure; the team found that lowering the gradient clipping threshold from 1.0 to 0.3 partway through training, alongside other interventions, was necessary to keep the run stable — a rare, publicly visible look at exploding-gradient firefighting at the largest scale
BLOOM-176B (BigScience, 2022): The open BLOOM project documented recurring loss spikes tied to its mixed-precision training setup, growing more frequent as training progressed, and addressed the issue partly through an embedding normalization technique — a concrete illustration of the Common Pitfalls point above about mixed-precision training narrowing the representable numeric range

Code Example

Gradient clipping in PyTorch — the standard fix for exploding gradients:

import torch
import torch.nn as nn

model = MyDeepNetwork()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for inputs, targets in train_loader:
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    loss.backward()

    # Clip gradients to a max L2 norm of 1.0 before the update
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

    optimizer.step()

    # Diagnostic: log gradient norm per layer to catch vanishing/exploding early
    total_norm = sum(
        p.grad.norm(2) ** 2 for p in model.parameters() if p.grad is not None
    ) ** 0.5
    if total_norm < 1e-6 or total_norm > 1e3:
        print(f"Warning: gradient norm = {total_norm:.6f}")

The same shrink-versus-blow-up dynamic, simulated directly in this page as plain JavaScript — a starting gradient magnitude of 1.0 multiplied repeatedly by 0.5 (vanishing) versus 1.5 (exploding) across 20 simulated layers:

Best Practices

Prefer ReLU-family activations (ReLU, GELU, Leaky ReLU) over Sigmoid/Tanh in hidden layers of deep networks, reserving saturating activations for output layers where they’re semantically needed (e.g., Sigmoid for binary probabilities)
Use He initialization with ReLU-family activations and Xavier/Glorot initialization with Tanh/Sigmoid — matching the init scheme to the activation function keeps early-training gradient magnitudes stable
Add residual/skip connections in any network deeper than roughly 20-30 layers
When switching to a deeper architecture variant of an existing model, re-tune learning rate and warmup schedule rather than assuming the shallower model’s settings still apply
Apply gradient clipping by default when training RNNs, LSTMs, or transformers, especially early in training or with a high learning rate
Use learning rate warmup for very deep or transformer-based architectures, since gradients are least stable in the first few hundred training steps before batch norm/layer norm statistics settle
Monitor per-layer gradient norms during training, not just the aggregate loss — a healthy network has roughly similar gradient magnitudes across layers, not ones that decay or grow by orders of magnitude from output to input
Checkpoint frequently during long training runs so an unrecovered exploding-gradient spike costs minutes of lost progress rather than days
Prefer normalization layers (batch norm, layer norm) as a default in any architecture deeper than a handful of layers, rather than adding them reactively after training instability appears
When debugging a stalled model, plot gradient norm by layer index as a first diagnostic step before touching data, architecture, or hyperparameters — it immediately distinguishes vanishing/exploding gradients from other failure modes
Use mixed-precision training with automatic loss scaling (e.g., PyTorch’s torch.cuda.amp) rather than plain float16, since unscaled float16 gradients are far more prone to underflow/overflow than float32
Treat vanishing and exploding gradients as two separate diagnoses requiring two separate fixes rather than a single problem with a single solution — see Mitigations by Failure Mode above before reaching for whichever fix happens to be most familiar
When scaling a working architecture to significantly more layers, re-validate gradient norms at the new depth before assuming the old initialization and learning rate still apply, rather than discovering the exponential depth effect from a failed training run
On recurrent architectures specifically, prefer LSTM/GRU gating as the first line of defense against vanishing gradients over long sequences, and layer gradient clipping on top for the exploding side — the two techniques address different halves of the same underlying problem (see LSTM (Long Short-Term Memory))

FAQ

Does gradient clipping fix vanishing gradients too? No — clipping only caps large gradients; it does nothing to amplify a gradient that’s already near zero. Vanishing gradients need architectural or initialization fixes, not clipping
Why don’t transformers suffer from vanishing gradients the way RNNs do? Attention lets any token attend directly to any other token in one step, so the “path length” for gradient flow between two tokens is short and constant, unlike an RNN where it grows with sequence length; residual connections around every attention/feedforward block add further shortcut paths for gradients
Is exploding gradient always a sign of a bug? Not necessarily a bug in code — it’s often just a mismatch between learning rate, initialization, and architecture depth, which is why it’s addressed with clipping and tuning rather than treated as an error to fix in the model definition
How do I tell vanishing from exploding gradients in practice? Log gradient norms per layer: a vanishing gradient shows norms shrinking by orders of magnitude from output layers to input layers while the loss stalls; an exploding gradient shows norms growing rapidly across steps, usually followed by the loss spiking or turning to NaN
Does a larger batch size help? It can reduce gradient noise and make exploding gradients less erratic, but it doesn’t address the underlying multiplicative depth problem — it’s a training-stability aid, not a fix for vanishing/exploding gradients themselves
Do modern architectures still need to worry about this at all? Yes, in attenuated form — even transformers with residual connections and layer normalization can show training instability at very large depth or scale without careful initialization and warmup, which is why large-model training recipes still treat this as a first-class concern
What’s a quick sanity check before a full training run? Run a single forward and backward pass on a small batch and print the gradient norm of the first and last layers — if they differ by several orders of magnitude before training even starts, the initialization or architecture needs adjustment first

Terminology Notes

“Vanishing gradient” and “exploding gradient” describe the same underlying chain-rule mechanism producing opposite outcomes; papers and textbooks often discuss them jointly as “the gradient problem” in deep and recurrent networks
The term “deep” in deep learning became practically meaningful only once these problems had workable mitigations — a “deep” network in the pre-2010s sense might have had 3-5 layers, versus 50-100+ today
The LSTM’s “constant error carousel” is the named mechanism most directly built to counter vanishing gradients specifically in the recurrent setting — worth knowing as the concrete architectural answer to the abstract problem described throughout this page

Example

A 50-layer network using only Sigmoid activations trains painfully slowly because gradients shrink toward zero before reaching the earliest layers — solved by switching to ReLU and adding residual (skip) connections. In practice, logging gradient norms during training would show near-zero gradients in the first few layers and healthy gradients near the output; after the fix, gradient norms stay within a comparable order of magnitude across all 50 layers, and the early layers finally start learning.

Referenced by

Dig deeper

Spot an error in this note?