LSTM (Long Short-Term Memory)

Definition: A type of RNN with a gating mechanism (input, forget, output gates) specifically designed to retain relevant information over long sequences and mitigate the vanishing gradient problem.

How It Works

Maintains a separate “cell state” that acts as a conveyor belt for long-term memory
Gates learn what to keep, forget, and output at each step, controlling information flow more deliberately than a plain RNN
At each timestep, the LSTM cell receives the current input, the previous hidden state, and the previous cell state, and produces a new hidden state and new cell state
The gates are themselves small neural network layers (sigmoid-activated), trained jointly with the rest of the network via Backpropagation through time
Unlike a vanilla RNN, which overwrites its entire hidden state every timestep, an LSTM’s cell state is modified only by addition and elementwise multiplication — never passed through a weight matrix and squashed by a nonlinearity at every single step — which is the structural reason it can preserve information across hundreds of timesteps rather than a handful
LSTMs were invented specifically to solve the recurrent version of the vanishing gradient problem — see Vanishing-Exploding Gradient for the general failure mode and the gating mechanism below for the fix that gives gradients a path through time that doesn’t require repeated multiplication by the same weight matrix

Under the Hood

Given input x_t, previous hidden state h_{t-1}, and previous cell state c_{t-1}, an LSTM cell computes:

f_t = sigmoid(W_f . [h_{t-1}, x_t] + b_f)   # forget gate: what to drop from cell state
i_t = sigmoid(W_i . [h_{t-1}, x_t] + b_i)   # input gate: what new info to add
c~_t = tanh(W_c . [h_{t-1}, x_t] + b_c)     # candidate values to add to cell state
c_t = f_t * c_{t-1} + i_t * c~_t            # updated cell state
o_t = sigmoid(W_o . [h_{t-1}, x_t] + b_o)   # output gate: what to expose as output
h_t = o_t * tanh(c_t)                       # updated hidden state

f_t (forget gate) outputs values between 0 and 1 for each cell state dimension — near 0 erases that piece of memory, near 1 keeps it fully
i_t (input gate) controls how much of the new candidate information c~_t gets written into the cell state
The cell state update c_t = f_t * c_{t-1} + i_t * c~_t is additive, not purely multiplicative through a nonlinearity at every step — this is the key structural difference from a vanilla RNN, and it’s what lets gradients flow backward through many timesteps without shrinking exponentially (mitigating, though not fully eliminating, the Vanishing-Exploding Gradient problem)
o_t (output gate) decides how much of the (squashed) cell state becomes the visible hidden state h_t, which is what gets passed to the next layer or timestep
The self-connected linear unit at the heart of the cell state is what Hochreiter and Schmidhuber’s original paper named the constant error carousel (CEC): it uses the identity function as its own activation and a fixed self-weight of 1.0, so in the absence of new gated input a gradient flowing backward through it is neither amplified nor attenuated — this is the literal mechanism behind “mitigating the vanishing gradient problem” above, not just a loose analogy
This doesn’t make LSTMs immune to vanishing or exploding gradients altogether — gradients still flow through the gate networks themselves, through the tanh squashing applied to the candidate values and the final output, and through however many layers are stacked, all of which can still misbehave; the CEC specifically protects the cell-state-to-cell-state path across time, not every path through the network

The LSTM Cell, Visualized

The equations above describe exactly one thing happening four times: three gate networks and one candidate network, all reading the same two inputs (the previous hidden state and the current input), feeding into two elementwise operations on the cell state, and one further elementwise operation to produce the hidden state. Laid out as a single timestep’s data flow:

Reading it left to right: the forget gate looks at the previous hidden state and current input and decides how much of the existing cell state to keep; the input gate and candidate layer together decide what new information to write in; those two updates combine additively into the new cell state; and the output gate filters a squashed copy of that new cell state into the hidden state that actually leaves the cell. Every gated multiply in this diagram is elementwise, not a matrix product — each gate scales every dimension of the cell state independently rather than mixing dimensions together the way the W . projections do.

Unrolled Through Time — LSTM vs. Plain RNN

A single cell diagram shows one timestep in isolation; what actually distinguishes an LSTM from a plain RNN only becomes visible once several timesteps are unrolled and it’s clear what gets passed from one to the next. A plain RNN has exactly one channel of memory — the hidden state — that must simultaneously carry short-term detail and whatever long-term signal the network needs later. An LSTM splits that job across two channels instead:

The plain RNN’s single arrow per step is precisely the bottleneck the vanishing gradient problem exploits: information from timestep 1 has to survive being repeatedly squashed through a nonlinearity and multiplied by the same recurrent weight matrix, once per intervening timestep, before it can influence timestep 100. The LSTM’s second arrow — the cell state — is the constant error carousel from the previous section made visible: it gives long-term information a path forward that doesn’t require surviving that same repeated squashing at every step.

A Worked Numeric Example

Plugging in actual numbers makes the gate arithmetic concrete. Suppose — for illustration only, since real LSTMs use vector-valued states across dozens or hundreds of dimensions rather than scalars — a single-unit LSTM cell has previous hidden state h_{t-1} = 0.3, previous cell state c_{t-1} = 0.5, and current input x_t = 1.0, and the four gate pre-activations (each normally W . [h_{t-1}, x_t] + b, collapsed here to one weighted sum for simplicity) evaluate to:

Gate	Pre-activation	Activation	Result
Forget (`f_t`)	0.8(0.3) + 0.2(1.0) − 0.1 = 0.54	sigmoid	0.632
Input (`i_t`)	0.6(0.3) + 0.9(1.0) + 0.1 = 1.18	sigmoid	0.765
Candidate (`c~_t`)	0.4(0.3) + 0.7(1.0) = 0.82	tanh	0.675
Output (`o_t`)	0.5(0.3) + 0.3(1.0) + 0.2 = 0.65	sigmoid	0.657

The cell state update is then c_t = f_t * c_{t-1} + i_t * c~_t = 0.632(0.5) + 0.765(0.675) ≈ 0.316 + 0.516 ≈ 0.832, and the hidden state is h_t = o_t * tanh(c_t) = 0.657 * tanh(0.832) ≈ 0.657 * 0.681 ≈ 0.447. Notice the cell state grew from 0.5 to 0.832: the forget gate kept most of the old memory (a factor of 0.632) while the input gate wrote in a substantial new contribution (0.765 of the candidate’s 0.675) — exactly the kind of controlled, additive update that keeps the cell state from being wholesale overwritten every step the way a plain RNN’s hidden state is. The runnable sandbox in the Code Example section below reproduces this exact calculation.

History

Introduced by Sepp Hochreiter and Jurgen Schmidhuber in their 1997 paper “Long Short-Term Memory,” which directly targeted the vanishing gradient problem that made vanilla RNNs unable to learn dependencies spanning more than roughly 10 timesteps. The forget gate — now considered essential — was actually added later, in a 2000 follow-up by Gers, Schmidhuber, and Cummins; the original 1997 architecture lacked it. LSTMs saw relatively little use until GPUs and larger datasets made training deep sequence models practical in the early-to-mid 2010s, at which point they became the dominant architecture for speech recognition, machine translation, and language modeling until transformers overtook them starting around 2017.

The 1997 paper appeared in Neural Computation (volume 9, issue 8, pages 1735-1780) and built its argument around what it named the constant error carousel — the self-connected linear unit described in Under the Hood above — demonstrating it could bridge minimal time lags in excess of 1000 discrete timesteps, versus the roughly 10-timestep ceiling that made plain RNNs of the era impractical for most real sequence-learning tasks. The 2000 forget-gate paper, “Learning to Forget: Continual Prediction with LSTM” (Gers, Schmidhuber, and Cummins; presented at ICANN 1999, published in Neural Computation volume 12, issue 10, pages 2451-2471), was motivated by a specific failure mode: on continuous, unsegmented input streams — unlike the neatly pre-segmented sequences the 1997 paper tested on — the original architecture’s cell state could grow unboundedly and saturate its own gates, since nothing in the design ever told it to actively release old memory. Hochreiter’s own group revisited the architecture nearly three decades later with “xLSTM” (2024), adding exponential gating and a matrix-valued memory in an attempt to close some of the performance gap with transformers on language-modeling benchmarks — a sign the core gating idea remains an active research direction rather than a purely historical one.

Variants

Vanilla LSTM — the standard three-gate (forget, input, output) architecture described above
Coupled Input-Forget Gate (CIFG) — a common simplification that ties the input gate to the forget gate (i_t = 1 - f_t) instead of learning them separately, on the reasoning that a cell should write new information roughly in proportion to how much old information it’s discarding; large empirical comparisons have found it cuts a gate’s worth of parameters with little measured performance loss
Peephole LSTM — lets the gates also look at the cell state directly (not just the hidden state), which can improve precise timing-dependent tasks
GRU (Gated Recurrent Unit) — a simplified alternative with only two gates (reset and update) and no separate cell state; fewer parameters, often trains faster, and performs comparably to LSTM on many tasks
Bidirectional LSTM — runs two LSTMs over the sequence, one forward and one backward, then concatenates their hidden states — useful when the full sequence is available at inference time (e.g., text classification) rather than being generated left-to-right
Stacked / Deep LSTM — multiple LSTM layers stacked so the hidden state sequence of one layer feeds as the input sequence to the next, increasing representational capacity at the cost of more compute and a higher risk of vanishing/exploding gradients through depth as well as time
ConvLSTM — replaces the fully-connected gate computations with convolutions, suited to spatiotemporal data like video frames
Attention-augmented LSTM — adds an attention mechanism over the sequence of encoder hidden states so a decoder isn’t limited to a single fixed-size summary vector — the architecture that bridged LSTMs and transformers before self-attention made recurrence unnecessary entirely

Why It Matters

Was the dominant architecture for sequence modeling (translation, speech recognition) for years before transformers displaced it
Still used in some latency-sensitive or streaming applications where transformers’ full-context attention is overkill
Its gating mechanism directly inspired later architectures — GRUs simplified it, and even some components of transformer variants borrow the idea of learned, data-dependent information flow
LSTMs process sequences step-by-step, which gives them natural support for streaming/online inference (you don’t need the whole sequence upfront) and constant memory per step, unlike a transformer’s attention over the full context window
Constant memory and constant compute per timestep make LSTMs cheap to run on hardware far smaller than what serving a transformer typically requires, which is part of why they persist in embedded and mobile deployments years after losing the research spotlight

Common Pitfalls

Assuming LSTMs handle arbitrarily long context as well as transformers do — they still degrade over very long sequences, just less than plain RNNs
Using LSTMs for new large-scale NLP projects today when a transformer would train faster (parallelizable) and perform better
Training LSTMs without gradient clipping — despite the gating mechanism reducing vanishing gradients, exploding gradients through time are still a real risk
Forgetting to reset or properly detach hidden/cell state between unrelated sequences (e.g., between batches during training) when using stateful LSTM configurations, which leaks information across independent examples
Not accounting for the sequential (non-parallelizable across time) nature of LSTMs when estimating training time — unlike transformers, an LSTM can’t process all timesteps of a sequence simultaneously, so training and inference scale linearly with sequence length in wall-clock time
Initializing the forget gate bias to zero — in practice, initializing it to a positive value (e.g., 1) is a well-known trick that helps the network default to “remember” early in training rather than forgetting everything
Underestimating parameter count — an LSTM layer has roughly 4x the parameters of a vanilla RNN of the same hidden size (one weight matrix per gate, plus the candidate values), which affects both memory footprint and overfitting risk on small datasets
Padding variable-length sequences without masking, which lets the LSTM process meaningless padding tokens as if they were real input and pollutes both the hidden state and the gradient
Assuming the cell state fully solves vanishing gradients for arbitrarily deep stacked LSTMs — the constant error carousel protects the cell-state-to-cell-state path specifically; gradients through stacked layers, through the gate networks themselves, and through the tanh nonlinearities can still vanish or explode, which is exactly why gradient clipping remains standard practice regardless of gating
Assuming bidirectional LSTMs can be dropped into an autoregressive generation setup — a bidirectional LSTM requires seeing the entire sequence, including future timesteps, which is structurally incompatible with generating output one step at a time

Best Practices

Prefer a transformer or a modern state-space model (e.g., Mamba) for new large-scale sequence modeling projects; reach for LSTM/GRU when you specifically need constant-memory streaming inference or are working with limited compute/data
Initialize the forget gate bias to a small positive value (commonly 1.0) to bias the network toward retaining information early in training
Apply gradient clipping (e.g., clip norm to 1.0-5.0) as standard practice when training any recurrent architecture
Use a bidirectional LSTM when the full input sequence is available at inference (classification, tagging) rather than being generated autoregressively
Consider a GRU first if compute or parameter count is constrained — it’s often a nearly-free swap for LSTM with similar performance and fewer parameters
Use padded-sequence utilities (e.g., pack_padded_sequence in PyTorch) so variable-length batches don’t waste compute or corrupt gradients on padding tokens
Normalize or scale input features before feeding them into the LSTM — like most gradient-based models, it trains more reliably on well-conditioned inputs than on raw, wildly-scaled features
Log gradient norms during LSTM training the same way you would for any deep or recurrent network — gating reduces vanishing gradients, it doesn’t remove the need to monitor for them (see Vanishing-Exploding Gradient)
Benchmark a CIFG or GRU variant before defaulting to a full three-gate LSTM on a new project — the parameter and compute savings are often close to free in measured accuracy terms, and worth skipping only when a task is already known to need the extra gate capacity
On very deep stacked LSTMs, consider residual connections between layers, the way GNMT’s 8-layer encoder does (see Real-World Example) — depth-wise vanishing gradients are a separate problem from the time-wise one the gating mechanism already addresses

Code Example

import torch
import torch.nn as nn

class SequenceClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim=128, hidden_dim=256, num_classes=2):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers=2,
                             batch_first=True, bidirectional=True)
        self.classifier = nn.Linear(hidden_dim * 2, num_classes)  # *2 for bidirectional

    def forward(self, x):
        embedded = self.embedding(x)                     # [batch, seq_len, embed_dim]
        outputs, (h_n, c_n) = self.lstm(embedded)         # h_n: final hidden states
        # Concatenate final forward and backward hidden states from the last layer
        final_hidden = torch.cat((h_n[-2], h_n[-1]), dim=1)
        return self.classifier(final_hidden)

model = SequenceClassifier(vocab_size=10000)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

The gate equations implemented directly, for a single timestep, to make the “Under the Hood” formulas concrete:

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def lstm_cell_step(x_t, h_prev, c_prev, W_f, W_i, W_c, W_o, b_f, b_i, b_c, b_o):
    combined = np.concatenate([h_prev, x_t])
    f_t = sigmoid(W_f @ combined + b_f)        # forget gate
    i_t = sigmoid(W_i @ combined + b_i)         # input gate
    c_candidate = np.tanh(W_c @ combined + b_c) # candidate cell values
    c_t = f_t * c_prev + i_t * c_candidate      # updated cell state
    o_t = sigmoid(W_o @ combined + b_o)         # output gate
    h_t = o_t * np.tanh(c_t)                    # updated hidden state
    return h_t, c_t

The same single-timestep arithmetic, runnable directly in this page as plain JavaScript, reproducing the numbers from “A Worked Numeric Example” above:

Comparison

Architecture	Introduced	Handles long-range dependencies	Parallelizable over sequence	Params per layer	Typical modern use
Vanilla RNN	~1990	Poorly (vanishing gradients)	No	Low	Rarely used today, teaching example
LSTM	1997	Well (via gating + cell state)	No (sequential)	High (4x RNN’s gate weights)	Streaming/low-latency sequence tasks
GRU	2014	Well, slightly less capacity than LSTM	No (sequential)	Medium (3x RNN’s gate weights)	Lighter-weight alternative to LSTM
Transformer	2017	Very well (direct attention to any position)	Yes	Depends on width/depth	Default for most modern NLP/sequence tasks

Real-World Example

Google’s Neural Machine Translation system (GNMT, 2016). Wu et al.’s production system stacked 8 LSTM layers in the encoder and 8 more in the decoder, connected the attention mechanism from the decoder’s bottom layer to the encoder’s top layer to preserve parallelism during training, and added residual connections between layers specifically to keep gradients well-behaved through a stack that deep. It cut translation errors by 60-87% relative to Google’s previous phrase-based system and served production Google Translate traffic before transformer-based models eventually replaced it — a publicly documented illustration both of what a heavily-engineered LSTM stack could achieve and of why depth-wise gradient management (the residual connections) mattered even with gating already handling the time-wise problem.

Apple’s on-device QuickType keyboard. Apple’s Machine Learning Journal has documented using bidirectional LSTMs for on-device language identification and word prediction that feed QuickType and Smart Replies — a real instance of the streaming, resource-constrained niche described below, running entirely on-device for latency and privacy reasons rather than calling a server. iOS 16 later moved QuickType’s core prediction engine to a quantized on-device transformer, which is itself a useful data point: even in exactly the low-latency, on-device niche LSTMs are supposed to fit best, transformers have kept encroaching as quantization and efficient-attention techniques closed the efficiency gap.

A keyboard app’s next-word prediction or an on-device voice assistant’s streaming speech recognizer is a realistic modern use case for an LSTM: the input arrives incrementally (one keystroke or audio frame at a time), a response is needed with very low latency, and the device has limited memory and no guarantee of network connectivity to run a large transformer server-side. An LSTM processes each new input in constant time and constant memory, updating its hidden and cell state incrementally, without needing to re-run attention over an ever-growing context window the way a transformer would. This is precisely the streaming, resource-constrained niche where LSTMs remain a reasonable engineering choice rather than a historical footnote.

FAQ

Why did transformers replace LSTMs for most NLP tasks? Two main reasons: transformers process all sequence positions in parallel during training (LSTMs are inherently sequential, since each timestep depends on the previous one), which makes transformers dramatically faster to train on modern hardware; and self-attention lets every position directly attend to every other position, avoiding the information bottleneck of squeezing an entire sequence’s history through a fixed-size hidden state.

Do LSTMs still have a place today? Yes, in specific niches: streaming/real-time applications with strict latency and memory constraints (constant memory per step vs. a transformer’s context window), edge devices, time-series forecasting where sequences are short and data is limited, and as components inside larger hybrid architectures.

What’s the actual difference between LSTM and GRU? GRU merges the forget and input gates into a single “update gate” and eliminates the separate cell state, using only the hidden state. This gives GRU roughly 25% fewer parameters per layer than LSTM for the same hidden size, faster training, and comparable accuracy on many (though not all) tasks — LSTM’s extra capacity can still win out on tasks needing more precise long-term memory control.

How does the cell state actually solve vanishing gradients? In a vanilla RNN, the hidden state is repeatedly passed through a nonlinearity (tanh) and multiplied by a weight matrix at every timestep, so gradients flowing backward through many steps get repeatedly shrunk (or occasionally blown up) — an exponential effect over long sequences. The LSTM’s cell state update is largely additive (c_t = f_t * c_{t-1} + i_t * c~_t), so a gradient can flow backward through the f_t * c_{t-1} term across many timesteps with much less multiplicative shrinkage, as long as the forget gate stays close to 1 for the relevant timesteps.

Can LSTMs be used for anything other than text? Yes — any sequential or time-series data: sensor readings, stock prices, audio waveforms, DNA sequences, user click streams. The “long short-term memory” framing (retaining relevant signal while discarding noise over time) is domain-agnostic; NLP just happened to be where LSTMs saw the most visible adoption during their peak years.

Common Interview Questions

Why does the LSTM’s cell state help with vanishing gradients when the hidden state alone doesn’t? (The cell state update is largely additive rather than repeatedly passed through a squashing nonlinearity and weight matrix, so gradients shrink far less across many timesteps.)
What’s the computational complexity difference between an LSTM and a transformer for a sequence of length n? (An LSTM processes a sequence in O(n) sequential steps that can’t be parallelized across time; a transformer’s self-attention is O(n²) in compute but fully parallelizable across the sequence.)
When would you still reach for an LSTM over a transformer today? (Streaming/low-latency inference with constant memory per step, small datasets where a lighter-weight model avoids overfitting, or resource-constrained edge deployment.)
What role does the forget gate bias initialization trick play, and why does it work? (Initializing it to a positive value, commonly 1.0, biases the sigmoid output toward 1 early in training, so the network defaults to retaining information rather than randomly discarding it before it has learned anything useful about what’s actually irrelevant.)
Why can’t a plain RNN just use a bigger hidden state to fix vanishing gradients? (Hidden state size controls capacity, not gradient flow — a larger hidden state still gets passed through the same nonlinearity and multiplied by a weight matrix every timestep, so the exponential shrinkage/growth problem is unchanged regardless of dimensionality.)
What’s the difference between the constant error carousel and the gates that surround it? (The CEC is the passive, identity-activated self-loop that lets a gradient survive a timestep unattenuated; the gates are the separately-trained sigmoid layers that decide, at each step, how much is allowed onto or off of that loop — the memory mechanism and the control mechanism are structurally distinct pieces.)

Terminology Notes

The “long” in Long Short-Term Memory refers to how far back in time information can survive; the “short-term” refers to what’s being preserved — the hidden/cell state, which is still fundamentally a working, updatable memory rather than permanent storage. The name describes a memory that is short-term in kind but can be made long in duration, not a contradiction.
“Constant error carousel” (CEC), “cell state,” and “memory cell” are used near-interchangeably in different papers and courses to refer to the same additive c_t pathway described in Under the Hood — CEC is the more precise, original-paper term for the specific self-connected linear unit, while “cell state” is the term almost universally used in modern frameworks and tutorials.
“Gating” in LSTM/GRU literature specifically means an elementwise multiplication by a learned, sigmoid-bounded (0 to 1) mask — not gating in the hardware-logic-gate sense, though the metaphor of a valve that opens and closes is intentional and accurate.

Example

Older Google Translate versions used LSTM-based sequence-to-sequence models before switching to transformer-based architectures. In that setup, an “encoder” LSTM would read an entire source-language sentence word by word, compressing its meaning into a final hidden and cell state; a separate “decoder” LSTM would then generate the target-language sentence one word at a time, conditioned on that compressed representation (later versions added an attention mechanism so the decoder could look back at all encoder hidden states, not just the final one — a precursor to the attention mechanism that transformers later made central rather than supplementary). The core limitation that pushed the field toward transformers was exactly this bottleneck: forcing an entire sentence’s meaning through one fixed-size vector loses information, especially for long sentences, and processing everything sequentially made both training and inference slow to parallelize.

Referenced by

Dig deeper

Spot an error in this note?