CPU Pipelining

Definition: An instruction execution technique that overlaps the CPU Instruction Cycle’s stages across multiple instructions concurrently, like an assembly line, to increase instruction throughput.

How It Works

Divides instruction processing into discrete stages: Fetch, Decode, Execute, Memory, Writeback (the classic 5-stage RISC pipeline; real cores use anywhere from a handful to 20+ stages)
While instruction 1 is in Execute, instruction 2 is in Decode, and instruction 3 is in Fetch, all in the same clock cycle, on separate hardware for each stage
Once the pipeline is full (after an initial fill latency equal to the stage count), a new instruction completes every clock cycle in the ideal case, even though any single instruction still takes multiple cycles start to finish
Structural Hazards: two instructions need the same hardware resource in the same cycle, for example both wanting the same memory port; resolved by duplicating resources or stalling one instruction
Data Hazards: an instruction needs a result that a still-in-flight earlier instruction hasn’t produced yet; resolved by forwarding/bypassing (routing a result directly from one stage to another before it’s officially written back) or, when forwarding isn’t possible, stalling
Control Hazards: a branch’s outcome isn’t known until it reaches Execute, but Fetch needs to know the next PC immediately; resolved with branch prediction (guess and proceed speculatively) plus a pipeline flush to discard wrongly fetched instructions if the guess was wrong
Superscalar pipelines duplicate stages to issue and execute more than one instruction per cycle; deeper pipelines (more, shorter stages) allow higher clock frequency at the cost of a larger misprediction penalty when a flush happens

Under the Hood

Given: four independent instructions I1-I4 enter a 5-stage pipeline one per cycle, with no hazards between them. Step: by cycle 5, I1 is finishing Writeback while I2, I3, and I4 are simultaneously in Memory, Execute, and Decode. Answer: after the initial 5-cycle fill, one instruction completes every single cycle instead of every 5, a near 5x throughput improvement over a non-pipelined design for this instruction mix.

Given: ADD R1, R2, R3 (R1 = R2 + R3) is immediately followed by SUB R4, R1, R5 (R4 = R1 - R5), a true data dependency on R1. Step 1: without forwarding, SUB would need to wait until ADD’s Writeback completes and R1 is actually readable from the register file, costing several stall cycles. Step 2: with forwarding, the ALU result from ADD’s Execute stage is routed directly into SUB’s Execute stage input in the very next cycle, bypassing the register file entirely. Answer: forwarding eliminates the stall for this specific hazard pattern (Execute-stage result needed by the immediately following instruction’s Execute stage), though a load-then-use hazard (needing a value from Memory, not Execute) typically still forces at least one stall cycle even with forwarding, since the value isn’t ready until one stage later.

Given: BEQ R1, R2, target (branch if equal) at instruction 10 of a loop, with a 15-stage pipeline and a branch predictor that gets it wrong. Step 1: the predictor guesses “not taken” and Fetch keeps pulling instructions 11, 12, 13… down the fall-through path speculatively. Step 2: several cycles later, Execute resolves the actual branch condition and discovers it should have been taken. Step 3: every speculatively fetched instruction behind the branch (11 through whatever has entered the pipeline since) is flushed, and Fetch restarts at the correct target. Answer: a misprediction this deep into the pipeline wastes roughly one cycle per pipeline stage between Fetch and the stage where the branch resolves, which is exactly why branch-predictor accuracy matters more, not less, as pipelines get deeper.

without forwarding (load-use hazard):
  LW  R1, 0(R2)   F  D  E  M  W
  ADD R3, R1, R4     F  D  *stall* E  M  W
                            (R1 not ready until LW's M stage completes)

with forwarding still present, one stall remains:
  the value forwards from LW's Memory stage to ADD's Execute stage,
  but ADD's Decode has to wait one cycle for it to exist at all

Why It Matters

Significantly boosts CPU instructions-per-cycle (IPC) without increasing clock frequency, which is why pipelining, not just faster clocks, drove much of the performance growth in CPU design through the 1990s and 2000s
Understanding hazards explains real, measurable code behavior: dependent instruction chains run slower than independent ones even at identical instruction counts, and branch-heavy code is disproportionately sensitive to prediction accuracy
Compilers exploit pipeline behavior directly: instruction scheduling reorders independent instructions between a load and its use specifically to fill the stall cycles a hazard would otherwise waste

Common Pitfalls

Pipeline flushes: branch mispredictions force the CPU to discard every speculatively fetched/decoded instruction behind the mispredicted branch, wasting a number of cycles equal to the pipeline depth up to that stage, which is why deeper pipelines (Intel Pentium 4’s ~20 stages) suffer larger misprediction penalties than shallower ones
Assuming pipelining gives a free N-times speedup for an N-stage pipeline: hazards, stalls, and flushes mean real IPC is always below the theoretical ideal, often well below it for branch-heavy or dependency-heavy code
Writing code with long dependency chains (a = f(a) repeated) that gives the pipeline nothing independent to overlap, effectively serializing execution despite pipelined hardware
Forgetting that not all hazards are solvable by forwarding: a load followed immediately by an instruction using the loaded value still stalls, since the loaded data isn’t available until the Memory stage completes, one stage later than an ALU result would be
Blaming “the CPU” for slow code when the real cause is compiler instruction scheduling: a poorly scheduled instruction stream leaves hazards unfilled that a better schedule would have hidden behind independent work

Comparison

Hazard type	Cause	Typical fix
Structural	Shared hardware resource	Duplicate the resource, or stall
Data	Dependent operand not yet produced	Forwarding/bypassing, or stall
Control	Branch outcome unknown at Fetch time	Branch prediction plus flush on misprediction

Design	Stages	Clock speed potential	Misprediction penalty
Shallow pipeline	~5	Lower	Small
Deep pipeline	15-20+	Higher	Large
Non-pipelined	1 (whole cycle)	N/A	None (no speculation happens)

Example

The classic 5-stage MIPS pipeline (Fetch, Decode, Execute, Memory, Writeback) is the textbook reference design nearly every architecture course teaches. Modern x86-64 cores go far further: Intel’s Skylake/Golden Cove-class cores use pipelines around 14-19 stages deep with sophisticated branch predictors specifically to keep misprediction penalties manageable despite that depth. ARM’s high-performance cores (Cortex-X, Apple’s M-series) similarly use deep out-of-order pipelines, while ARM’s small in-order cores (Cortex-A55) use much shallower pipelines to save power at the cost of peak IPC.

Given: the infamous Pentium 4 “Prescott” design pushed to roughly 31 pipeline stages chasing higher clock speed. Step 1: the deeper pipeline let Prescott clock significantly higher than its predecessors at the same process node. Step 2: but every branch misprediction now flushed roughly twice as many in-flight instructions as the prior generation’s pipeline, and real-world code has enough branches that this cost outweighed the clock speed gain for many workloads. Answer: Prescott became the textbook cautionary example of over-deepening a pipeline; Intel’s subsequent Core architecture deliberately reduced pipeline depth back down and focused on IPC instead, a design lesson that still shapes how CPU architects balance depth against misprediction cost today.

Referenced by

Dig deeper

Spot an error in this note?