Skip to content

Pipeline Parallel#

concept

Pipeline parallelism (PP) splits the model by layers into consecutive stages, each on a different GPU or group of GPUs. Activations flow forward from stage to stage, and gradients flow backward. Only the tensors at stage boundaries cross the network, which makes PP the strategy best suited to slower, inter-node links.

The bubble problem#

A naive pipeline runs one batch through stage 1, then stage 2, and so on, so every stage sits idle most of the time. The fix is to cut the batch into m micro-batches and keep several in flight.

GPipe (all forwards, then all backwards), p = 4 stages, m = 4 micro-batches
stage 1 │F1 F2 F3 F4 -- -- -- -- -- -- B4 B3 B2 B1│
stage 2 │-- F1 F2 F3 F4 -- -- -- -- B4 B3 B2 B1 --│
stage 3 │-- -- F1 F2 F3 F4 -- -- B4 B3 B2 B1 -- --│
stage 4 │-- -- -- F1 F2 F3 F4 B4 B3 B2 B1 -- -- --│
         ↑ idle slots at the start and end = the "bubble"

The idle time relative to useful compute is approximately:

bubble overhead ≈ (p − 1) / m          p = pipeline stages, m = micro-batches

With p = 4 and m = 4, that is 75% overhead. With m = 32, about 9%. More micro-batches shrink the bubble, but each one is smaller and the global batch size caps how many you can have.

Schedules#

Schedule Idea Effect
GPipe All forwards, then all backwards Simple; holds activations for all m micro-batches
1F1B After warm-up, alternate one forward and one backward Same bubble as GPipe, but at most p micro-batches of activations in flight
Interleaved 1F1B Each GPU holds v non-contiguous chunks of layers Bubble shrinks to about (p − 1) / (v·m), at the cost of v× more point-to-point messages
Zero-bubble variants Split backward into input-grad and weight-grad, schedule weight-grad into idle slots Bubble approaches zero; more complex scheduling 
1F1B, p = 4, m = 8 (unit-time F and B; steady state alternates one F and one B)
stage 1 │F1 F2 F3 F4 -- -- -- B1 F5 B2 F6 B3 F7 B4 F8 B5 -- B6 -- B7 -- B8│
stage 2 │-- F1 F2 F3 -- -- B1 F4 B2 F5 B3 F6 B4 F7 B5 F8 B6 -- B7 -- B8 --│
stage 3 │-- -- F1 F2 -- B1 F3 B2 F4 B3 F5 B4 F6 B5 F7 B6 F8 B7 -- B8 -- --│
stage 4 │-- -- -- F1 B1 F2 B2 F3 B3 F4 B4 F5 B5 F6 B6 F7 B7 F8 B8 -- -- --│
         stage 1 never holds more than 4 micro-batches of activations

Communication#

  • Point-to-point send/recv of the activation tensor at each boundary: micro_batch × seq × hidden elements, forward and backward.
  • Messages are relatively small and frequent, so latency matters more than peak bandwidth.
  • Traffic is between neighbouring stages only, not all-to-all, so it maps well onto any fabric, including across racks.

Balancing stages#

The pipeline runs at the speed of its slowest stage. The embedding layer and the output layer (vocabulary projection) are heavier than average transformer layers, so the first and last stages are often given fewer transformer layers. An unbalanced pipeline looks like a busy GPU next to idle ones in DCGM metrics, with nothing wrong in the network.

Effect on the rest of the stack#

  • Control plane: stages should be placed so neighbouring stages are close; the whole pipeline must be gang-scheduled.
  • L7 fabric: modest bandwidth needs, but tail latency on each hop stalls the pipeline.
  • Failure blast radius: one failed stage stops the whole pipeline replica.

When to use#

  • The model does not fit even with TP across a full NVLink domain.
  • Inter-node bandwidth is too low for FSDP's all-gathers at the needed scale.
  • In inference, PP is used to span models across nodes when TP cannot, accepting extra latency per token.