Data Parallel, ZeRO and FSDP

concept

Data parallelism (DP) keeps a full copy of the model on every GPU and gives each GPU a different slice of the batch. It is the simplest strategy, the most widely used, and the outermost dimension in almost every large job.

Plain data parallel (DDP)

sequenceDiagram
  participant G0 as GPU 0 (batch shard 0)
  participant G1 as GPU 1 (batch shard 1)
  participant G2 as GPU N-1 (batch shard N-1)
  Note over G0,G2: forward pass on local shard (no communication)
  Note over G0,G2: backward pass computes local gradients
  Note over G0,G2: all-reduce gradient buckets across all GPUs (starts during backward)
  Note over G0,G2: every GPU now holds the averaged gradient
  Note over G0,G2: identical optimizer step on every replica

1. Each GPU runs forward and backward on its own micro-batch.
2. Gradients are all-reduced so every replica holds the average.
3. Every replica applies the same optimizer step and stays identical.

PyTorch DDP groups gradients into buckets and starts the all-reduce for a bucket as soon as backward has produced it. Communication therefore overlaps with the rest of backward, which is why DP tolerates slower inter-node links.

Communication cost

With a ring all-reduce over N GPUs, each GPU sends and receives about 2 × (N−1)/N × S bytes per step, where S is the gradient size. For a 7B model in BF16, S ≈ 14 GB, so each GPU moves roughly 28 GB per step regardless of N.

Bus bandwidth

nccl-tests reports busbw = algbw × 2(N−1)/N for all-reduce. It is the number to compare against link speed, because it is independent of N.

Limits of plain DP

Every GPU holds the full 16 bytes/parameter of training state. DDP alone stops working once the model no longer fits in one GPU, at roughly 3–4B parameters on an 80 GB GPU with Adam, before activations.

ZeRO and FSDP

ZeRO (DeepSpeed) and FSDP (PyTorch) keep the DP structure but shard the training state across the DP group, so each GPU holds only 1/N of it.

Stage	Sharded across N GPUs	Memory per GPU (Ψ = parameters)	Extra communication vs DDP
DDP	nothing	16Ψ	baseline (2Ψ)
ZeRO-1	optimizer state	4Ψ + 12Ψ/N	none
ZeRO-2	+ gradients	2Ψ + 14Ψ/N	none
ZeRO-3 / FSDP full shard	+ parameters	16Ψ/N	≈ 1.5× (3Ψ)

With full sharding, a layer's parameters exist on a GPU only while that layer runs:

1. All-gather the layer's parameter shards before its forward.
2. Free them, then all-gather again before its backward.
3. Reduce-scatter the gradients so each GPU keeps only its shard.

Frameworks prefetch the next layer's all-gather while the current layer computes, so the cost is hidden as long as the link is fast enough.

70B model, full shard

16 × 70e9 / 64 GPUs ≈ 17.5 GB of state per GPU on 64 GPUs, which leaves room for activations on 80 GB H100s. On 8 GPUs it would be 140 GB per GPU, which does not fit.

HSDP: shard inside, replicate outside

Hybrid sharding (HSDP) shards within a group (usually one node, over NVLink) and replicates across groups (over the fabric). The chatty all-gathers stay on NVLink, and only a DDP-style gradient all-reduce crosses nodes.

What DP demands from the infrastructure

Layer	Requirement
L5 NCCL	Efficient all-reduce, all-gather, reduce-scatter; ring or tree algorithm selection
L6/L7 Transport and fabric	Sustained bandwidth; GPUDirect RDMA so gradients skip host memory
Control plane	Gang scheduling: one missing replica stalls every step
Storage	Enough read throughput to feed N copies of the data loader

When to use

• Plain DDP: the model fits on one GPU and you want more throughput.
• FSDP / ZeRO-3: the model does not fit, and you prefer not to restructure it for TP/PP. This is the default for full fine-tuning up to tens of billions of parameters.
• HSDP: multi-node FSDP where inter-node bandwidth is the constraint.
• As the outer dimension: in every hybrid job, DP replicates whatever TP/PP/CP unit sits inside it.