Skip to content

L3 · Parallelism Strategy#

conceptstub

What it is#

The decision of how to split one job across many GPUs. You can replicate the model and split the data (data parallel, FSDP), split each layer's matrices (tensor parallel), split the model into stages (pipeline parallel), split the sequence (context parallel), or spread experts across GPUs (expert parallel). It is configured in the framework, but it is a design choice in its own right.

Why it matters#

Parallelism is needed when a model does not fit in one GPU's memory, or would take too long on one GPU. Each strategy creates a different traffic pattern: blocking all-reduces, overlapped gradient exchange, point-to-point transfers, or all-to-all. That pattern dictates what the communication libraries, fabric, interconnect and scheduler must provide. This is why it is a design layer: choose it first, and the layers below follow. The theory is in Concepts → Parallelism.

In this module#

Configuring parallelism in practice, and how it constrains the layers below.

Planned content

  • Choosing TP/PP/DP/CP/EP for a given model and cluster
  • Framework configuration examples (Megatron, FSDP, DeepSpeed)
  • What the choice demands from scheduler, fabric, and GPU topology
  • Theory lives in Concepts → Parallelism