Skip to content

Expert Parallel (MoE)#

concept

Mixture-of-Experts (MoE) models replace the dense MLP in each layer with many smaller experts, and a router sends each token to only a few of them (top-k). Total parameters grow, but compute per token stays close to a smaller dense model. Expert parallelism (EP) places different experts on different GPUs.

Token flow through one MoE layer#

flowchart LR
  T[Tokens on each GPU] --> R[Router: pick top-k experts per token]
  R --> D[[All-to-all: dispatch]]
  D --> E1[Experts on GPU 0]
  D --> E2[Experts on GPU 1]
  D --> E3[Experts on GPU N-1]
  E1 --> C[[All-to-all: combine]]
  E2 --> C
  E3 --> C
  C --> O[Tokens back on their home GPU]
  1. The router chooses the top-k experts for every token.
  2. Dispatch all-to-all: each GPU sends each token to the GPUs that host its chosen experts.
  3. Experts process the tokens they received.
  4. Combine all-to-all: results return to the token's original GPU and are weighted and summed.

Forward needs two all-to-alls per MoE layer, and backward needs two more.

Why EP is the hardest pattern on the fabric#

Property Consequence
All-to-all, not ring or tree Every GPU talks to every other GPU in the EP group at once; there is no neighbour structure to exploit
Data-dependent volumes The router decides how many tokens go where; traffic is uneven from step to step
Load imbalance A popular expert receives more tokens; its GPU and its incoming links become hotspots
On the critical path Experts cannot start until dispatch completes
Bursty, synchronized Every GPU sends at the same moment; incast and congestion control (PFC/ECN) are stressed

The result is that MoE training performance is often decided by tail latency of all-to-all, not average bandwidth.

Controls that matter#

  • Capacity factor: caps tokens per expert; overflow tokens are dropped or re-routed. Higher capacity means less dropping but more memory and traffic.
  • Load-balancing loss or bias terms: train the router to spread tokens evenly.
  • Expert placement: put experts so that most traffic stays inside an NVLink domain; cross-node traffic goes over rails.
  • Optimized all-to-all kernels: libraries such as DeepEP use NVLink for intra-node and RDMA for inter-node legs, and overlap communication with compute.

How EP combines with other dimensions#

  • Experts are usually sharded across the DP group: GPUs that are data-parallel replicas for the dense parts hold different experts for the MoE parts.
  • Attention layers still use TP/CP/DP as usual; only the MoE blocks use EP.
  • EP for inference (wide-EP serving) spreads experts over many GPUs to serve large MoE models efficiently, and puts the same all-to-all on the serving fabric.

Effect on the rest of the stack#

  • L5: all-to-all efficiency in NCCL or specialized libraries.
  • L6/L7: low-latency RDMA, adaptive routing or well-tuned ECN/DCQCN, and enough bisection bandwidth; rail-optimized topologies help.
  • L8: NVLink domain size sets how much all-to-all stays off the fabric.
  • Observability: per-expert token counts and all-to-all duration are the key signals.