The Nine-Layer Model

concept

The AI infrastructure stack as nine layers in three planes, with a parallel storage data path and a control plane that spans everything. It is a co-design and teaching model: its value is the method of designing and validating the layers together, not the layer list itself. See related models for how it compares with other public views of the stack.

Workload plane

Software runtime

Data path

L1ApplicationTraining, inference, RAG, agents (MCP), HPCL2Frameworks & Serving EnginesPyTorch, JAX, Megatron, vLLM, SGLang, llm-dL3Parallelism StrategyDP/FSDP, TP, PP, CP, EP (design layer)L4Compute Runtime & KernelsCUDA/ROCm, cuDNN/cuBLAS, XLA, Triton kernelsL5Communication LibrariesNCCL/RCCL, MPI, NIXLL6TransportRDMA verbs, libfabric, GPUDirect RDMAL7Network FabricInfiniBand, RoCE v2, Ultra Ethernet, EFA, SlingshotL8Node InterconnectNVLink, NVSwitch, UALink, PCIe, NUMAL9Accelerator & MemoryNVIDIA H/B-series, AMD MI-series, TPU, HBMSStorage Data Path (parallel)Parallel FS, NVMe cache, GPUDirect StorageControl Planespans all layers

• Kubernetes
• Slurm
• Kueue
• Volcano
• GPU/Network Operators
• DRA
• CNI
• CSI

Cross-cutting: observability · security · reliability · power and cooling

Click any layer, the control plane or the cross-cutting band to open its module.

Design choices in this model

Each choice answers an objection an expert reviewer would raise against a simpler "nine boxes in a column" picture.

Choice	Why
Parallelism is a design layer	It is implemented by the framework, but it is the decision that sets the traffic, memory and placement every layer below must serve.
Orchestration is a control plane, not a layer	NCCL never calls Kubernetes. The scheduler shapes placement, attaches devices and networks, then steps out of the data path.
Kernels and runtime are a layer	CUDA, math libraries, compilers and attention kernels decide how well each GPU is used, separate from how GPUs talk to each other.
Storage is a parallel data path	Data loading and checkpointing decide whether GPUs wait. Storage is designed alongside the fabric, not after it.
The data path branches	Inside a node, GPUs talk directly over NVLink. Only traffic between nodes goes PCIe → NIC → fabric.
Some concerns cut across everything	Observability, security, reliability, and power and cooling apply to every layer, and often set the real limit at scale.

Where other technologies fit

New AI technologies map onto the existing layers rather than needing new ones.

Technology	Where it fits
Agents and MCP (tool and data access)	L1 application; MCP servers run as ordinary pods
Inference gateways, model routing, prefix-aware load balancing (e.g. llm-d)	L2 serving engines, with routing policy in the control plane
KV-cache offload and tiering	L2 serving engines, using L9 memory and the storage path
Quantization (FP8, FP4, INT4)	L4 kernels and L9 hardware support
Non-NVIDIA accelerators (AMD ROCm/RCCL, TPU/XLA)	Same layers: L4 runtime, L5 libraries, L9 hardware
Ultra Ethernet, UALink	L7 fabric and L8 scale-up interconnect
Vector databases	Storage path, serving L1 RAG and agent applications
MLOps: experiment tracking, model registry, pipelines	L1 application tooling, artifacts on the storage path
Virtualization (KubeVirt, vGPU, GPU passthrough)	Control plane plus the host side of L6–L9
Evaluation, guardrails, confidential computing	L1 application and the security cross-cutting concern

Data path vs control plane

flowchart LR
  subgraph CP["Control plane (Kubernetes)"]
    S[Scheduler / Kueue] --> K[Kubelet]
    K --> DP[Device plugin / DRA]
    K --> CNI[Calico + Multus]
    K --> CSI[CSI driver]
  end
  subgraph DPATH["Data path (bypasses Kubernetes)"]
    G1[GPU] -- "NVLink (intra-node)" --> G2[GPU]
    G1 -- "PCIe / GPUDirect (inter-node)" --> NIC[RDMA NIC]
    NIC -- IB / RoCE --> F[(Fabric)]
    ST[(Storage)] -- GPUDirect Storage --> G1
  end
  DP -. allocates .-> G1
  CNI -. attaches .-> NIC
  CSI -. mounts .-> ST

Once a pod is running, NCCL traffic flows GPU → NVLink inside a node, or GPU → PCIe → NIC → fabric between nodes, without touching kube-proxy or the primary CNI. The control plane decides where the data path is built, not how the bytes move. From pod spec to data path walks through it step by step, and life of a training step follows the bytes.

Where each layer runs

Category	What lives there	Layers
Inside containers	AI apps, frameworks, CUDA runtime libraries, NCCL, user-space RDMA libraries (libibverbs, libfabric)	L1–L5, user-space part of L6
Control plane	Scheduler, Kueue/Volcano, Slurm, GPU and Network Operators, device plugins, DRA, CNI (Calico, Multus), CSI	Spans all layers
Outside Kubernetes	Host kernel drivers, NICs, switches, NVLink/NVSwitch, GPUs, storage systems, firmware	L6–L9, storage

Delivered by Kubernetes, executed by the host

The GPU Operator and Network Operator can deploy the GPU driver, MOFED/DOCA, nvidia-peermem and the container toolkit as DaemonSets. Kubernetes manages their lifecycle, but the code runs in the host kernel, not in a pod's process space.

Version alignment across the boundary

The CUDA runtime in the image (L4) must be supported by the host driver, and the user-space verbs and libfabric in the image (L6) must match the host's kernel modules. Mismatches across this boundary are a classic failure point.

How the control plane influences topology

The control plane shapes placement at L8 and L9 rather than just counting devices:

• GPU Feature Discovery / Node Feature Discovery publish GPU model, memory, MIG capability and topology labels.
• Kubelet Topology Manager aligns GPUs and NICs to the same NUMA node / PCIe root.
• MIG partitioning is driven by the GPU Operator's MIG manager.
• Dynamic Resource Allocation (DRA) expresses structured device claims, including multi-node NVLink compute domains.

This is the mechanism behind keeping tensor-parallel groups inside one NVLink domain.