Skip to content

// AI infrastructure on Kubernetes

The AI Stack, Layer by Layer#

An open knowledge base for building GPU platforms on Kubernetes: from the AI workload, through parallelism and NCCL, down to RDMA fabrics, NVLink and silicon. Every layer comes with concepts, design decisions, runnable implementation, and tests.

ai-factory — kubectl
mars@ai-factory:~$ kubectl get layers --stack=ai-infra -o wide
LAYER  NAME                           K8S-ROLE      EXAMPLES
L1     Application                    in-container  Training, inference, RAG, agents (MCP), HPC
L2     Frameworks & Serving Engines   in-container  PyTorch, JAX, Megatron, vLLM, SGLang, llm-d
L3     Parallelism Strategy           in-container  DP/FSDP, TP, PP, CP, EP (design layer)
L4     Compute Runtime & Kernels      in-container  CUDA/ROCm, cuDNN/cuBLAS, XLA, Triton kernels
L5     Communication Libraries        in-container  NCCL/RCCL, MPI, NIXL
L6     Transport                      host          RDMA verbs, libfabric, GPUDirect RDMA
L7     Network Fabric                 host          InfiniBand, RoCE v2, Ultra Ethernet, EFA, Slingshot
L8     Node Interconnect              host          NVLink, NVSwitch, UALink, PCIe, NUMA
L9     Accelerator & Memory           host          NVIDIA H/B-series, AMD MI-series, TPU, HBM
S      Storage Data Path              host          Parallel FS, NVMe cache, GPUDirect Storage
CP     Control Plane                  kubernetes    Kubernetes, Slurm, Kueue, Volcano, GPU/Network Operators, DRA, CNI, CSI
mars@ai-factory:~$ █

Project status

Concepts: complete. Implementation of the layer modules and lab testing on the Proxmox lab: in progress. GPU validation: planned. Each page's badge shows its own status.

The stack map#

Nine layers in three planes, the storage data path, and the control plane beside them. Click any part to open its module.

Start here#

  • By goal

    Guided, step-by-step paths: fine-tune a model, build a multi-tenant GPU cluster, wire RDMA into pods.

    Paths

  • By layer

    Nine layers in three planes, the storage data path, and the control plane that spans them, plus observability, security and tenancy.

    Layers

  • By workload

    Pre-training, fine-tuning, RL post-training, inference, RAG, agents: which parallelism fits and which layers each one stresses.

    Workloads

  • Concepts first

    The nine-layer model, parallelism strategies, and hardware/software co-design.

    Concepts

The stack matrix#

How much each layer matters to each workload. Rows link to the workload, columns and cells link to the layer.

workload \ layerL1L2L3L4L5L6L7L8L9SCP
Pre-training■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
Full fine-tuning■■■■■■■■■■■■■■■■■■■■■■■■■■
PEFT / LoRA■■■■■■■■■··■■■■■■
RL post-training■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
Online inference■■■■■■■■■■■■·■■■■■■■■■■■
Disaggregated inference■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
RAG■■■■■·■····■■■■■■
Agentic AI (MCP)■■■■■■■■■···■■■■■■■■
Batch inference■■■■■■···■■■■■■■
Notebooks / dev■■·■····■■■■■
·low■moderate■■high■■■critical

Principle#

Co-design, not component tuning

Optimal GPU performance cannot be achieved by tuning GPUs, networking, Kubernetes, or AI frameworks independently. The model's parallelism strategy, GPU topology, NCCL communication patterns, RDMA fabric, scheduler placement and storage throughput must be designed and validated together.

Maturity#

Every page carries a badge so you know what has been proven.