Skip to content

L4 · Compute Runtime & Kernels#

conceptstub

What it is#

The GPU programming platform and libraries that execute each operation: the CUDA or ROCm runtime, math libraries (cuBLAS, cuDNN), compilers (XLA, TorchInductor, Triton) and hand-tuned kernels such as fused attention.

Why it matters#

Two clusters with identical hardware can differ several-fold in throughput depending on kernels and numeric precision. This layer decides how close each GPU gets to its peak compute and memory bandwidth, and its runtime version must be compatible with the host driver.

In this module#

The software that turns framework operations into GPU work, and decides how well each GPU is used.

Planned content

  • CUDA runtime and driver compatibility: what ships in the image vs on the host
  • Math libraries: cuBLAS, cuDNN, CUTLASS, and their precision modes (BF16, FP8, FP4)
  • Compilers: XLA, TorchInductor, Triton, TensorRT
  • Attention and fused kernels (FlashAttention and successors) and their memory effect
  • Profiling kernels: Nsight Systems / Nsight Compute, tensor-core utilization
  • How kernel choices interact with parallelism (L3) and memory (L9)