Inference Engines

conceptstub

vLLM, SGLang, TensorRT-LLM and what they need from the platform.

Planned content

• KV cache, paged attention, continuous batching
• TP within a node for latency
• Prefill/decode disaggregation and KV transfer