LLM Inference Capacity Planning Handbook

This handbook addresses a gap between introductory explanations of concepts like the KV cache and line-by-line vLLM source walkthroughs: given a certain amount of GPU hardware, how much traffic a serving system can actually sustain, and what latency it can commit to. It presents capacity planning as a three-layer model: a physical layer computing theoretical limits with a roofline model for prefill and decode and memory limits from the KV cache, a system layer taking the minimum of a compute wall, a latency-SLA wall and a KV-cache-slot wall to get a single node's sustainable load, and a queueing layer using queueing theory to translate that sustainable load into real P50 and P95 latencies under actual traffic.

It argues that common capacity estimates are wrong in consistent ways, such as using closed-loop, saturated-concurrency benchmarks as if they were a capacity number, or estimating throughput from decode speed alone while treating prefill as free. Later chapters cover simulation methods and their error sources, load-testing methodology, a full worked example sizing infrastructure for a support system serving several million daily active users, and cluster-level topics such as parallelism strategies and network communication volume.

It is written for inference platform engineers, technical managers making purchasing decisions, and researchers, and states explicitly that it does not cover CUDA kernel-level tuning.