Efficient LLM Inference & Serving Systems
A hands-on, first-principles course explaining why LLM inference is memory-bound and how techniques such as batching, KV-cache management, quantization, and parallelism address it, with GPU labs building up to vLLM and SGLang.
This course explains, from first principles, why generating one token from a large language model barely uses a GPU's compute: decoding must read every model weight from memory to do very little math with each one, so the expensive compute units sit mostly idle waiting for data. It frames nearly every serving technique as a different way to reclaim that idle capacity, working module by module from batching, through key-value cache management and PagedAttention, quantization, tensor and pipeline parallelism, long-context serving techniques such as eviction and sparse attention, and speculative or chunked decoding.
Later modules cover structured and constrained decoding, tool-use and agentic serving with cross-turn prefix reuse, the surrounding framework, API, and observability layer a real deployment needs, and an evaluation methodology for telling whether a change actually helped rather than just moving the bottleneck elsewhere. Every result in the course is reproducible, with GPU labs run on open models.