Learn LLM Inference Serving by Shipping One Service
A 10-week, 30-minutes-a-day roadmap that has an engineer build, instrument and optimize one OpenAI-compatible LLM inference service, covering vLLM, SGLang, quantization, speculative decoding and cost-aware routing.
This roadmap is built around fifty 30-minute sessions over ten weeks, five sessions a week plus two buffer days, and every session feeds a single growing artifact: an OpenAI-compatible inference service deployed on a rented GPU, instrumented, load tested past 1000 concurrent requests, optimized with quantization and speculative decoding, fronted by a cost-aware router, and published as a reproducible benchmark. It targets engineers comfortable with Python, transformer architecture and the command line.
By the end, a learner has a serving stack configured and tuned themselves, Grafana dashboards showing time-to-first-token, inter-token latency, throughput, queue depth and cost per request, a reproducible load-test harness, benchmarked variants across FP16, FP8, INT4, speculative decoding and KV eviction, a cost/latency/quality router with per-request token budgeting, and a published benchmark writeup with pinned versions.
The roadmap orders topics around a roofline model, so each later optimization is framed as a move on the same compute-versus-memory-bandwidth plot.