Nano-vLLM
A lightweight, from-scratch reimplementation of vLLM in about 1,200 lines of Python, offering comparable offline inference throughput with prefix caching, tensor parallelism and CUDA graphs.
Nano-vLLM is a compact reimplementation of the vLLM inference engine, written from scratch to stay small and readable while matching vLLM's offline inference speed.
It implements an optimization suite including prefix caching, tensor parallelism, Torch compilation and CUDA graph capture, and exposes an API similar to vLLM's LLM.generate interface for loading a local model and sampling completions with configurable parameters.
A bundled benchmark script compares its throughput against vLLM; in one test configuration on a single GPU, Nano-vLLM slightly outperformed vLLM's own throughput on the same workload.