tiny-vllm
A course and full source code for building a smaller version of vLLM from scratch in C++ and CUDA, covering model loading, CUDA kernels, KV caching, batching and PagedAttention.
tiny-vllm is both a working LLM inference engine and a course that walks through building it from scratch in C++ and CUDA, as a smaller sibling of vLLM. It is intended as a learning resource, usable independently or as a teaching resource, that derives the underlying ideas and math rather than only showing finished code.
The engine it builds loads a real model (Llama 3.2 1B Instruct) from Safetensors, runs a full forward pass split into prefill and decode with all computation implemented as CUDA kernels, and progressively adds a KV cache, static and then continuous batching, online softmax with FlashAttention-style computation, and PagedAttention. The course covers prerequisites such as floating-point formats and bfloat16, GPU and CPU memory, tokenization, embeddings, CUDA kernel engineering for embeddings, RMSNorm, RoPE, residual connections, and the column-major-to-row-major transposition trick, before reaching attention, grouped-query attention, and the batching and paging techniques themselves.
It is released under the Apache-2.0 license.