nanoPD
A from-scratch inference engine that separates the prefill and decode phases of LLM generation onto dedicated GPUs, avoiding the interference between the two phases that limits throughput when they share hardware.
nanoPD implements a custom paged KV cache, chunked prefill, a hand-written CUDA paged-attention kernel, multi-GPU KV cache transfer, and an adaptive router that decides per request whether to run the prefill-heavy or decode-heavy phase together or apart, based on an analytical cost model.
All three serving strategies it implements and benchmarks, collocated, disaggregated and adaptive, are measured against a Poisson-arrival load generator with configurable arrival rate, workload distribution, warmup and drain phases. The router's cost-model parameters, such as prefill and decode speed, interference coefficient and inter-GPU bandwidth, are measured live on the actual hardware at startup rather than assumed in advance.
It is written in Python with a custom CUDA kernel.