vllm.cpp
A community-built, vLLM-alike inference engine written entirely in C++, with continuous batching, paged key-value cache and RadixAttention, aiming for the same throughput as vLLM with a much smaller install.
vllm.cpp mirrors vLLM's serving behavior and is benchmarked against it, but has no Python dependency: it runs on CUDA, CPU, Metal or Vulkan and registers 44 model architectures. It is an independent, unofficial community port, not affiliated with the vLLM project, the PyTorch Foundation or the Linux Foundation, brought by the team behind LocalAI.
The project moves quickly and warns that internals, CLI flags and server behavior can change between commits, though its versioned C ABI stays stable and grows only by appending zero-defaulted fields. Recent work adds multimodal input over HTTP for vision and audio models, more engine controls exposed through the C ABI, a native ROCm path for one quantization format, and CUDA support for a large mixture-of-experts model with its draft model.