candle-vllm
A Rust inference and serving platform for local large language models with an OpenAI-compatible API, offering native flash attention, aggressive key-value cache compression and multi-GPU support on both CUDA and Metal.
candle-vllm is an efficient, easy-to-use platform for running and serving local large language models, exposing an OpenAI-compatible API server alongside a built-in chat-style web interface. It supports native flash attention, FlashInfer, CUDA graphs, continuous batching and prefix caching for performance, and its TurboQuant key-value cache compression extends usable context length several times over with a reported minimal quality loss.
The same codebase runs on CUDA under Linux and on Metal under macOS, supports multi-process and multi-threaded tensor parallelism across multiple GPUs and nodes, and can load a model directly from a Hugging Face model identifier or from a local directory, including single or split-shard GGUF files. A trait-based architecture is meant to let new model pipelines be added quickly.
It installs with a one-line script, as Docker images, or built from source with configurable CUDA or Metal features, and is written in Rust.