Fox
A local LLM server, wrapping llama.cpp, built as a drop-in replacement for Ollama that reuses a shared prompt's KV cache across concurrent requests instead of reprocessing it each time.
Fox is a local large language model server written in Rust around llama.cpp, built as a drop-in replacement for Ollama and compatible with both the Ollama and OpenAI APIs on one port. Its advantage over plain llama.cpp shows up specifically when multiple requests share a prompt: instead of each request prefilling that shared prefix again, they reuse each other's KV cache, which the project's own benchmarks show cutting time-to-first-token substantially under concurrent load, at the cost of a small disadvantage on short, unrelated single-turn prompts that have nothing to reuse.
It runs on CUDA, ROCm, Vulkan, or CPU, and supports GGUF models. A published install script picks the right build automatically on Linux x86_64; other platforms currently require building from source or running the Linux installer under WSL2. Fox is dual-licensed MIT or Apache-2.0, has no paid tier, and the project states it plans to keep it that way.