arle
A single-binary local inference server for coding agents, written in Rust, that keeps a request's KV cache resident across turns so later turns in a long conversation start as fast as the first.
arle is a local inference server aimed at coding agents, running as one Rust binary on Apple Silicon (Metal) or NVIDIA GPUs (CUDA), with no separate Python runtime. It exposes both Anthropic Messages and OpenAI-compatible Chat Completions APIs with streaming, so tools such as Claude Code, aider, or the OpenAI SDK can point at it directly.
Its main design goal is keeping a coding agent's repeated context fast: since an agent resends the whole conversation every turn, arle keeps the previous turn's KV cache on the accelerator, shares prefix pages across requests through a radix cache, and only re-prefills the newly added tokens, rather than reprocessing the whole growing conversation each time.
It ships four modes from one binary: an HTTP server, an interactive REPL with a built-in tool-using agent, a one-shot agent runner, and a hardware and model self-check. Source builds require explicitly selecting a CUDA or Metal backend feature.