Narwhal
A disaggregated LLM inference framework that reallocates prefill and decode roles across a fixed GPU fleet as demand changes, while model weights stay loaded.
A controller continuously estimates prefill and decode pressure against configured service-level objectives for the current and neighboring role splits, using measured per-engine performance curves, and moves an engine between roles only when doing so improves the worst-projected SLO ratio by a configured margin and passes role-floor, cooldown and health checks; resident requests finish on their current engine while new requests follow the revised split.
It routes prefill and decode traffic separately with KV-cache transfer between them, offers streaming and non-streaming chat and completion APIs including function tools and reasoning output where the underlying engine supports them, and includes request deadlines, disconnect cancellation, bounded queues, Prometheus metrics and a Grafana dashboard.
It installs from PyPI on Linux with Python 3.11 or newer, with a development command that stands up a local NVIDIA CUDA fleet on Ubuntu, or Ubuntu under WSL2, for development.