# Narwhal

A disaggregated LLM inference framework that reallocates prefill and decode roles across a fixed GPU fleet as demand changes, while model weights stay loaded.

A controller continuously estimates prefill and decode pressure against configured service-level objectives for the current and neighboring role splits, using measured per-engine performance curves, and moves an engine between roles only when doing so improves the worst-projected SLO ratio by a configured margin and passes role-floor, cooldown and health checks; resident requests finish on their current engine while new requests follow the revised split.

It routes prefill and decode traffic separately with KV-cache transfer between them, offers streaming and non-streaming chat and completion APIs including function tools and reasoning output where the underlying engine supports them, and includes request deadlines, disconnect cancellation, bounded queues, Prometheus metrics and a Grafana dashboard.

It installs from PyPI on Linux with Python 3.11 or newer, with a development command that stands up a local NVIDIA CUDA fleet on Ubuntu, or Ubuntu under WSL2, for development.

Website: https://athrael-soju.github.io/Narwhal/