Dynamo
An open-source, datacenter-scale inference orchestration layer that coordinates SGLang, TensorRT-LLM or vLLM across multiple GPUs and nodes with disaggregated serving and KV-aware routing.
Dynamo is an orchestration layer above inference engines: it does not replace SGLang, TensorRT-LLM or vLLM but turns them into a coordinated, multi-node inference system for LLM, reasoning, multimodal and video generation workloads. It combines disaggregated serving, intelligent routing, multi-tier KV caching and automatic scaling to maximize throughput and minimize latency.
It is intended for serving models across multiple GPUs or nodes that need coordination, KV-aware routing to avoid redundant prefill computation, independently scaled prefill and decode stages, automatic scaling to meet latency targets at lower cost, and fast cold starts for new replicas; a single model on a single GPU is generally served well enough by the inference engine alone.
It is built in Rust for performance and Python for extensibility, and reports results including higher throughput per GPU and faster time to first token when paired with KV-aware routing on models such as DeepSeek-R1 and Qwen3-Coder.