SGLang
A high-performance serving framework for large language models and multimodal models, aimed at low-latency, high-throughput inference across NVIDIA and AMD GPUs as well as TPUs.
SGLang serves large language and multimodal models with techniques such as RadixAttention for reusing cached prefixes across requests, a zero-overhead batch scheduler, a cache-aware load balancer, and speculative decoding, alongside a compressed finite-state-machine approach for faster structured JSON output. It provides day-0 support for many newly released open models and typically supports both NVIDIA and AMD GPUs, and, through an SGLang-Jax backend, Google TPUs, including large-scale expert-parallel deployment of mixture-of-experts models such as DeepSeek.
A separate SGLang Diffusion component applies the same serving infrastructure to accelerate image and video generation models.
It is part of the PyTorch ecosystem, is written in Python, and is licensed under Apache-2.0.