gLLM
An inference engine for distributed LLM serving that supports dense, mixture-of-experts, multimodal and hybrid-attention models with continuous batching, paged attention and multiple forms of parallelism.
gLLM is an inference engine for distributed serving of large language models, supporting dense, mixture-of-experts, multimodal/vision-language and hybrid-attention architectures from Hugging Face across offline, online and interactive-chat deployment. It combines continuous batching, paged attention, chunked prefill, prefix caching, CUDA graphs, token throttling, and pipeline, expert and tensor parallelism, aiming for throughput at least on par with mainstream inference engines from a comparatively small code base.
It has added support over time for models including DeepSeek V2 through V4, Qwen2 through Qwen3.6, and Kimi K2 variants, along with encoder disaggregation that separates a multimodal encoder from the language model for better resource use.
It is written in Python and released under the Apache-2.0 license.