xLLM
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for Chinese AI accelerators and donated to the OpenAtom Foundation.
xLLM is an efficient inference framework specifically optimized for Chinese AI accelerators, including Ascend NPU, Cambricon MLU, Moore Threads GPU, Hygon DCU, MetaX MACA and Iluvatar CoreX GPU, aimed at enterprise-grade deployment with lower cost and higher efficiency than general-purpose stacks on that hardware. Its architecture decouples the service layer, which handles scheduling and availability, from the engine layer, which handles computation.
It has been battle-tested at scale across JD.com's core retail business, and provides day-zero support for models such as GLM-5.3, MiniMax-M3 and DeepSeek-V4 as they are released, alongside a hybrid key-value cache management system built on Mooncake with intelligent offloading and prefetching. xLLM was donated to the OpenAtom Foundation, publishes a technical report on arXiv, and is written in C++ under the Apache 2.0 license.