Chitu
A high-performance inference framework for large language models that targets efficiency, flexibility, and support for multiple hardware vendors, from single GPUs to large clusters.
Chitu is a production-oriented inference framework for large language models such as DeepSeek, Qwen, GLM, and Kimi, aimed at taking a model from small-scale experiments to large production deployments. It focuses on efficiency, flexibility, and reliability under real production traffic.
It supports a range of accelerators beyond NVIDIA's newest and older GPU lines, including domestically produced Chinese chips, and scales from CPU-only or single-GPU setups up to large clusters, including multi-node and disaggregated prefill/decode deployment through a single executable. Recent releases added FP4-to-FP8/BF16 conversion, CPU+GPU heterogeneous inference for very large models, and native support for Huawei Ascend 910B hardware.
Chitu is written in Python and released under the Apache License 2.0, and it reuses code from and builds on projects such as FlashAttention, FlashInfer, and vLLM.