BatchGen
A batch-inference engine for large mixture-of-experts language models that minimizes the total time to finish a batch of requests rather than single-request latency.
BatchGen coordinates sequence-level scheduling, expert-level batching and host-to-device KV-cache movement across GPU clusters, using CPU memory as an extension of the KV-cache hierarchy for workloads whose weights and KV state exceed a single GPU's capacity. The same batch API drives offline generation, evaluation, synthetic-data pipelines, test-time scaling and reinforcement-learning rollouts.
Recent releases add support for GLM-5.3-FP8 with a dedicated chat template and reasoning parser, GLM-5.2-FP8 on an eight-GPU H200 node with long-context prefill deployment guides, and the 2.8-trillion-parameter Kimi-K3 model across multi-node H200 or H20 clusters, following earlier support for DeepSeek-R1 and V3 at 671 billion parameters. The project publishes deployment guides and workload-specific throughput comparisons against other serving engines, explicitly describing these as measurements on a particular workload rather than a universal ranking.
It is intended for both production users and researchers, exposing the scheduling and systems internals needed for experimentation.