FlagGems-vllm
A Triton-based operator library providing optimized vLLM-specific kernels such as MoE routing and fused expert operators, used alongside the general FlagGems operator library for faster LLM inference.
FlagGems-vllm is part of FlagOS. It is a high-performance operator library, implemented in the Triton programming language, that provides optimized implementations of common vLLM operators for high-performance inference and deployment of widely used models across multiple hardware backends.
It complements two related projects: FlagGems, the general-purpose operator library that provides common PyTorch and Triton operator replacements registered into PyTorch dispatch through flag_gems.enable(), and vllm-plugin-fl, the vLLM plugin layer that uses FlagGems as its global operator backend and explicitly calls FlagGems-vllm for vLLM-specific fused kernels not covered by PyTorch dispatch. FlagGems-vllm exposes its operators through the flaggems_vllm Python package, for example grouped_topk, fused_experts_impl and moe_align_block_size.
Built with Python and Triton, installed with pip alongside scikit-build-core, pybind11, ninja and cmake, and requires a compatible PyTorch build for the target accelerator. Apache-2.0 licensed.