FlashInfer
A kernel library and generator for LLM inference that provides unified, high-performance attention, GEMM, and mixture-of-experts kernels across GPU backends.
FlashInfer automatically selects among multiple backend implementations, including FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM, for the best performance on a given GPU architecture and workload, from Turing (SM75) through Blackwell. Its attention kernels cover paged and ragged KV cache, prefill, decode, and append phases, DeepSeek's Multi-Latent Attention, cascade attention for shared prefixes, and sparse attention patterns; its GEMM kernels support BF16, FP8, and FP4 precision, including grouped GEMM for LoRA and multi-expert routing.
It also provides fused mixture-of-experts kernels with several routing methods, sorting-free top-k, top-p, and min-p sampling, speculative decoding support, and communication primitives such as AllReduce and multi-node NVLink support, and is compatible with CUDA Graphs and torch.compile for low-latency serving.
It is written primarily in CUDA and C++ and distributed under the Apache-2.0 license.