blackwell-geforce-nvfp4-gemm
A set of patches and benchmarks that make NVFP4-quantized model inference run correctly and at full speed on Blackwell-generation GeForce and RTX PRO 6000 consumer GPUs, whose architecture differs from datacenter Blackwell.
This repository documents why consumer Blackwell GeForce GPUs such as the RTX 5090, 5080, 5070 Ti, and RTX PRO 6000 need separate handling from datacenter Blackwell: their SM120 architecture mixes instruction sets and memory limits from three prior GPU generations and lacks features such as tensor memory that datacenter Blackwell has, so kernels written for datacenter hardware fail to compile or crash at runtime.
It provides the specific patches needed to run vLLM, FlashInfer, and CUTLASS with NVFP4 quantization correctly on this architecture, covering grouped GEMM tile sizes, an optimized matmul kernel, and device-capability checks for mixture-of-experts dispatch, along with attention-related patches. Each patch is explained alongside the underlying hardware constraint it works around, and results are benchmarked against a Qwen mixture-of-experts model.