Frontier MoE on consumer Blackwell (SM120)
This project patches vLLM with hand-written SASS kernels so very large Mixture-of-Experts language models can run on consumer and workstation Blackwell GPUs that cannot otherwise fit their official checkpoints.
This project combines an official vLLM release with a generated runtime patch and hand-written SM120 SASS kernels to serve frontier Mixture-of-Experts models, including GLM-5.2 (753B), DeepSeek-V4-Flash (159B) and Kimi-K2.7-Code (1T), on consumer and workstation Blackwell cards such as the RTX PRO 6000 and RTX 5090.
Routed experts compress to a 2-bit codebook with an FP4 recovery tier that restores precision where it matters most. When a model still outgrows VRAM, experts move to pinned host RAM and then to an NVMe pack file, turning the GPU into an expert cache with batched fetch and replay on a miss. Two further large models are served from their official checkpoints and official vLLM images with the same SM120 gap fixes and hand-written decode kernels, shipped as self-contained images with launchers and a deploy guide.