vLLM-Moet-GB10
A fork of vLLM tuned to run mixture-of-experts models several times larger than the available memory on a single NVIDIA GB10 machine (DGX Spark, ASUS Ascent GX10), using 2-bit expert planes with hand-written SASS kernels, an FP4 quality-recovery tier and speculative decoding.
vLLM-Moet-GB10 serves large mixture-of-experts language models on a single GB10-class machine with 121.6 GiB of unified memory, even when the model's checkpoint is two to three times larger than that memory. The routed experts - roughly 97 percent of a large MoE model's parameters, of which each token only uses a handful - are stored as 2-bit codes computed by hand-written SASS GEMM kernels, while attention, shared experts, dense layers, embeddings and the router keep the checkpoint's original precision; a pool of the hottest experts, chosen at runtime from actual routing decisions, is instead served from an FP4 tier read from NVMe for quality recovery.
Quantization into these compressed expert planes runs once on first boot and is cached for every later boot, and speculative decoding - DeepSeek's own draft head for DeepSeek models, or a block-diffusion drafter for GLM - supplies additional decode speed. The project documents, per supported model, measured throughput and code-quality benchmark scores against each model's original published numbers.
It currently supports DeepSeek-V4-Flash-0731 and GLM-5.3-Flash, each with its own setup guide covering what had to change to make that model run.