Qwen3.8-Flash-Next on 2x RTX 3090
A serving build and set of patches that run a large mixture-of-experts language model across two consumer GPUs and system memory, offloading inactive experts to RAM to fit a 256,000-token context.
This project packages a checkpoint and serving code to run the Qwen3.8-Flash-Next language model on two RTX 3090 GPUs with 128 gigabytes of system memory. Because the model has high sparsity and a low active-parameter count, it is a good candidate for offloading to the CPU: the build keeps the full set of experts and a compressed lookup table in system memory, caches the most recently used experts on the GPUs, and uses a small draft model to recover decoding speed.
The checkpoint combines a 4-bit weight quantization method with 8-bit compressed auxiliary tensors to fit the model in 128 gigabytes, and the serving code is a pinned build of vLLM plus this project's own patches, including streamed prefill staging that copies each layer's experts to the GPU ahead of need and a smaller draft vocabulary for its speculative decoding step.