Qwen3.6-35B-A3B-MTP on NVIDIA L4 24GB
A deployment guide and benchmark for running the Qwen3.6-35B-A3B mixture-of-experts model with multi-token-prediction speculative decoding on a single NVIDIA L4 24GB GPU using the official llama.cpp Docker image.
This guide deploys Qwen3.6-35B-A3B, quantized to Unsloth's Q4_K_XL GGUF format, with multi-token-prediction speculative decoding on a single NVIDIA L4 24GB GPU, using a GCP g2-standard-8 instance and the official llama.cpp Docker image. It reports honestly measured decode throughput of roughly 91 tokens per second on chat workloads up to about 99 on math and reasoning workloads, a roughly 45 percent improvement over the out-of-the-box configuration at the same quantization and no quality loss.
The reported gains come from two lossless changes: forcing every mixture-of-experts expert onto the GPU instead of offloading some to the CPU, and disabling GDDR6 ECC, which the guide measures as silently costing about 10 percent of memory bandwidth on this memory-bound workload. A benchmark script reproduces the reported numbers, and later sections in the guide isolate each parameter, such as ECC state or prompt difficulty, in separate controlled experiments rather than mixing them into the headline figure.