GLM-5.2 NVFP4 KV cache — 4x DGX Spark
A ported 4-bit NVFP4 key-value cache for serving GLM-5.2 across four DGX Spark machines with vLLM, trading some decode speed for a much larger context pool.
This project ports a 4-bit (NVFP4/E2M1) key-value cache into an existing GLM-5.2 serving stack built on vLLM across four DGX Spark (GB10) machines with tensor parallelism, and measures the change against the project's earlier fp8 baseline on the same hardware.
Measured results include a KV-cache record of 400 bytes per token versus 656 bytes for fp8, and a KV pool of about 317,000 tokens in 10.95 GB versus 200,064 tokens for fp8, a gain of roughly 59%. Peak single-stream decode speed on high-acceptance content reached about 42 tokens per second, ahead of the fp8 baseline's reported peak of 36, but on mixed everyday content NVFP4 was about 25% slower than fp8 because the noisier 4-bit cache lowers the speculative-decoding draft's acceptance rate.
The README documents its benchmark methodology in detail and states plainly where NVFP4 trades decode speed for a larger context pool rather than winning outright.