GLM-5.3-Flash-EXL3 on 2x NVIDIA DGX Spark
A reproduction kit for serving GLM-5.3-Flash, a 320-billion-parameter mixture-of-experts model, in production across two NVIDIA DGX Spark machines with a 1,000,000-token context window and documented fixes for prefix-cache misses.
This repository documents a specific production deployment of GLM-5.3-Flash-EXL3 across two NVIDIA DGX Spark units connected by a direct 200 Gb link, using tensor parallelism, speculative decoding, and a serving image built by the repository's own Dockerfile rather than a pulled artifact. It focuses on a problem the author found in earlier configurations: because the model mixes a state-space component with attention and caches state in large fixed-size pages, a single cache miss could void an otherwise-successful prefix-cache hit, collapsing effective throughput under real coding-agent traffic even when every other check passed.
The repository documents each fix applied, including a page-size setting and patch that restored high cache-hit rates, a scheduling change for multi-session and co-batched requests, a patch for prompts shorter than one cache page, a fix for a template setting that previously invalidated the cache when reasoning mode was toggled, and a warm-up routine at boot, each with before-and-after measurements.
It is written in Python and released under the Apache-2.0 license.