Project Chronos
An experimental storage-aware mixture-of-experts inference stack that prefetches predicted experts before decoding starts, aimed at keeping SSD-offloaded MoE models from stalling generation.
Project Chronos targets a specific bottleneck in mixture-of-experts models such as Mixtral, DeepSeek-MoE and Qwen-MoE: routing decisions are normally made at decode time, so when a needed expert is not already in VRAM, generation stalls while it is moved from SSD to RAM to VRAM. Chronos instead predicts which experts prefill will need and loads them ahead of the first decode token, and synchronizes per-expert rather than blocking the whole system on one transfer.
It stores experts as cluster-aware safetensors files built offline by Louvain clustering, memory-maps whole clusters into RAM in one shot to turn random reads into sequential ones, and degrades gracefully on a cache miss by blending in a shared expert's output while the missing expert finishes loading, rather than stalling generation. Under a simulated 30ms SSD delay this keeps more than 35ms of per-token pipeline slack compared with a blocking offload path. Python, Apache-2.0, marked experimental.