Qwen4-Exp (qwen4exp)
A llama.cpp fork that adds sliding-window attention, multi-token prediction and a 4-bit KV cache to a Qwen hybrid model, keeping decode speed high at long context without losing long-range recall.
This fork targets a hybrid model whose dense self-attention layers attend the entire window on every step, so decode speed collapses as context grows. Sliding-window attention windows those dense layers so each step only attends the most recent tokens, while a configurable number of layers stay global for long-range recall; it is opt-in, so a plain model file with no matching metadata loads the original dense path unchanged.
Combined with multi-token prediction, n-gram support on the sliding-window path and a 4-bit KV cache, measured decode speed holds around 22.8 tokens per second at 12 thousand tokens of context and 15.7 tokens per second at 100 thousand, and a code value planted at the very start of the context is still recalled exactly at 100 thousand tokens. The repository ships as two branches built from different upstream base commits for two related model variants. It is licensed MIT and written in C++.