llama.cpp — DualDeadline branch
An independent research branch of llama.cpp implementing DualDeadline, a component-staged expert-prefetching technique for offloaded mixture-of-experts inference, letting a GPU claim only router-selected expert tensor slices while routing and outputs stay exact.
This branch of llama.cpp implements DualDeadline, described in the paper 'DualDeadline: Component-Staged Expert Prefetching for Exact Offloaded Mixture-of-Experts Inference.' It treats a gated mixture-of-experts expert as having two transfer deadlines instead of one: the gate and up projections are needed when expert computation starts, while the down projection is not needed until that computation finishes. When expert tensors are kept in pinned host RAM, the GPU can claim batch-1 expert matrix multiplications by copying only the router-selected experts' slices into a persistent, per-tensor cache, overlapping the down-projection copy with gate/up computation, with no prediction or approximation involved.
Measured on a Qwen3.6-35B-A3B model on an AMD Ryzen AI MAX+ 395 unified-memory system, staged DualDeadline beats a monolithic single-deadline version of the same mechanism by three to five percent at identical bytes, cache size and kernels, reproducing the paper's core claim inside a real decode loop; on that particular unified-memory hardware, however, a CPU-only baseline still wins because there is no PCIe transfer to hide behind staging. The authors note the mechanism targets discrete GPUs whose VRAM cannot hold the experts, which they have not yet measured.
It is a branch of the C++ project llama.cpp, licensed under MIT.