Programmable KV Cache
Research code and results accompanying a paper showing that a transformer's KV cache can be edited in place or recombined across contexts instead of always being recomputed, with large throughput and latency gains.
This repository accompanies the paper Models Take Notes at Prefill, which shows that editing a single token inside a cached prefill normally invalidates the whole downstream cache, because the transformer has already written a field-conditioned conclusion onto downstream tokens at prefill time. It establishes this causally and traces it to a specific component-level circuit, replicated across four model families.
It demonstrates that the KV cache can be treated as an editable, append-only notebook: amending it with a one-line erratum instead of recomputing reaches up to a 14.5x throughput gain and much lower time-to-first-token. It also treats the cache as composable: precompiling a reusable skill once and repositioning its cached KV into any context is nearly indistinguishable from full recompute, in linear instead of quadratic time. The approach is validated across model scale, quantization, mixture-of-experts and multimodal image caches.
Contents include the paper (arXiv:2606.17107), an interactive web companion driven by the released result records, a written explainer, and the code and experiment results themselves.