KV Packet
A recomputation-free key-value caching library for large language models that stitches independently precomputed document caches together at serving time with cheap positional realignment instead of recomputing tokens.
KV Packet lets an LLM reuse document key-value caches that were precomputed independently of each other, by learning trainable soft-token header and trailer adapters and stitching the caches together at serving time with only cheap RoPE realignment, avoiding the token recomputation that ordinary cache-recomputation methods require. The main branch holds the maintained library; a separate branch preserves the paper's original experiments and baselines.
It is installed from PyPI, requires Python 3.12 or newer and a matching PyTorch build, and offers optional extras for KVPress cache compression, an OpenAI Responses-compatible HTTP and SSE transport, and MCP support for agent examples. The repository focuses on the library itself, and also includes a controlled evaluation track for agentic and skill-based flows built on top of it.