ContextCache
Open-source middleware that compiles LLM tool-calling schemas into a reusable key-value cache on disk, so repeated tool definitions are not reprocessed on every request, keeping response latency flat regardless of tool count.
ContextCache addresses a specific cost in tool-calling LLM requests: the full JSON schema for every available tool is normally reprocessed on each call, even though the tools themselves rarely change. It compiles the system prompt and all tool definitions into a single key-value cache once, keyed by a hash of the schemas, stores it on disk, and reuses it across users, sessions, and requests, so only the user's own query needs to go through prefill.
Reported benchmarks on a Qwen3 model show time-to-first-token staying near 200 milliseconds even with 50 registered tools, versus several seconds without caching. The project ships a demo mode that needs no GPU, a live mode requiring roughly 8 gigabytes of VRAM, a browser dashboard, and a Python client that can also hold server-side language model credentials so client requests do not need their own API keys.