quant.cpp
A zero-dependency C implementation for running local LLM inference with lossless KV-cache compression, letting full documents fit in context on consumer hardware instead of relying on chunked retrieval.
quant.cpp is a pure C, zero-dependency LLM inference engine built around lossless KV-cache compression. By keeping recent tokens at full precision and compressing the rest, it reports up to 6.4x smaller KV-cache memory with a small increase in perplexity, letting whole documents fit in context on a laptop instead of relying on retrieval-augmented chunking.
It ships an Ollama-style command-line interface for pulling, running, and serving models, and an OpenAI-compatible HTTP server with streaming support, plus a three-line Python API for loading a model and asking it a question. Models are downloaded and cached locally on first use; no API key or GPU is required.
The project installs with pip and is written in C.