tachyon
A small language-model inference engine for consumer GPUs that implements continuous batching, prefix caching and per-request KV-cache control, invoked directly from Python in a few lines of code.
Each request in tachyon keeps its own state, with KV caching enabled or disabled per request, and the decode step batches active requests together to keep the GPU busy, recycling a completed request's slot for a new one as soon as it finishes rather than waiting for the whole batch to complete. The prefill step uses a separate, smaller batch size to avoid running out of memory, and vectorized operations are used throughout to improve batch throughput.
Prefix caching reuses the expensive prefill computation for requests that share a system prompt or few-shot examples. The project also exposes an OpenAI-compatible server so an existing OpenAI client library can call it unchanged, and its own benchmarks on an RTX 4060 Ti report throughput rising from about 13 tokens per second with naive PyTorch to more than 660 tokens per second with prefix caching enabled on similar requests.