autotune — Local LLM Inference Optimizer
autotune is a middleware layer that computes the exact KV cache a local LLM request needs, pins the system prompt in memory and manages context windows automatically for Ollama, LM Studio and Apple Silicon MLX, without any code or workflow changes.
autotune sits between an application and its local model server and dynamically sizes the KV cache instead of relying on a fixed context window such as Ollama's default, which reduces both memory use and time to first token. It ships with a built-in monitoring dashboard, run with a single command, showing live RAM pressure, requests, average time to first token, tokens per second and cache savings compared to the model server's default, all computed from data on the same machine with nothing sent to the cloud.
The project reports independently benchmarked reductions in time-to-first-word, KV prefill time and KV cache RAM use across several small local models on Apple Silicon, using nanosecond-precision timers from the underlying runtime rather than Python wall-clock estimates, with statistical significance testing.
It is a Python CLI released under the MIT license.