Throttle
An open-source CLI that benchmarks any OpenAI-compatible inference endpoint, such as vLLM, SGLang, Ollama or LMDeploy, to measure real cost per million tokens, with statistics designed not to call a winner from noisy or overlapping evidence.
Throttle is built for teams that self-host models, pay for GPU time, and want to know their actual dollars-per-million-tokens instead of guessing; a browser or Kaggle notebook demo serves a small model on a free GPU, measures its cost, changes one setting and prints a verdict in a few minutes.
It ships recorded real comparisons, including one showing that four checks of an unchanged server swung between minus 11 percent and plus 27 percent purely from noise, which is why it insists on statistically sound before/after comparisons rather than single-run readings; results can be shared as a pre-filled GitHub issue.
It installs with pipx install throttle-pro and runs against a local Ollama server, a real GPU host, or as a fully simulated demo with no GPU at all.