homebench
A single-command terminal benchmark that discovers the language models already installed in a local runner, such as Ollama, LM Studio, llama.cpp or vLLM, and measures their speed, memory use and answer quality together on a live leaderboard.
homebench measures three things existing tools each cover only in part: tokens per second and time-to-first-token, timed from the actual token stream; two memory figures, the resident model size the runner itself reports and the peak process memory growth sampled across the whole run; and quality, graded across 31 deterministic tasks spanning math, reasoning, factual recall, instruction-following, extraction and code understanding, with an optional LLM-as-judge mode adding open-ended tasks such as summaries and explanations. A composite value score, normalized within a given run, blends quality, speed and memory to name the best-fitting model for the machine it is run on rather than only ranking the options.
It needs no configuration, no API keys and no cloud connection: it is installed with pip, then run with no arguments against whichever models are already pulled into a local runner, rendering results as a live terminal leaderboard.
It is a Python package licensed under MIT.