local-llm-bench
A scenario-based benchmark for local LLM inference on Apple Silicon that measures effective end-to-end throughput, including the prefill phase, across real workloads such as agent conversations and document classification.
local-llm-bench runs realistic multi-turn scenarios, such as an eight-turn agent conversation, document classification and creative writing, against a running local inference backend and reports effective throughput, defined as output tokens divided by the total wall-clock time including prefill, rather than only the peak generation speed a chat UI usually shows. Since prefill time grows with conversation history, a model advertised at a high tokens-per-second rate can deliver much lower effective throughput once real context is accounted for.
It compares engines such as MLX and llama.cpp and multiple serving backends, including LM Studio, on the same hardware and workloads, and lets a user add custom single-shot or conversational scenarios to test their own use case. It requires only Python 3.8 or later with no dependencies and a model already loaded in a supported backend, and results are saved automatically as JSON files.