llm_speedtest
A benchmarking tool for local large-model inference that measures prefill and decode speed, offering a browser-only version and a Python backend version that removes the browser's low connection-concurrency limit for higher-concurrency testing.
This tool measures the prefill (input processing) and decode (token generation) performance of locally deployed large-language-model inference services. A pure front-end version runs entirely in the browser, can optionally upload results to a shared cloud-backed leaderboard, and supports batch CSV upload of past results; a Python back-end version issues requests directly, bypassing the browser's connection-concurrency limit to support fifty or more concurrent requests.
Both versions produce charts covering prefill and decode throughput, time-to-first-token and inter-token latency with percentile statistics, and historical comparisons across runs, and can export results to CSV. The tool is a heavily modified version of an existing community project, adding a retry mechanism, separated prefill and decode measurement, concurrency-throughput fixes, and automatic configuration saving.
It targets users comparing local inference performance across GPUs and inference frameworks, with a one-click Windows startup script for the Python version.