GuideLLM
An SLO-aware benchmarking platform that simulates realistic workloads against OpenAI-compatible and vLLM-native servers to measure LLM deployment latency, throughput and capacity limits.
GuideLLM is a platform for evaluating how language models perform under realistic production workloads. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native inference servers, generating traffic patterns such as synchronous, concurrent and rate-based execution profiles, and supports real and synthetic datasets including multimodal inputs.
It captures latency and token-level statistics such as time to first token and inter-token latency, and produces standardized, exportable reports for dashboards, regression tracking and capacity planning. It is built for high-throughput benchmarking with multiprocessing, threading and async execution, and offers both a CLI and a Python API.