Optimum-Benchmark
A unified multi-backend, multi-device benchmarking utility for Transformers, Timm, PEFT, Diffusers and Sentence-Transformers, covering inference and training under Optimum's hardware optimizations and quantization schemes.
Optimum-Benchmark measures latency, throughput, memory usage and model-loading cost across many backends, including a LlamaCpp backend for llama-cpp-python, a vLLM backend, and Py-TXI, and devices, in both distributed and non-distributed settings, for the Transformers, Diffusers, PEFT, Timm and Optimum library ecosystem. It supports numactl-based NUMA node control for its process and Torchrun launchers, and ships minimal Docker images for CPU, CUDA and ROCm testing.
It is aimed at hardware partners comparing devices on the same models, ecosystem users comparing models on latency, throughput, memory and energy, and anyone benchmarking hardware- or backend-specific optimizations and quantization schemes. It offers both a Python API for isolated or distributed benchmark scripts and a Hydra-based CLI for running benchmark sweeps, and hosts the codebase behind the public LLM-Perf Leaderboard. The project describes itself as a work in progress, not yet ready for production use.