ConsumerBench
A benchmarking framework that measures the real-world runtime performance of GenAI applications such as chatbots, deep research agents, image generation, and live captioning under realistic conditions on consumer-grade hardware.
ConsumerBench evaluates how user-defined generative AI applications actually perform on end-user devices rather than in a controlled server environment, currently supporting four application types out of the box: a chatbot using a local OpenAI-compatible backend powered by llama.cpp, a deep-research agent built on the open-deep-research framework served through LiteLLM, an edge-optimized image generator using stable-diffusion-webui in API mode, and real-time or offline audio transcription using a Whisper-based backend.
A workflow is defined in a YAML configuration file and run through a single benchmark command, and the framework collects GPU compute and memory bandwidth, CPU utilization, CPU memory bandwidth, GPU power, and CPU power during the run, saving timestamped results with automatically generated PDF plots. It has been tested on both a workstation with a Xeon CPU and an RTX 6000 GPU, and on an Apple Silicon MacBook Pro.