Benchmark-Agent
An autonomous agentic system that turns a natural-language evaluation goal into a verified, executable benchmark dataset, covering requirement analysis, subtask design, data grounding and quality control.
Benchmark-Agent is an autonomous agentic system for constructing and customizing evaluation benchmarks. Given a high-level evaluation goal written in natural language, it orchestrates a full pipeline: requirement analysis, subtask design, data grounding, transformation planning, sample realization and quality control, without manual dataset curation.
Specialized planner and executor agents coordinate the pipeline stages, dynamically choosing datasets, transformation tools and next-step actions, and it targets multimodal evaluation across text, image, audio and cross-modal tasks in areas such as reasoning, understanding and domain-specific evaluation. Intermediate results are cached so that reruns can proceed incrementally instead of repeating earlier work.
It is written in Python, described in an accompanying paper on arXiv, and installed from source with uv into a Python 3.11 virtual environment; API and model settings are configured in a YAML file.