BenchmarkQED
A suite of tools for automated benchmarking of retrieval-augmented-generation systems, generating synthetic queries, judging answers with an LLM, and preparing datasets for reproducible testing at scale.
BenchmarkQED provides three components for testing RAG systems: AutoQ generates synthetic queries across a range of data scope, from local queries answerable from a few text regions to global queries that need reasoning over most or all of a dataset, including multi-hop local queries that combine related questions sharing named entities; AutoE evaluates RAG answers side by side on relevance, comprehensiveness, diversity and empowerment using a large-language-model-as-judge approach, and can also score correctness and completeness against ground truth or run assertion-based scoring from manually written or AutoQ-generated assertions; and AutoD provides data utilities for sampling and summarizing datasets so query generation gets consistent input.
The project also releases two evaluation datasets: transcripts of 70 episodes of a technology podcast series, an updated version of the dataset used in the original GraphRAG paper, and a collection of nearly 1,400 health-related news articles from the Associated Press.
It is a Python project from Microsoft Research, distributed under the MIT license.