BERGEN
A benchmarking library for retrieval-augmented generation systems, focused on question answering, that standardizes how retrievers, rerankers and language models are compared across many datasets.
BERGEN benchmarks RAG systems by giving every experiment a common, YAML-configured pipeline of a retriever, an optional reranker, a language model and an evaluation dataset, so results across papers and setups can be compared on equal terms. It supports more than 20 retrievers, four rerankers and more than 20 large language models, and includes match, exact-match and LLM-based evaluation metrics.
A typical run is a single command line naming the retriever, reranker, generator and dataset; results can be cached at the retrieval stage and reused across generator runs, including through vLLM for faster generation. The library also supports multilingual experiments and pairwise comparisons judged by an LLM, and its results are documented against two accompanying papers on RAG benchmarking and multilingual RAG.