MTRAG: Multi-Turn RAG Benchmark
A human-generated, multi-turn benchmark and four document corpora for evaluating retrieval-augmented-generation systems on realistic, multi-turn conversations.
MTRAG provides 110 multi-turn, human-written conversations converted into 842 evaluation tasks spanning conversation, retrieval and generation, built on four document corpora from Wikipedia, cloud technical documentation, finance and government domains, with two of the four corpora assembled specifically for this benchmark.
Alongside the human-authored data, the project publishes 200 synthetically generated conversations that follow the same properties, plus a related benchmark, MTRAG-UN, targeting open challenges in multi-turn RAG conversations, and MTRAGEval, a SemEval shared task built on the same data.
It is written in Python and licensed under Apache-2.0; the document corpora must be downloaded and uncompressed separately before use.