XRAG
A benchmarking framework for evaluating the foundational components of retrieval-augmented generation systems, comparing text splitters, retrievers, and orchestrators against multiple evaluation metrics.
XRAG is a benchmarking framework for dissecting and evaluating the core modules of retrieval-augmented generation (RAG) systems. It measures how different configurations and components affect overall RAG performance, using LLM-based evaluation, deep evaluation, and traditional metrics to judge retrieval quality, response faithfulness, and answer correctness.
It includes multiple text splitters, such as semantic, sentence, and sentence-window node parsers, and multiple orchestrators, including self-RAG, adaptive-RAG, and similarity-guided RAG. It can also generate its own question-answer pairs from a folder of documents, supports Ollama models, and can run as a backend service through its API, with a web interface for interactive use.
It is written in Python, licensed under Apache-2.0, and accompanies a paper accepted at ICDE 2026.