RAG Experiment Accelerator
A config-driven tool for running experiments and evaluations of RAG search-and-response quality, testing different Azure AI Search and Azure OpenAI hyperparameters to find the best combination.
RAG Experiment Accelerator is a config-driven tool for running experiments and evaluations of search queries and response quality using Azure AI Search and the retrieval-augmented generation pattern.
It lets researchers, data scientists and developers test different Azure AI Search and Azure OpenAI hyperparameters, compare search strategies, fine-tune and optimize parameters, and find the best combination, generating detailed reports and visualizations from the results. Experiments are defined by specifying search engine parameters, search types, query sets and evaluation metrics, and it integrates with Azure AI Search, Azure Machine Learning, MLflow and Azure OpenAI.
It supports multiple document loaders, including basic LangChain loaders and a custom loader built on Azure Document Intelligence's prebuilt-layout model, and creates multiple search indexes from the hyperparameter configurations in its config file. It can also sample a dataset by clustering content and sampling each cluster, to keep a sampled subset representative of the whole.