rag-params-finder
A RAG parameter-sweep and evaluation toolkit that systematically tests combinations of embedding models, chunking strategies, and retrieval methods against a user's own data and questions, before they build a RAG application.
rag-params-finder inverts the usual approach to building a RAG system, where a team guesses an embedding model, chunking method, and retrieval method, discovers it performs poorly, and refactors. Instead, given a user's own data and questions, it runs every combination of embedding model, chunking method, and retrieval method, stores the resulting retrieval scores, and shows exactly which configuration performs best, using embedding-only calls rather than LLM calls, which the project describes as far cheaper than an LLM-based evaluation loop.
It sweeps across 13 Voyage AI embedding models, five chunking methods (fixed, recursive, token, sentence, and semantic), and three retrieval methods (dense, sparse, and hybrid), against questions organized by persona, either user-supplied or generated as part of a golden-master generation process. Storage backends include MongoDB Atlas Vector Search by default or Postgres with pgvector, supporting local Docker or Supabase-hosted Postgres, and embeddings can come from Voyage AI, local sentence-transformers with no API key required, or an open-source SIE embedding service.
Configuration is a single YAML file, and results persist so different sweeps can be compared across sessions.