rag-ci
A CLI and GitHub Action that regression-tests RAG pipeline changes against a baseline using paired bootstrap significance tests, blocking a pull request only when a retrieval metric change is statistically real rather than noise.
rag-ci measures every change to a RAG pipeline against a baseline and fails a build when retrieval performance actually got worse, using paired bootstrap significance tests with confidence intervals rather than comparing a bare score to a threshold. A deterministic demo corpus is generated and measured on the machine running it, through the same code paths the run and gate commands use, so results are reproducible rather than pre-recorded.
With only a few paired test cases, the tool deliberately refuses to block a build even when a metric drop looks large, because the interval cannot separate it from noise; a wider golden set narrows that interval until a genuine regression trips the gate. This responds to research showing many retrieval enhancements stop contributing once a strong reranker is present, so teams pay in latency and tokens for changes that do nothing.
Unlike other RAG evaluation tools that compare a score to a fixed threshold, rag-ci conditions its pass or fail verdict on a significance test between two runs over the same questions, behind a CLI and a GitHub Action.