VisRAG
A vision-language-model-based retrieval-augmented generation pipeline that embeds documents as images instead of parsed text, and its newer EVisRAG variant that reasons over per-image evidence for multi-image questions.
VisRAG is a vision-language model (VLM) based RAG pipeline that embeds a document directly as an image with a VLM instead of first parsing it into text, then retrieves those image embeddings to enhance a VLM's generation. Compared to text-based RAG, this avoids the information lost when parsing a document.
EVisRAG, described as VisRAG 2.0, is an evidence-guided extension for multi-image questions: it first has a model linguistically observe retrieved images to collect per-image evidence, then reasons over that evidence to answer, training with Reward-Scoped GRPO for token-level rewards that jointly optimize visual perception and reasoning.
The project releases benchmarks, training and test data, and trained models such as EVisRAG built on Qwen2.5-VL, and reproduces within the UltraRAG v2 project.