Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
A plug-and-play, training-free retrieval-augmented generation pipeline that adds visually-aligned OCR, ASR and object-detection text to a large video-language model, improving long-video comprehension without extra fine-tuning.
Video-RAG retrieves three kinds of visually-aligned auxiliary text, OCR, ASR and object-detection output, produced by external open-source tools, and brings them into an existing large video-language model's context through retrieval-augmented generation, rather than requiring a commercial API or additional training. Applied to a 72-billion-parameter open-source model, the authors report state-of-the-art results on Video-MME, surpassing models such as Gemini-1.5-Pro.
The reference implementation is built on LLaVA-NeXT, using a separate conda environment for the APE object-detection service and packages such as spaCy, FAISS, EasyOCR and ffmpeg-python for the retrieval pipeline; the pipeline runs against LLaVA-Video-7B out of the box and can be adapted to other large video-language models by swapping in the target model, its inference call, and its video-preprocessing step.