Multimodal Retrieval-Augmented Generation (RAG) chat with videos
An example project that builds a question-answering system over a collection of videos, extracting frames and transcripts, embedding them into a shared multimodal space, and answering questions with a vision-language model.
This example project implements multimodal Retrieval-Augmented Generation for chatting with a collection of ingested videos. A preprocessing step handles video and transcript pairs, typically in WEBVTT format, by extracting a representative frame from the middle of each transcript segment's time range and pairing it with metadata such as the segment's timestamp, transcript text, and source video path.
It uses the BridgeTower embedding model to embed both text and images into one shared multimodal semantic space, LanceDB as the local vector database holding the ingested videos, and LLaVA, an open-source large vision-language model, to combine visual and textual understanding for retrieval and answering. Queries call the Prediction Guard API by default, though the project notes other model-hosting providers can be substituted.
The Python scripts must be run in order, and questions can be asked either through a Gradio web interface or directly in Python code.