Multi-Modal RAG with ColPali on Azure Kubernetes Service (AKS)
A Microsoft accelerator that deploys a multi-modal RAG pipeline on Azure Kubernetes Service using ColPali, a vision-language model that embeds document pages as images to preserve layout, tables, and charts without OCR.
This repository is a Microsoft-provided accelerator for running multi-modal Retrieval-Augmented Generation using ColPali, a document-understanding model created by the Illuin Technology research team, providing Azure infrastructure, deployment scripts, and integration patterns to operationalize it on Azure Kubernetes Service. It is explicitly a demonstration meant to be reviewed and adjusted before use, not a production-ready solution.
ColPali processes each document page as an image using late-interaction embedding, producing hundreds of token-level embeddings per page instead of compressing a whole document into one vector, so query tokens can be compared directly against document tokens, preserving layout, charts, tables, and visual elements that OCR-based pipelines typically lose. Traditional multi-modal RAG needs complex chunking, OCR preprocessing, and separate handling for text, tables, and visuals, whereas the vision-language-model approach handles all of these uniformly by processing pages holistically as images.
While the repository is built around ColPali, it states that any late-interaction embedding model could be substituted.