ColPali
A code library for training and running vision-language document retrieval models such as ColPali, ColQwen2, and ColSmol that embed and search documents by their visual layout as well as their text.
ColPali is a document retrieval approach and its code library that uses vision-language models to build multi-vector embeddings of a document image directly, instead of running layout recognition and OCR first. It feeds the visual patches from a model such as PaliGemma into a linear projection to create a representation that captures both a document's text and its visual layout, tables, and charts, training it to maximize similarity between matching document and query embeddings using a ColBERT-style late-interaction method.
The engine that implements this is now deprecated in favor of Sentence Transformers, which added first-class multi-vector, ColPali-style retrieval support in a recent release; the original repository remains available for research, reproducibility, and existing projects that depend on it.
It is written in Python and released under the MIT license.