ArtSeek
Research code for a system that answers open-ended questions about artworks by having a multimodal LLM read a painting and a predicted artwork card, then retrieving supporting passages from a multimodal Wikipedia knowledge base when it needs external knowledge.
Given an image of an artwork and a question, ArtSeek has a multimodal LLM examine the painting and an artwork card predicted by a classifier, decide when it needs external knowledge, and retrieve it from Wikipedia before answering, using the language model zero-shot; the repository includes an interactive demo.
WikiFragments is a multimodal knowledge base of 8.2 million fragments from 973,000 visual-arts Wikipedia pages, searched with ColQwen2 late-interaction retrieval. A two-stage retrieval step keeps only the query's text tokens and prefetches on pooled vectors before reranking on full vectors, matching full multi-vector retrieval at under 7% of its cost. A late-interaction classifier predicts artist, genre, style, media and tags over an open label set, and a single worked in-context example teaches the model when and what to retrieve over several steps.
Built with Jupyter notebooks and Qwen2.5-VL. MIT licensed; from the University of Bari Aldo Moro.