VLM2Vec
VLM2Vec is the code and data interface for the MMEB family of benchmarks for embedding models, with the latest version, MMEB-V3, covering text, image, video, audio, visual document and agent-centric retrieval.
VLM2Vec provides the code and data interface behind the MMEB family of benchmarks for embedding models, most recently MMEB-V3, which measures performance gaps in omni-modality embedding models across text, image, video, audio, visual document and agent-centric retrieval. MMEB-V3 extends the earlier MMEB-V2 benchmark with 111 additional tasks, for 190 tasks in total, adding audio tasks (classification, cross-modal retrieval, temporal grounding), text retrieval tasks (instruction-following, reasoning, long-context and multi-condition retrieval) and agent tasks (tool retrieval, GUI control and agent memory retrieval).
It also introduces OmniSET, Omni-modality Semantic Equivalence Tuples, a diagnostic set that groups semantically equivalent instances across text, image, video and audio for controlled analysis of how well a model follows modality-specific instructions.
The benchmark's dataset is distributed on Hugging Face, and a setup script materializes the downloaded archives into the directory layout the evaluation scripts expect.