MTEB
A benchmark toolbox for evaluating text and multimodal embedding and retrieval models across languages, with an interactive leaderboard and a command-line runner.
MTEB is a multimodal toolbox for evaluating embedding and retrieval systems across languages and modalities. It installs with pip or uv and can be run either from Python or from a command-line interface that takes a model, a task and an output folder.
It covers areas such as classification, clustering, retrieval, reranking, semantic textual similarity, bitext mining and multilingual and low-resource-language evaluation, and results can be compared against an interactive public leaderboard. Documentation covers defining custom models, selecting tasks and benchmarks, running evaluations with caching, and loading existing results, and contributors can submit new models, tasks or benchmarks to the project.
It is written in Python and released under the Apache 2.0 license.