VLMEvalKit
An open-source evaluation toolkit for large vision-language models, supporting one-command evaluation across more than 220 models and 80 benchmarks with generation-based scoring.
VLMEvalKit is an open-source evaluation toolkit for large vision-language models, distributed as the Python package vlmeval. It enables one-command evaluation of these models across various benchmarks without the workload of preparing data separately for each one, adopting generation-based evaluation for every model and providing results from both exact matching and LLM-based answer extraction.
It supports over 220 large multi-modality models and more than 80 benchmarks. Recent updates add a custom split-thinking function so models with a distinct thinking mode have their reasoning separated from their final answer before scoring, support for saving predictions in TSV format so very long model responses are not truncated by spreadsheet cell limits, and refinements to its option and text answer extractors that route more evaluation through LLM-based choice extraction for measurable, if modest, accuracy gains on multiple-choice benchmarks.