Video-MME-v2
Video-MME-v2 is a benchmark for evaluating video understanding in multimodal large language models, built from more than 3,300 human-annotation hours across a three-level progression from information retrieval to temporal reasoning.
Video-MME-v2 follows up the original Video-MME benchmark, which became a standard evaluation set for frontier video-understanding models but has since saturated as model capability grew faster than the gap between leaderboard scores and real user experience narrowed. The new benchmark, built on more than 3,300 human-annotation hours, redesigns evaluation around a three-level progression: multi-point information aggregation across frames, audio and subtitles; temporal understanding of dynamic evolution and causal relations; and temporal complex reasoning that combines multimodal timing with outside world knowledge.
The repository includes a standalone evaluation script reporting both a grouped, non-linear score and plain accuracy, with task types broken out for several logic groups, and the EvalScope tool supports one-command evaluation of any OpenAI-compatible model endpoint against the benchmark with standardized reports. The project accompanies a paper, a public leaderboard and a dataset release.