ML Benchmark by Simreal
An externally scored benchmark for machine-learning research agents, with 60 tasks across tabular, forecasting, vision, language, audio and scientific data, and a protocol that keeps holdout labels away from the agent under test.
An agent studies a dataset, develops a modeling approach, runs experiments and submits predictions; this benchmark covers 60 such tasks in three difficulty tiers, each with its own research-time and compute budget, spanning tabular learning, forecasting, vision, language, audio, multimodal learning and structured scientific data.
The task definitions, evaluation protocol, scoring rule and verification tooling are public under Apache-2.0, so anyone can read exactly how a score is produced and recompute it. The evaluation runtime itself is a separate, commercially operated service that runs agents in isolated workers behind a restricted gateway, enforces budgets and a two-submission rule, and routes submissions to official scoring; test labels stay with the competition platform rather than with the benchmark's own operator, so the entity scoring a run is never the same one that could see the answers in advance. This repository is the public preview: the task catalog, protocol, scoring code and a client SDK.