Claw-Eval
A human-verified evaluation harness for scoring large language models as autonomous agents, requiring a model to pass the same task in three independent trials before it counts as a success.
Claw-Eval is an evaluation harness for testing large language models acting as autonomous agents, built around 300 human-verified tasks across nine categories and more than 2,000 grading rubrics covering completion, safety and robustness. Results and individual task cases are published on a public leaderboard and website.
Its primary metric requires a model to pass a task consistently across three independent trials rather than crediting a single lucky run, and only a model that meets the success criteria in all three trials earns credit for that task. The project states it is auditing its codebase toward full reproducibility of every published leaderboard result, and re-triggers evaluations when network or API instability disrupts a trial, to keep exactly three trajectories per task.
It integrates with EvalScope, which can run Claw-Eval against any OpenAI-compatible model endpoint from the command line or from Python, with saved predictions, scoring and standardized reports.