DELEGATE-52
A benchmark and code release for evaluating how LLMs corrupt documents over long-horizon delegated editing, simulating round-trip edits across 52 professional domains such as source code and accounting ledgers, scoring 19 LLMs by document recovery.
DELEGATE-52 is a benchmark for evaluating large language models on long-horizon delegated document editing across 52 professional domains, including crystallography files, music notation, accounting ledgers and Python source code, accompanying the paper LLMs Corrupt Your Documents When You Delegate. It simulates a knowledge worker delegating edits to an LLM: given a seed document, the model performs a structural forward edit, then a second task meant to undo it, a backward edit, and round-trips are chained to simulate long interactions, for example ten round-trips meaning twenty separate LLM interactions.
The repository includes a main experiment runner that chains multiple round-trip edits per sample and model, a script for testing individual forward and backward edits in isolation, an OpenAI and Azure OpenAI model wrapper, an agentic harness where the model uses file and Python tools in a multi-turn loop, 52 domain-specific parsers and evaluators, and the prompt templates used in simulation. It evaluated 19 LLMs from six model families using a Reconstruction Score that measures how well the original document survives after several rounds of delegated editing.
The accompanying dataset, hosted on Hugging Face, contains 234 work environments across 48 domains, each with a seed document, five to ten reversible edit pairs and distractor context, and downloads automatically when running a simulation.