Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits
NITR is a repository-level benchmark that checks whether an AI agent's code edits stay maintainable, not just behaviorally correct, pairing multi-file change requests with hidden functional tests and structural oracles across nine maintainability dimensions.
The benchmark is built to expose a specific failure mode: an agent producing an edit that passes its tests but quietly weakens modularity, testability or architecture, such as reusing an entrypoint improperly or letting configuration sprawl. Each case ships starter source code, one or more natural-language task files describing the requested change, and an evaluator that checks both functional correctness and the structural properties the case targets.
The current public suite is predominantly C++, with a smaller but growing set of Python cases covering things such as session lifecycle, logging side effects and pricing open-closed-principle violations, and the repository reports a pass and fail matrix across twenty-three evaluated agent and model configurations on twenty-one cases.
It vendors dependencies needed to build cases and evaluators, includes local submission helpers, but does not include a hosted submission service, and accompanies an academic paper.