RepoGuardBench
A benchmark that measures whether a local coding agent keeps fixing a bug when repository content, such as a README, an issue or a code comment, tries to redirect it with a hidden instruction.
RepoGuardBench studies bug-fix utility and prompt-injection robustness together by placing inert, non-executing instructions inside repository artifacts, README files, issue text, code comments, test logs and agent-rule files, and then measuring whether an open-weight local coding agent stays on its actual task. It runs eighty controlled core tasks and fourteen hand-curated applied tasks against five carrier types and five attack goals, scoring each matched clean-versus-poisoned run on seven independent utility and safety metrics.
Its main experiments run locally against Qwen2.5-Coder models of different sizes on one consumer GPU through Ollama, and a deterministic, GPU-free mock pipeline exercises the whole evaluation pipeline for artifact verification. The accompanying paper was accepted at the ICML 2026 Deep Learning for Code workshop, and reports that code comments and agent-rule files are the carriers most likely to redirect an agent, with susceptibility increasing along with model capability.