Skillgrade
A tool for writing automated tests for AI agent skills, checking that a coding agent discovers and uses a skill correctly across repeated trials, with deterministic and LLM-judged graders.
Skillgrade evaluates whether an AI coding agent correctly discovers and uses an Agent Skill. Run inside a skill's directory, it scaffolds an evaluation file describing tasks and graders, using an available Gemini, Anthropic, or OpenAI API key to generate AI-assisted tasks and graders, or a commented template when no key is available.
Trials run the target agent, automatically detected from the configured API key or set explicitly, against the generated tasks inside a Docker container, and results can be reviewed from the command line or in a small web report. Preset trial counts range from a quick five-trial smoke check to a thirty-trial regression run for higher-confidence detection of a skill's failure rate, and a CI mode exits non-zero when the pass rate falls below a set threshold.
It requires Node.js and Docker, and installs as a global npm package.