safelabs-eval
An open-source red-teaming and evaluation framework for AI agents that fires 300 curated adversarial prompts across an OWASP-inspired ten-category agent-security taxonomy at any HTTP endpoint or Python callable, scoring responses without further LLM calls.
safelabs-eval tests AI agents built on frameworks such as LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google ADK, or Semantic Kernel for adversarial safety issues before production. It runs 300 curated prompts, 30 per category, across ten categories inspired by, but not identical to, the OWASP Agentic Top 10, scoring every response with pattern-based detectors rather than further model calls, and producing a structured security report with a confidence figure per finding.
It works by pointing the CLI at a supported HTTP agent endpoint or by wrapping any Python callable, synchronous or asynchronous, in its evaluation function, and results can be filtered to specific categories, summarized, or inspected record by record with a remediation hint where one applies. No modification to the agent's own code is required, and no proprietary infrastructure is needed to run an evaluation.
It installs with pip and requires Python 3.11 or newer.