Promptfoo
A CLI and library for evaluating and red-teaming LLM-based applications, comparing model providers side by side and scanning for security vulnerabilities from the command line or CI/CD.
Promptfoo is a CLI and library for evaluating and red-teaming LLM-based applications. It runs automated evaluations of prompts and models, compares providers such as OpenAI, Anthropic, Azure, Bedrock and Ollama side by side, and can review pull requests for LLM-related security and compliance issues through code scanning.
It also performs red teaming and vulnerability scanning to help secure an LLM application, and integrates into CI/CD pipelines to automate these checks. Evaluations run entirely on the local machine, so prompts are not sent anywhere beyond the configured model providers, and results can be viewed in a local UI or shared with a team.
It installs with npm, Homebrew or pip, or can be run without installing through npx. It is written in TypeScript, is now part of OpenAI, and remains open source under the MIT license.