ChainForge
A visual, node-based environment for comparing prompts, models, and settings across multiple LLMs and scoring their responses.
ChainForge is a visual, data-flow environment for testing and comparing prompts across large language models. It lets a user query several LLMs at once with the same prompt variations, compare response quality across prompts, models, and settings side by side, and define evaluation metrics to score results automatically.
It ships with example evaluation flows, including 188 generated from OpenAI's evals benchmark suite, to show what is possible before building a custom flow. Built-in generative AI features can create synthetic test tables and starter scoring code. It can be installed locally with Python, where it can load API keys from environment variables, run Python-based evaluators, and query models hosted locally through Ollama, or tried with a reduced feature set directly on the web.
ChainForge is built on ReactFlow and Flask, and is open source under the MIT license.