Tracely
A trace-native CI and CD tool for AI agents that turns a production failure directly into a regression test, replaying the exact recorded trace on every pull request instead of relying on hand-written evaluation datasets.
Tracely grades every AI agent trace as it arrives in production, clusters failures into issues, freezes the bad runs as hermetic, replayable test cases, and can block a pull request that would reintroduce one of them, alerting a team the moment any of that happens. Its premise is that a recorded failing run already is the ideal test case, rather than a dataset of hand-written questions and ideal answers that has to be invented and kept up to date as the product changes.
Traces arrive over plain OTLP, with agent-specific fields such as agent, conversation and turn promoted to indexed columns so runs group into conversation threads instead of a flat span list; evaluators appear as columns on the trace table itself rather than a separate tab. Replaying a frozen case in CI reuses recorded tool and model responses, so it costs nothing in live model calls, and a regression can notify a team over Slack, email or a custom webhook.
It self-hosts as one stack of API, worker, UI, Postgres, ClickHouse, Redis and MinIO with a single command.