AgentOps

An observability and devtool platform for AI agents, giving Python developers session replay, LLM cost tracking, and integrations with agent frameworks such as CrewAI, AutoGen and LangGraph.

AI evals

Visit website agentops.ai

AgentOps helps developers build, evaluate and monitor AI agents from prototype to production. Initializing its client and wrapping a program records session replay analytics with step-by-step execution graphs, so an agent's calls can be inspected after the fact.

It tracks spend across LLM providers, integrates natively with CrewAI, AG2 (AutoGen), Agno and LangGraph, and can be self-hosted as a full dashboard-plus-API-backend app instead of the hosted service. Its Python SDK adds decorators for session, agent, operation, task and workflow spans, each supporting input/output recording, exception handling, async functions and generators.

It installs with pip install agentops; the app is open source under the MIT license.

More in AI evals

See all 269

Provenance gate
github.com
A deterministic authorization gateway that sits between an AI agent and its tools, tracing where each parameter of a side-effect call came from, and blocking the call or asking a human when the source does not check out.
- Agent skills
- AI evals
JevBench
benchmarkheaven.com
A reproducible benchmark from Benchmark Heaven that ranks Jev-class typed decision models on intelligence, calibration, speed, and cost, with a published harness, scoring rules, and results.
- AI evals
LIFE FORGE - The Autonomous Flight Simulator for AI Agents
github.com
A co-evolutionary adversarial red-teaming tool for AI agents that uses quality-diversity evolutionary search to autonomously breed edge-case scenarios, testing whether an agent with tool access can be made to hallucinate, deadlock or leak funds.
- AI evals
- Security

More in AI evals

Provenance gate

JevBench

LIFE FORGE - The Autonomous Flight Simulator for AI Agents

Cookies