AI evals

A real-time control room for observing and steering simulated societies of language-model agents, letting a user pause time, question a resident, change what happens next, and replay or export a run.

• Agent frameworks
• AI evals

A reproducible security evaluation workflow for generative AI, with one product testing an LLM or API's behavior under adversarial scenarios and a second extending that evaluation to a running agent's tool calls and file changes.

• AI evals
• Security

A Python library that detects likely hallucinations in large language model outputs using uncertainty quantification techniques, scoring each response's confidence between 0 and 1.

• AI evals

A single Python interface for running and evaluating AI agents built with different frameworks such as Agno, Google ADK, LangChain, LlamaIndex, the OpenAI Agents SDK, and smolagents.

• Agent frameworks
• AI evals

oh-my-agent is a skill pack and harness for AI coding agents that verifies their work with stop-hook test gates, artifact checks, and an independent judge, instead of trusting the agent's own summary.

• Agent skills
• AI evals

A tool that checks every structural or behavioral claim an AI makes against ground truth with deterministic tools, so a claim about a binary, an API, or a piece of code only counts once it is verified rather than asserted.

• AI evals
• Testing

A benchmark of 113 original, long-horizon software-engineering tasks across five languages, drawn from active open-source repositories, for measuring how well frontier coding agents perform on real engineering work.

• AI evals

An SLO-aware benchmarking platform that simulates realistic workloads against OpenAI-compatible and vLLM-native servers to measure LLM deployment latency, throughput and capacity limits.

• Inference
• AI evals

A CLI and set of standards for starting a new agent project, scaffolding scenario tests, versioned prompts and evaluation notebooks so agent behavior stays testable from day one.

• Agent frameworks
• AI evals

An open ecosystem where AI agents complete real-world tasks through workflow loops, get evaluated by mentor agents or humans, and turn the results into reusable training data.

• Agent frameworks
• AI evals

A real-time visualizer that turns Claude Code and Codex agent activity into an interactive node graph, so a developer can watch how an agent breaks down and executes a task.

• AI evals
• Debugging

An open-source, continuously updated benchmark research platform that tracks real-world inference performance of popular open-source serving frameworks such as vLLM, SGLang and TensorRT-LLM across current GPU hardware and models.

• Inference
• AI evals

A desktop app for agent builders that lets you prototype agent ideas, trace every model call and tool run, replay failed runs, and measure performance in one place.

• Agent frameworks
• AI evals

An open-source, self-hostable platform for evaluating, tracing, simulating and guarding LLM and AI agent applications in one feedback loop, with a low-latency Go gateway and OpenTelemetry-native observability.

• AI evals

A CVPR 2025 benchmark for evaluating document-parsing systems across 1,651 pages of diverse real-world documents, with detailed block- and span-level annotations and evaluation code.

• RAG
• AI evals

A unified, open benchmark and framework for building, orchestrating and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 140+ scenarios across 9 asset classes and specialist agents for IoT, failure modes and work orders.

• Agent frameworks
• AI evals

A benchmark suite and execution harness for testing how well AI agents complete real-world, end-to-end tasks in a real terminal environment, such as compiling code, training models or setting up a server.

• AI evals

A CLI and dashboard that benchmarks live latency and stability across hundreds of free coding LLMs from many providers and installs the fastest one straight into a coding tool.

• AI evals

An open-source observability and evaluation platform that traces LLM calls, tool use, prompts, cost, and coding-agent sessions using OpenTelemetry.

• AI evals

A framework from the UK AI Security Institute for evaluating large language models, with built-in support for prompt engineering, tool use, multi-turn dialogue, and model-graded scoring.

• AI evals

An open-source library and benchmark for routing each query to the most suitable large language model based on cost, performance, and task complexity.

• LLM gateways
• AI evals

A visual, node-based environment for comparing prompts, models, and settings across multiple LLMs and scoring their responses.

• AI evals

A benchmark, presented at NeurIPS 2024, for evaluating multimodal AI agents on open-ended tasks inside real virtualized computer environments rather than sandboxed simulations.

• AI evals

An open-source, cloud-native LLMOps platform for designing and versioning prompts, observing and troubleshooting AI operations, and delivering prompt changes instantly across an application.

• AI evals