AI evals

A human-verified evaluation harness for scoring large language models as autonomous agents, requiring a model to pass the same task in three independent trials before it counts as a success.

• AI evals

A command-line tool that automatically records every action a coding agent takes inside a project, so a change can be traced to the exact prompt that produced it and any step can be inspected afterward.

• AI evals

An open-source agent engineering platform that traces, evaluates and monitors AI agents so a team can debug them together, available as a managed cloud service or self-hosted with Docker.

• AI evals

An Unreal Engine 5 simulation platform for developing and evaluating LLM and vision-language model agents in realistic physical and social city environments, from a lightweight starter map to fully custom worlds.

• AI evals

A browser-hosted, fully programmable Android simulator for training and evaluating mobile GUI agents, with many parallel instances, deterministic task judges, and results that carry over to real devices.

• AI evals
• Mobile

A lightweight, real-time debugging tool for AI agents that traces every model and tool call through an OpenAI-compatible proxy, with a local UI for inspecting and analyzing interactions.

• AI evals
• Debugging

A security-governance layer for OpenClaw and other agentic AI runtimes that scans capabilities before use, inspects runtime traffic, and exports durable audit evidence.

• AI evals
• Security

An open-source benchmark that evaluates browser AI agents on everyday online tasks such as booking travel, ordering food and applying for jobs, across a set of live websites.

• AI evals
• Browser automation

A tool for generating synthetic training data for language models and agent evaluations, combining reasoning traces with tool-calling patterns through topic-graph-guided sample generation.

• Models
• AI evals

A Python framework for LLM security research that builds and runs jailbreak attacks against language models from reusable components, then scores how well each attack worked.

• AI evals
• Security

A reproducible benchmark and environment, running on a live Android emulator, for building and evaluating autonomous computer-control agents across 116 hand-crafted tasks in 20 real apps with millions of randomly generated task variations.

• AI evals
• Mobile

A benchmark and CLI that tests whether AI agents can architect and implement a complete codebase reproducing a compiled binary's behavior, given only that binary and its documentation.

• AI evals

A terminal and browser tool that reads a Claude Code or Codex session transcript and draws it as a live, replayable flow graph of the main agent, the agents it spawns, and the tools each one runs.

• Coding agents
• AI evals

A cross-platform desktop app, written in Rust and React, that iterates over combinations of local Ollama models, prompts and inference parameters and lets a user visually compare and A/B test the results.

• Inference
• AI evals

Infrastructure for autonomous AI agent organizations that run coding-agent experiments against a grader, share knowledge, and evolve across isolated workspaces, integrated with Claude Code, OpenCode, Codex, Cursor Agent and Kiro.

• Agent frameworks
• AI evals

ClawKeeper is host-agnostic safety middleware between an AI agent and its tools, blocking risky shell, Python, browser, filesystem and network calls, redacting sensitive results, and escalating harder decisions to a Watcher that learns new guard patterns.

• AI evals
• Security

AgentLens is a trust-first marketplace and workspace for AI agents where sellers publish an agent for others to rent under a time-limited contract, buyers compare execution boundaries and audit evidence, and a six-dimensional audit framework verifies trust claims.

• Agent frameworks
• AI evals

Nemo Skills is NVIDIA's collection of pipelines for improving large language models: synthetic data generation, model training, and evaluation across benchmarks in math, code, science and instruction following, scaling from a workstation to a Slurm cluster.

• Models
• AI evals

Harden.Run AIF is a local safety layer for coding agents that installs native hooks in tools such as Claude Code, Codex, Cursor and Gemini CLI to block risky actions before they run, with a local dashboard and anonymous, code-free operational telemetry.

• Agent skills
• AI evals

numbat gives visibility into what AI coding agents do on a developer's machine, watching supported desktop, CLI, IDE and gateway agents through local hooks, logs and session files, and can optionally block a risky action before it runs.

• AI evals
• Security

skill-up is a CLI that evaluates Agent Skills, agents and their workspaces by running declarative test cases against an agent engine, grading the response and any workspace changes, and producing reports locally or in CI.

• Agent skills
• AI evals

Arbor is an autonomous research agent that is given a benchmark and a goal, then proposes hypotheses, edits code, runs real experiments, and keeps only the changes that hold up on held-out data, building a tree of hypotheses instead of forgetting what failed.

• Agent frameworks
• AI evals

Hivemind is a Python-first runtime layer that gives multi-agent applications persistent run history, ordered step and tool-call records, and streamed execution events, without replacing orchestration frameworks such as LangGraph, AutoGen or CrewAI.

• Agent frameworks
• AI evals

AxisAgentic is an extensible runtime for long-horizon AI agents that also records the trajectories those agents produce, so the same execution traces can support replay, benchmark evaluation and supervised fine-tuning data export.

• Agent frameworks
• AI evals