AI evals

A source-linked index of published AI model benchmark scores that keeps each score tied to its model release, benchmark variant and evaluation setting, without combining results from different harnesses into an invented composite score.

• AI evals

A Python library that stores an AI agent's execution history as Git commits, so every prompt, response and tool call is a version-controlled, inspectable, replayable and forkable event.

• AI evals
• Git

A benchmark that measures whether a local coding agent keeps fixing a bug when repository content, such as a README, an issue or a code comment, tries to redirect it with a hidden instruction.

• AI evals
• Security

An observability software development kit for multi-agent AI systems, extending OpenTelemetry's generative-AI semantic conventions and exporting to any OpenTelemetry-compatible backend.

• AI evals
• Monitoring

An open-source authorization engine for AI agents that checks every proposed action against policy before it runs, adding prompt-injection detection and confidence-based human review.

• AI evals
• Security

A runtime reliability layer for AI agents that runs deterministic, zero-model checks on every run to catch silent failures such as circular delegation, and can block an action before it executes.

• AI evals
• Debugging

A fail-closed execution firewall that sits between AI coding agents and the system, allowing, denying or escalating each action and writing a signed, offline-verifiable receipt of the decision.

• AI evals
• Security

An open-source, local runtime for debugging AI agent workflows built as DAG pipelines, with tracing, replay and artifact lineage rather than a framework for building agents.

• AI evals
• Debugging

A benchmark site that runs the same prompt through different combinations of model, agent harness and thinking effort, then shows the resulting runnable artifacts side by side for comparison.

• AI evals

A benchmark for evaluating whether an AI agent or memory system can answer personal, evidence-grounded questions about roughly four years of a person's own multimodal history, including images, videos and emails.

• Agent memory
• AI evals

Reproducible on-device LLM benchmarks for Apple Silicon, comparing Apple Core AI, MLX, LiteRT-LM and Core ML on the same models and harness, with every result tied to its quantization and capture session.

• AI evals

Run-aware token budget governance for multi-agent systems, enforcing a spending limit before every model call rather than reporting cost after the fact.

• AI evals

A Ruby command-line tool and companion website that runs AI language models against code-generation and program-repair benchmarks, then publishes the rankings and metrics.

• AI evals

A set of small practical scripts and task packs for comparing local language models and coding-agent harnesses, measuring raw inference speed, tool-choice reasoning and whether an agent can finish real coding tasks.

• AI evals

A self-improvement engine for AI agents that scores what an agent relied on against how a run turned out, then hands the next similar run a briefing and a recommendation instead of a blank slate.

• Agent memory
• AI evals

A permission-request hook for AI coding tools that hands each tool-execution approval decision to a language model, judged against rules written in a configuration file, with working defaults out of the box.

• AI evals
• Security

An open-source framework for testing and observing multi-agent delegation, tool execution, MCP workflows and permissions, with policy checks, replayable runs and diagrams of who delegated what to whom.

• AI evals
• Testing

A local, portable evidence ledger for research and engineering work done with an AI model, tracking claims, sources and open gaps across sessions and models so an answer can be checked rather than taken on faith.

• AI evals

A governance layer for autonomous AI work where a single YAML file declares an agent's mission, spending limits and allowed capabilities, and every task is planned, budget-checked, audited and recorded.

• Agent frameworks
• AI evals

An externally scored benchmark for machine-learning research agents, with 60 tasks across tabular, forecasting, vision, language, audio and scientific data, and a protocol that keeps holdout labels away from the agent under test.

• AI evals

A suite of tools for automated benchmarking of retrieval-augmented-generation systems, generating synthetic queries, judging answers with an LLM, and preparing datasets for reproducible testing at scale.

• RAG
• AI evals

A benchmark that measures how well frontier AI models can write Blender Python code that procedurally builds a specific 3D object, scored on executability, image similarity, 3D shape distance and a language-model judge.

• AI evals

A prompt-injection firewall for MCP that scans every tool response through four layers of detection before an agent reads it, wrapping any MCP server with no configuration change to the agent.

• MCP servers
• AI evals

A local governance daemon for AI agents that enforces declarative tool-call policies deterministically before each call runs, with identity-bound credentials and a tamper-evident audit log.

• AI evals
• Security