AI evals

A local-first observability tool that reads Claude Code and Codex session transcripts to show token usage, cache performance, model choice, and cost across every machine a developer connects to a self-hosted team hub.

• Coding agents
• AI evals

Keystone is a clinical decision-evidence benchmark, built on HealthBench, that tests whether a chat assistant's recommended action changes correctly when the one fact it depends on is removed, contradicted or altered.

• AI evals

A cross-platform command-line tool that measures the real tokens-per-second throughput of local LLMs running through Ollama.

• AI evals

A benchmarking framework for evaluating the foundational components of retrieval-augmented generation systems, comparing text splitters, retrievers, and orchestrators against multiple evaluation metrics.

• RAG
• AI evals

A tool that converts a developer's Claude Code, Codex CLI, Copilot, Cursor, and Gemini CLI session transcripts into a searchable, interlinked local knowledge base, following Andrej Karpathy's LLM Wiki pattern.

• Coding agents
• AI evals

A runtime that checks an AI agent's tool calls against deterministic rules before they run, in a fraction of a millisecond per check and without calling any model.

• AI evals
• Security

A question-answering dataset and evaluation code for testing retrieval and reasoning across multiple documents in retrieval-augmented generation pipelines, with 2,556 queries whose evidence spans two to four documents each.

• RAG
• AI evals

A benchmark and paper repository for evaluating Graph Retrieval-Augmented Generation (GraphRAG) models against traditional RAG, testing hierarchical knowledge retrieval and deep contextual reasoning.

• RAG
• AI evals

A source-available control plane that enforces policy and requires human approval before an AI agent's risky tool calls, shell commands, workflows or production changes, keeping an auditable record of what was approved.

• AI evals
• Security

A survey repository accompanying a paper on how large language model benchmarking has moved from static test sets, which risk data contamination from training corpora, to dynamic evaluation methods designed to avoid it.

• AI evals
• Papers

An open-source Python package for data-driven evaluation of LLM-powered applications, with a modular metric library covering retrieval-augmented generation, code generation, agent tool use and classification.

• AI evals

A dataset and benchmark of more than 500,000 simulated company documents and 500 questions for evaluating retrieval-augmented generation systems on realistic enterprise data.

• RAG
• AI evals

An open-source tool that checks whether an AI API relay or proxy service genuinely forwards to the Claude, GPT or Gemini model it claims, by diffing probe requests against official API baselines.

• AI evals
• Security

A testing library that generates and runs more than 60 kinds of tests for NLP and LLM models, covering robustness, bias, fairness, toxicity and factuality, with automatic training-data augmentation.

• AI evals
• Testing

Adrian is an open-source runtime security monitor for AI agents that watches both an agent's actions and its reasoning traces to catch prompt injection, tool poisoning, data exfiltration and out-of-remit behavior as it happens, and can intervene before the action completes.

• AI evals
• Security

OPFOR is an open-source red-teaming tool for AI agents, LLM applications and MCP servers, generating attacks across the OWASP LLM, Agentic AI, MCP and API Security top-ten lists and judging each response with an LLM, from a command line, an IDE, or a browser extension.

• AI evals
• Security

An open protocol and gate that checks an AI agent's authority before it can complete a sensitive action, such as releasing a payment, and produces evidence that can be independently verified rather than trusting the agent.

• AI evals
• Security

VLM2Vec is the code and data interface for the MMEB family of benchmarks for embedding models, with the latest version, MMEB-V3, covering text, image, video, audio, visual document and agent-centric retrieval.

• AI evals

This is the official code release for a research paper on benchmarking long-form factuality in large language models, including a fact-seeking prompt set and an automatic evaluator for long-form model responses.

• AI evals

A tool for writing automated tests for AI agent skills, checking that a coding agent discovers and uses a skill correctly across repeated trials, with deterministic and LLM-judged graders.

• Agent skills
• AI evals

agenttrail is a local observability tool for AI coding agents that watches project files and agent activity and turns them into two live views: a project structure map and a 3D task-and-role visualization.

• AI evals

A trace-native CI and CD tool for AI agents that turns a production failure directly into a regression test, replaying the exact recorded trace on every pull request instead of relying on hand-written evaluation datasets.

• AI evals
• CI/CD

A local-first dashboard that shows what coding agents such as Claude Code, Codex and OpenCode actually did on a task, breaking each one into tool calls, files changed, tests run, time spent and estimated cost.

• AI evals

A TypeScript SDK, agent documentation and private server emulator for building and running bots inside a RuneScape-style game, built as a testing environment for coding-agent research rather than for the live commercial game.

• AI evals