AI evals

A deterministic authorization gateway that sits between an AI agent and its tools, tracing where each parameter of a side-effect call came from, and blocking the call or asking a human when the source does not check out.

• Agent skills
• AI evals

A reproducible benchmark from Benchmark Heaven that ranks Jev-class typed decision models on intelligence, calibration, speed, and cost, with a published harness, scoring rules, and results.

• AI evals

A co-evolutionary adversarial red-teaming tool for AI agents that uses quality-diversity evolutionary search to autonomously breed edge-case scenarios, testing whether an agent with tool access can be made to hallucinate, deadlock or leak funds.

• AI evals
• Security

Evaluations and guardrails for AI agents that replace an LLM-as-judge with typed decision models, scoring every metric for a trace in one request that costs a fraction of a cent and returns in a few hundred milliseconds.

• AI evals

An engineering research agent that runs a discover, read, synthesize, verify and review loop over technical questions, reading sources directly rather than inferring from titles, and publishing its own scored verifier benchmark, weaknesses included.

• Agent skills
• AI evals

An open framework for governed AI development that redacts personal data, blocks prompt injection, audits agent tool calls, and tracks EU AI Act compliance, installed as a proxy or a small Python SDK.

• AI evals
• Security

An open-source CLI that benchmarks any OpenAI-compatible inference endpoint, such as vLLM, SGLang, Ollama or LMDeploy, to measure real cost per million tokens, with statistics designed not to call a winner from noisy or overlapping evidence.

• AI evals

An offline AI agent security and governance scanner that checks a codebase and its installed agents against 80 deterministic rules, with no LLM involved in the evaluation.

• AI evals
• Security

A 1,350-image benchmark for evaluating document-parsing robustness against real-world captured pages, derived from OmniDocBench with physical printing, deformation and photography.

• RAG
• AI evals

Runs the same document corpus and question set through 19 retrieval architectures and reports citable, statistically tested evidence for which one fits a given dataset best.

• RAG
• AI evals

A local sidecar that turns an AI agent's completed execution traces into reviewed memories, reusable skills and evidence-backed training data, gating every proposed change behind human review instead of silently mutating agent state.

• Agent memory
• AI evals

An independent, reproducible benchmark that compares APIs choosing one option from a list and returning a confidence score, measuring accuracy, latency, cost, failure modes and calibration across a decision model and several constrained LLMs.

• AI evals

A self-hosted AI-text detector that runs 23 independent detection engines on CPU and combines their scores into one calibrated forensic result, with published accuracy figures and its own documented failure modes.

• AI evals

A self-hosted evidence store for coding agents that captures model calls, code changes, developer decisions and check results, so teams can compare models, reuse prior session context, and build training datasets.

• AI evals

A monitoring tool that watches the traffic between AI agents and tool calls to catch coordination failures a normal tracing tool misses, such as silent loops, repeated work, runaway subagent fan-out and hung or retrying tool calls.

• AI evals

A library that logs the actions AI agents take in production as a tamper-evident, cryptographically hash-chained record, so teams can prove what an agent did and detect if a log was altered afterward.

• AI evals

A CLI-first, local-first observability tool for LLM applications that traces and stores LLM calls, vector-store queries and LangChain steps in SQLite, then uses an LLM of your choice to help investigate the root cause of failures.

• AI evals
• Monitoring

Reference implementations from Vector Institute's Agentic AI Evaluation Bootcamp, covering agent evaluation fundamentals and four example agents evaluated with LLM-as-judge metrics, batch evaluation and a misalignment probing experiment across several commercial models.

• AI evals
• Tutorials

A runtime governance proxy that sits between an AI agent and its model API, detecting and blocking prompt injection and instruction-authority transfer from web pages, emails, documents and tool output before the agent acts on it.

• AI evals
• Security

A community-run registry for AI agent memory-system benchmark results, requiring per-question evidence records, a fixed reader, judge and prompt set per version, and reproducible harnesses instead of self-reported scores.

• AI evals

A stateless policy kernel that gates AI agent tool calls with five verdicts (allow, deny, dry run, approve required or transform), adds crash-safe checkpoints and a tamper-evident audit trail, and can wrap any agent harness or run a minimal loop of its own.

• Agent frameworks
• AI evals

A pure-Python decision engine that scores LLM responses for attribution, specificity, relevance and context quality to detect hallucinations and output an accept, review or reject verdict.

• AI evals

An open-source visual canvas and runtime for building a governance and reliability control layer around an AI agent, compiling one flow specification to LangGraph, CrewAI, Mastra or the Microsoft Agent Framework and tracing every decision in Langfuse.

• Agent frameworks
• AI evals

A continuous-evaluation pipeline for multi-agent systems built with Google's Agent Development Kit and the A2A protocol on Cloud Run, deploying agents to shadow revisions and running automated evaluation suites to drive data-driven deployment decisions.

• AI evals