AI evals

A streaming integrity monitor that detects decoding corruption in an LLM's output, such as repetition loops or language drift, while the answer is still generating, so the host can cut the stream and regenerate before the user sees it.

• AI evals

A benchmark and toolkit for skill-retrieval-augmented LLM agents that treats reusable agent skills as a large corpus to retrieve from, rather than listing every skill in the prompt.

• RAG
• AI evals

A Windows-first reliability plugin for the OpenCode desktop agent that adds edit boundaries, command evidence, and verification gates around its AI-generated diffs.

• Coding agents
• AI evals

A command-line tool that independently reruns a repository's own test commands after an AI coding change, then reports what actually ran and what remains unverified.

• AI evals
• Testing

A benchmark and code release for evaluating how LLMs corrupt documents over long-horizon delegated editing, simulating round-trip edits across 52 professional domains such as source code and accounting ledgers, scoring 19 LLMs by document recovery.

• AI evals
• Papers

An open-source safety layer for autonomous AI agents that groups background activity by originating request, surfaces scope drift and failed verification for review, lets a user set per-tool allow, ask or block policy, and creates recovery points.

• AI evals

A snapshot-testing tool for AI agents that records the tools an agent calls, in what order and with what output, then flags any drift from that recorded baseline in CI, working with LangGraph, CrewAI, OpenAI Assistants and Anthropic-based agents.

• AI evals
• Testing

A public, symptom-first registry of LLM serving-path bugs that silently produce wrong benchmark numbers, covering chat templates, tool parsers, quantization kernels and eval harnesses, each entry with a reproduction status and the check that catches it.

• Inference
• AI evals

A TypeScript framework for loop engineering that runs an agent toward a stated goal in rounds, with a separate skeptical, read-only verify agent deciding when the goal is actually met.

• Agent frameworks
• AI evals

A reproducible, self-hosted platform for benchmarking local large language models with versioned question sets, containerized programming tests, rule-based and optional AI-judge scoring, and auditable reports.

• AI evals

An evaluation library for African-language and code-switching NLP systems, with metrics for dialectal robustness, terminology accuracy and hallucination that mainstream benchmarks do not measure.

• AI evals

A frozen-task, frozen-hardware public leaderboard for speedrunning LoRA fine-tuning of small language models, with every record independently re-verified on a cloud GPU sandbox.

• Models
• AI evals

A human-generated, multi-turn benchmark and four document corpora for evaluating retrieval-augmented-generation systems on realistic, multi-turn conversations.

• RAG
• AI evals

A plugin library that lets developers and researchers inspect, analyze, and steer the internal workings of large language models running on the vLLM inference engine, such as their attention and activations.

• AI evals
• Libraries

An NVIDIA AI Workbench example that adds an agentic layer to a RAG pipeline, routing to live web search when retrieved context is thin and evaluating each response for relevance and hallucination.

• RAG
• AI evals

A macOS menu-bar tool that checks whether Codex is actually using the model you selected, by reading Codex's own per-turn model records and periodically fingerprinting the live model with a bias test on random-number generation.

• AI evals
• Desktop utilities

A multi-language runtime that adds visibility and policy control over an agent's tool and LLM calls without modifying the existing agent stack, exporting events and traces to observability tools.

• AI evals

A free, open-source Mac app for AI safety and interpretability research, letting a user inspect a model's internal activations, run probes and sparse autoencoders, and test a hypothesis interactively.

• AI evals

An official OWASP project that screens an AI agent's memory writes against a policy, blocking prompt-injection-style attempts to poison persistent memory that survive a context reset.

• Agent memory
• AI evals

A local-first, proof-carrying agent runtime that records a cryptographically verifiable replay of every model call and tool observation, letting a developer fork a past run at any turn to try a different model or prompt.

• Agent frameworks
• AI evals

A harness for agentic data science that runs an analysis multiple independent ways and checks whether the results converge, instead of trusting a single AI-generated model and conclusion.

• AI evals
• Data engineering

A Harbor-compatible framework for evaluating coding agents such as Claude Code, Codex and Cursor CLI in sandboxed Docker or Modal environments, with support for installing agents in air-gapped tasks and a trajectory viewer.

• AI evals
• Testing

A native macOS app for benchmarking and comparing local large language models across any OpenAI-compatible backend, with real-time hardware telemetry alongside inference performance history.

• Inference
• AI evals

An autonomous research loop that has an AI agent iterate on a fixed vision-language model's training code, evaluate each change against the official DocVQA benchmark, and keep only changes that produce a real measured improvement.

• Models
• AI evals