AI evals

A benchmarking framework for single- and multi-agent LLM systems that pairs built-in datasets with evaluators, visual debugging of tool use and accuracy, and automated workflow optimization.

• Agent frameworks
• AI evals

A research framework for measuring emergent safety failures that appear only when many AI agents interact, even when each individual agent behaves safely on its own.

• Agent frameworks
• AI evals

A framework for evaluating semantic search across a developer's own datasets, comparing embedding models, BM25 and other backends with metrics such as Accuracy@k and NDCG@k.

• RAG
• AI evals

A dual-engine benchmarking pipeline for GGUF and safetensors language models on NVIDIA GPUs, measuring both speed and answer quality with a live dashboard and exportable benchmark cards.

• Inference
• AI evals

A collection of 50 standalone browser-based tools for analyzing, refining, scoring and building prompts for large language models, each a single HTML file with no build step.

• Agent skills
• AI evals

A research toolbox for training and evaluating multimodal large language models under continual instruction tuning, where a model learns new tasks in sequence without forgetting earlier ones.

• Models
• AI evals

A community-contributed benchmark database of language-model speed and code-quality results measured on Apple Silicon Macs, organized by chip generation and configuration so someone can look up expected performance before choosing a model.

• Inference
• AI evals

A training platform that repeats a four-phase cycle to build adversarial robustness into a text-classification model without eroding the defenses learned in earlier cycles, and generates EU AI Act compliance reports.

• Models
• AI evals

An open benchmark, harness and set of embedding models for fashion product retrieval, letting a shopper find products from a photo or a plain-language description and measuring results at full catalog scale against other embedding models.

• RAG
• AI evals

A long-running, append-only benchmark that checks whether a frontier language model quietly gets worse after it ships, running the same frozen prompts and graders daily against a fixed baseline set by launch day.

• AI evals

A benchmark dataset of 1,475 document pages that evaluates document-parsing and OCR models on text, formulas, tables and reading order, matched across clean, digitally degraded and real-world scanned versions of the same pages.

• AI evals

An open, executable benchmark that scores how well large language models write 1C:Enterprise (BSL) code, judging generated code on syntax, meaning, optimization and platform fit by actually compiling and running it.

• AI evals

A lightweight, general-purpose harness for tool-using language model agents, meant as a fair execution substrate for agent benchmarks, a reference baseline, and a personal assistant runtime.

• Agent frameworks
• AI evals

An ICML 2026 benchmark that tests whether AI code agents can handle data-intensive analytical tasks, requiring an agent to discover the right files among hundreds in a Linux sandbox before writing code.

• AI evals

A defense-in-depth security toolkit for LLM agents that tracks where every piece of text came from, scans it for prompt injection, applies policy before a tool runs, and scrubs outbound secrets.

• AI evals
• Security

A measured serving recipe and benchmark comparing two 4-bit quantized checkpoints of a 27-billion-parameter Qwen model against an FP8 baseline on a single NVIDIA DGX Spark, showing where quantization helps and where it stops mattering.

• Inference
• AI evals

A research benchmark and training project for small vision-language models that spot factory procedure errors in first-person video by identifying exactly when a step starts and ends, aimed at real-time deviation warnings before an incident happens.

• Models
• AI evals

A personal collection of benchmark results for small language models and quantizations that fit on a single NVIDIA DGX Spark machine, comparing coding and evaluation scores across several models.

• AI evals

A dependency-free command-line tool that runs the same prompt against locally installed Ollama models, repeats it a chosen number of times, and saves every response to disk for comparison.

• AI evals
• CLI tools

An open source security layer for AI agents that enforces role-based access control, requires human approval for high-risk actions, and writes a cryptographically signed, tamper-evident audit log.

• AI evals
• Security

A tool for scoring local and cloud language models against custom prompts using a configurable judge model, with a bundled web dashboard for building prompts and browsing a leaderboard.

• AI evals

A single-command terminal benchmark that discovers the language models already installed in a local runner, such as Ollama, LM Studio, llama.cpp or vLLM, and measures their speed, memory use and answer quality together on a live leaderboard.

• Inference
• AI evals

A benchmark for evaluating multimodal large language models on front-end code generation, covering React, Vue, Angular and plain HTML and CSS across generation, editing and bug-repair tasks on nine hundred real webpage samples.

• AI evals

A Python SDK and set of tools that add runtime governance to AI agents: cost limits, step limits, loop detection and policy enforcement, plus an MCP wrapper and Claude Code hooks, working offline with local YAML policies.

• AI evals