AI evals

A benchmark toolbox for evaluating text and multimodal embedding and retrieval models across languages, with an interactive leaderboard and a command-line runner.

• RAG
• AI evals

A curated list of articles, playbooks, benchmarks, specifications, and open-source projects for harness engineering: shaping the environment around AI agents so they work reliably.

• AI evals
• Curated lists

An open-source evaluation toolkit for large vision-language models, supporting one-command evaluation across more than 220 models and 80 benchmarks with generation-based scoring.

• AI evals

An open-source observability platform for AI agents that groups failing traces into tracked signals, dispatches a coding agent with the relevant context to fix the underlying bug, and replays fixes against real failures.

• AI evals

A free desktop app and open-source Python library for building, evaluating and optimizing AI systems, covering evals, RAG, agents, fine-tuning and synthetic data in one workbench.

• Agent frameworks
• AI evals

Observability and policy enforcement for AI agent harnesses, hooking coding CLIs and chat gateways to capture every run and block dangerous tool calls before they execute.

• AI evals
• Security

An open-source machine learning platform that unifies training, fine-tuning, chat and evaluation of models into one interface, running locally or scaled to GPU clusters.

• Models
• AI evals

A framework from the creators of Terminal-Bench for evaluating and optimizing coding agents and language models across thousands of parallel sandboxed environments.

• AI evals

An open-source edition of a developer platform for the full lifecycle of AI agents, covering prompt development, evaluation and observability.

• Agent frameworks
• AI evals

An observability and devtool platform for AI agents, giving Python developers session replay, LLM cost tracking, and integrations with agent frameworks such as CrewAI, AutoGen and LangGraph.

• AI evals

A benchmark that evaluates whether a language model can generate a patch resolving a real GitHub issue, given the codebase and the issue description.

• AI evals

An open-source AI gateway and LLM observability platform that logs, routes, and analyzes requests across model providers with a single line of code.

• LLM gateways
• AI evals

A Python toolkit that intercepts every AI agent tool call, message, and delegation in application code to enforce policy, identity, and sandboxing before it happens.

• AI evals
• Security

An AI-native proxy server and data plane for agentic applications that centralizes LLM routing, agent orchestration, observability, and guardrails.

• LLM gateways
• AI evals

A Python framework for adding input/output guards to large language models, detecting and mitigating risks and generating structured data from LLM output.

• AI evals

An LLM evaluation platform supporting models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek and others across more than 100 datasets covering knowledge, reasoning, coding, science and safety.

• AI evals

An LLM vulnerability scanner that probes a model for hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks, in the spirit of nmap or Metasploit but for language models.

• AI evals
• Security

An end-to-end platform for building foundation models: fine-tuning, evaluating and deploying open-weight LLMs such as Qwen, Gemma and Llama, with reinforcement learning, tool use and MCP integration.

• Models
• AI evals

An AI agent engineering platform combining an open-source TypeScript framework for building agents with memory, tools and workflows, and a console for observability, evaluation and deployment.

• Agent frameworks
• AI evals

A generator of fresh ARC-AGI-1-style reasoning tasks, matched to the public evaluation set's distribution, for testing frontier models on problems they have not seen before.

• AI evals

A suite of tools for building LLM applications: create flows that link prompts, LLMs and Python code, then test, evaluate and deploy them to production.

• Agent frameworks
• AI evals

An open-source AI observability and evaluation platform for tracing, evaluating and troubleshooting LLM applications, with an integrated debugging agent and a remote MCP server for querying traces.

• MCP servers
• AI evals

A framework for few-shot evaluation of language models across a large, config-driven library of benchmark tasks, with pluggable model backends, filters and metrics.

• AI evals

An open-source evaluation framework for large language model applications, agents and RAG pipelines, similar to Pytest but specialized for scoring outputs with metrics such as answer relevancy, hallucination detection and task completion.

• AI evals