AI evals

An open-source red-teaming and evaluation framework for AI agents that fires 300 curated adversarial prompts across an OWASP-inspired ten-category agent-security taxonomy at any HTTP endpoint or Python callable, scoring responses without further LLM calls.

• AI evals
• Security

A Python framework for evaluating safety risks in multi-agent systems, wrapping an existing MAS to run structured checks across 20 risk types such as prompt injection, tool misuse, and sandbox escape, and producing evidence-backed safety reports.

• AI evals
• Security

A benchmarking tool for local large-model inference that measures prefill and decode speed, offering a browser-only version and a Python backend version that removes the browser's low connection-concurrency limit for higher-concurrency testing.

• Inference
• AI evals

A benchmark generator that creates multi-hop reasoning problems over shuffled hash pairs, used to evaluate how well a language model's long-context architecture retrieves and chains facts across very large prompts.

• AI evals

A benchmark suite that measures the runtime of Apple MLX operations and layers across Apple Silicon chips and select CUDA GPUs, compared against equivalent PyTorch operations.

• AI evals

An open-source framework for quickly building, evaluating and reproducing retrieval-augmented generation systems, covering text-based, multimodal and web-accessible scenarios with an end-to-end pipeline from data preparation to evaluation.

• RAG
• AI evals

A real-time instrument that visualizes what a language model is internally 'thinking' word by word during a live conversation, using the Jacobian lens on residual-stream activations.

• AI evals

A research ablation study that diagnoses why GRPO training collapses on long-horizon, multi-tool conversational agents and proposes a combined PRM-Lite and LATA method that achieves a 37 percent relative gain over vanilla GRPO on the tau-bench airline benchmark.

• AI evals
• Papers

A desktop app that proxies MCP tool call traffic between AI apps such as Cursor, Claude, VS Code and Windsurf and their MCP servers, checking each request and response against a set of signatures and asking the user before allowing anything it flags as harmful.

• AI evals
• Security

An evolutionary runtime that breeds and grades populations of autonomous coding agents against a benchmark over repeated generations, evolving each agent's system prompt, code, tools and dependencies rather than just its output.

• Agent frameworks
• AI evals

An open-source control plane for centrally managing MCPs, agent skills and assistants across a company, adding fine-grained permissions, real-time policy enforcement over prompts and responses, and full observability of tool calls, token use and cost.

• MCP servers
• AI evals

A self-hosted, single-binary observability backend for local AI coding assistants that tracks token usage, cost, latency and error rates across tools such as Claude Code, Gemini CLI and OpenAI Codex CLI in one dashboard.

• AI evals
• Monitoring

A benchmarking library for retrieval-augmented generation systems, focused on question answering, that standardizes how retrievers, rerankers and language models are compared across many datasets.

• RAG
• AI evals

A Python library of reliability metrics for AI agents in production, designed to catch behavior such as declining decision confidence or a growing pile of unapproved actions that conventional infrastructure monitoring reports as healthy.

• AI evals
• Monitoring

An endpoint-detection-and-response system built for AI agents that evaluates an agent's file access, process execution and network activity against the same policy engine as ordinary endpoint events, denying anything a policy does not explicitly allow.

• AI evals
• Security

A policy engine that intercepts an AI agent's destructive or expensive tool calls, routes them to a human for remote approval, and records every decision in a signed ledger.

• AI evals
• Security

A config-driven tool for running experiments and evaluations of RAG search-and-response quality, testing different Azure AI Search and Azure OpenAI hyperparameters to find the best combination.

• RAG
• AI evals

A standardized environment platform for agentic AI development and reinforcement learning, treating benchmarks, tools and even other agents as a single, swappable kind of environment.

• Agent frameworks
• AI evals

Open-source infrastructure for world models, unifying inference and benchmark evaluation across video generation, 3D/4D representation and embodied-action models behind one TUI, CLI and Studio interface.

• AI evals
• Generative media

A self-hosted 2D visualization of AI agents as characters in a virtual office, showing what agents connected through OpenClaw or Hermes gateways are doing in real time.

• Coding agents
• AI evals

A Bun CLI that ingests Claude Code and Codex session transcripts, generates LLM-written daily summaries, and serves a browsable web journal with a calendar view and full-text search.

• Coding agents
• AI evals

A passive, offline-by-default tool that builds a decision graph linking every git commit to the coding-agent prompt that produced it, with drift detection, a DLP audit of what agents read, and signed provenance attestations.

• AI evals
• Security

A unified multi-backend, multi-device benchmarking utility for Transformers, Timm, PEFT, Diffusers and Sentence-Transformers, covering inference and training under Optimum's hardware optimizations and quantization schemes.

• AI evals

Video-MME-v2 is a benchmark for evaluating video understanding in multimodal large language models, built from more than 3,300 human-annotation hours across a three-level progression from information retrieval to temporal reasoning.

• AI evals