AI evals

A TypeScript agent framework that records every read, write, decision, and tool call an agent makes as connected evidence, so a developer can trace a wrong answer back to what the model was given rather than only what the code did.

• AI evals
• Debugging

A self-hosted cognitive engine that plugs into any language model to give it persistent, tiered memory, a background self-improvement loop, and a runtime self-monitoring layer that watches for and corrects degraded output.

• Agent memory
• AI evals

A workspace for running and benchmarking local LLMs on Apple Silicon with MLX, built to measure which local model can handle routine coding operations well enough to replace paid cloud requests in an internal coding tool.

• Inference
• AI evals

An open-source safety and control layer for AI agents that blocks unsafe tool calls, pauses for human approval on risky actions, enforces per-agent cost budgets, and keeps an audit trail that can survive a crash.

• AI evals
• Security

A curated collection of research papers on GUI agent safety, organized by environment and keyword, covering benchmarks, datasets, attacks, defenses and evaluation frameworks for computer-use and mobile or desktop agents.

• AI evals
• Security
• Papers

A benchmark suite for local large language models on x86 hardware, measuring time-to-first-token, throughput, total duration and memory use across llama.cpp, LM Studio and Ollama backends.

• Inference
• AI evals

A live, public benchmark where entrants forecast what a population will do before a real release happens, with forecasts hashed and timestamped before the outcome exists and scored once it is published.

• AI evals

A Claude Code plugin that sends session telemetry, including prompts, tool calls, model generations, token usage and cost, to Langfuse for tracing and observability.

• AI evals

Enforces the rules already written in a project's CLAUDE.md, AGENTS.md or .cursorrules files, blocking AI coding agents from crossing them using semantic conflict detection rather than keyword matching.

• Agent skills
• AI evals

A reinforcement-learning training framework that composes an entire multi-agent system at each step rather than building it incrementally, paired with MASBench, a controlled benchmark for when multi-agent systems beat single-agent ones.

• Agent frameworks
• AI evals

A scenario-based benchmark for local LLM inference on Apple Silicon that measures effective end-to-end throughput, including the prefill phase, across real workloads such as agent conversations and document classification.

• Inference
• AI evals

A fork of the HumanEval evaluation harness adapted to benchmark how well large language models generate correct Python code for bio-image analysis tasks.

• AI evals

A local LLM call debugger that captures every API call an app makes to a language model during development, with full request and response detail, token usage, cost and latency, viewed in a local web UI.

• AI evals
• Debugging

A large-scale dataset and benchmark for detecting errors in multi-agent systems, with systematically generated failure scenarios and verifiable ground-truth labels across several agent frameworks.

• AI evals

A governed, local-first system for training small language models that measures each new model against a benchmark, promotes only improvements through a regression-aware gate, and archives the evidence for every decision.

• Models
• AI evals

An integration and governance layer for deployed AI agents that bounds what an agent may do, keeps inspectable evidence of what it did, and connects it to hosted operation or agent-to-agent commerce only when needed.

• Agent frameworks
• AI evals

A platform that turns an AI agent's own production traces into a fine-tuned model, benchmarked against user-defined evaluation metrics and served behind one OpenAI-compatible API, available hosted or self-hosted.

• Models
• AI evals

An autonomous agentic system that turns a natural-language evaluation goal into a verified, executable benchmark dataset, covering requirement analysis, subtask design, data grounding and quality control.

• Agent frameworks
• AI evals

This benchmark compares three memory architectures for multi-agent LLM systems - raw conversation history, vector-only RAG, and a structured context graph - on questions that require combining facts from different points in a conversation, using no LLM calls.

• Agent memory
• AI evals

NITR is a repository-level benchmark that checks whether an AI agent's code edits stay maintainable, not just behaviorally correct, pairing multi-file change requests with hidden functional tests and structural oracles across nine maintainability dimensions.

• AI evals
• Code quality

This is a community node for the n8n workflow automation platform that sends Langfuse tracing and observability data for tool-calling AI agent workflows, including reasoning steps, tool calls and structured output.

• AI evals
• Workflow automation

A reproducibility study measuring llama.cpp speculative decoding for a quantized Qwen3.6-35B model on a single RTX 3090, with re-derived figures, a coverage probe and a public errata of the study's own retracted claims.

• AI evals

An ICML 2026 benchmark and a family of domain-specialized vision-language models for understanding mechanical engineering drawings, covering recognition, reasoning and judging tasks over dense technical diagrams.

• Models
• AI evals

A lightweight RAG provenance middleware that checks, without calling an LLM, whether every claim in a generated answer is actually grounded in a retrieved source, and flags claims backed only by outdated sources.

• RAG
• AI evals