AI evals

An open-source governance layer that intercepts AI agent tool calls and checks them against African financial and data-protection regulations before they execute, with adapters for LangChain, LangGraph, CrewAI and AutoGen.

• AI evals
• Security

A research framework for auditing collusion between cooperative, tool-calling LLM agents, built on the Terrarium multi-agent environment, that identifies a taxonomy of collusion types across different models, network topologies and environments.

• AI evals

A security and reliability layer for AI agent tool calls that intercepts MCP and other tool traffic through a seven-layer interceptor, flags prompt injection and schema drift, and runs automated fuzz tests against an agent's tools in CI.

• AI evals
• Security

An open-source framework for building controlled multi-agent environments to study how communication structure, shared memory and adversarial conditions affect coordination between LLM-based agents, aimed at researchers rather than production use.

• Agent frameworks
• AI evals

Published benchmark artifacts scoring Prism ML's Ternary-Bonsai-27B model on tool-eval-bench, a multi-turn tool-calling evaluation, including per-trial Markdown traces and two self-contained interactive HTML dashboards.

• AI evals

A policy and approval layer that intercepts AI agent tool calls, checks them against configured rules, and holds risky actions for human review before they execute, with integrations for MCP, LangChain, OpenAI Agents and Claude Agent SDK.

• AI evals
• Security

A self-evolving AI security research platform that publishes code-fix tasks through a honeypot, evaluates submitted fixes in a Docker sandbox with cheat detection, and exports the results as training data for LoRA fine-tuning.

• AI evals
• Security

A visualization tool that inspects the internals of open-source LLMs the way an MRI images a brain, mapping activations, weight distributions, information flow, and anomalies for a given model and prompt across five scan modes.

• AI evals
• Debugging

An open-source framework that benchmarks GPT-4, Claude, Gemini, Mistral, and Llama side by side on the same prompts, measuring accuracy, latency, per-token cost, a hallucination signal, and reasoning quality.

• AI evals

A Python library that enforces spend limits on AI agents by monkey-patching LLM provider calls, stopping runs, warning, or downgrading models before a budget is exceeded.

• AI evals

An open-source control plane that sits on top of an AI agent's own logic, deciding whether each action is allowed, pausing it for human approval, and logging proof.

• Agent frameworks
• AI evals

A pip-installable benchmark toolkit that measures routing entropy, expert utilization and collapse in sparse Mixture-of-Experts language models.

• AI evals

A CPU-first reference platform showing the systems around LLM inference: an OpenAI-compatible gateway, a deterministic eval CLI, an experiment registry and a RAG toolkit.

• Inference
• AI evals

A Swift telemetry SDK that traces AI workflows such as inference calls, tool executions and streaming generations, and exports the traces to OpenTelemetry.

• AI evals

A RAG parameter-sweep and evaluation toolkit that systematically tests combinations of embedding models, chunking strategies, and retrieval methods against a user's own data and questions, before they build a RAG application.

• RAG
• AI evals

An AI agent framework built around a deterministic governance engine that validates, authorizes, and audits every agent action, using control theory and statistical estimation to keep agents from drifting from policy.

• Agent frameworks
• AI evals

A benchmark suite that evaluates typed decision models on more than raw accuracy, measuring probability calibration, selective-risk coverage, latency and reproducibility, comparing TypeSafe Jev against a zero-shot classification baseline.

• AI evals

A production-style Retrieval-Augmented Generation pipeline built from scratch without LangChain, combining FAISS and BM25 hybrid retrieval with cross-encoder reranking and a built-in evaluation harness.

• RAG
• AI evals

A lightweight Python framework for evaluating information-retrieval systems such as RAG and search pipelines, computing standard metrics like Hit Rate, Recall, NDCG, and MRR.

• RAG
• AI evals

A constitutional governance runtime for autonomous AI agents that enforces defined rules, verified agent identities, and bounded permissions, working with any agent framework and halting or escalating automatically on a violation.

• AI evals
• Security

A deterministic AI defense framework that filters and validates inputs and outputs for an autonomous agent, combining keyword and pattern detection, a self-learning adaptive ruleset, and optional multi-model cryptographic consensus for high-assurance verdicts.

• AI evals
• Security

A one-stop NLP platform for small teams that trains, evaluates, and deploys small encoder models such as BERT and BGE, and separately runs blind, human-and-AI-scored prompt evaluations for large model prompts, without fine-tuning large models itself.

• Models
• AI evals

A verification-first layer for local LLMs that adds hard test execution, symbolic math checking, and an agentic repair loop on top of a small open-weight model, aiming to match a much larger teacher model's accuracy on checkable tasks.

• Inference
• AI evals

A benchmarking framework that measures the real-world runtime performance of GenAI applications such as chatbots, deep research agents, image generation, and live captioning under realistic conditions on consumer-grade hardware.

• AI evals