Back to directory

AI Observer

A self-hosted, single-binary observability backend for local AI coding assistants that tracks token usage, cost, latency and error rates across tools such as Claude Code, Gemini CLI and OpenAI Codex CLI in one dashboard.

AI Observer

AI Observer ingests telemetry from local AI coding tools, either by watching each tool's session files directly with no OpenTelemetry configuration needed, by importing past session files with cost calculated after the fact, or by receiving standard OTLP data over HTTP in JSON or protobuf form. A real-time dashboard, updated over WebSocket, shows token usage, API latency, error rates and session activity, with a drag-and-drop builder for saved widgets.

Embedded pricing data covers Claude, Codex and OpenAI, Gemini and GitHub Copilot model aliases, and OpenCode's own plugin supplies its cost metrics natively. Storage and queries run on DuckDB for fast analytics over large datasets, and the whole application ships as one roughly 54MB binary with an embedded frontend, or as small multi-architecture Docker images, so no external services are required to run it locally.

More in AI evals

See all 269
  • Provenance gate

    github.com

    A deterministic authorization gateway that sits between an AI agent and its tools, tracing where each parameter of a side-effect call came from, and blocking the call or asking a human when the source does not check out.

  • JevBench

    benchmarkheaven.com

    A reproducible benchmark from Benchmark Heaven that ranks Jev-class typed decision models on intelligence, calibration, speed, and cost, with a published harness, scoring rules, and results.

  • A co-evolutionary adversarial red-teaming tool for AI agents that uses quality-diversity evolutionary search to autonomously breed edge-case scenarios, testing whether an agent with tool access can be made to hallucinate, deadlock or leak funds.