Back to directory

OpenLIT

An open-source observability and evaluation platform that traces LLM calls, tool use, prompts, cost, and coding-agent sessions using OpenTelemetry.

OpenLIT

OpenLIT is an open-source observability and evaluation platform for AI applications and coding agents, built on OpenTelemetry. After starting its Docker Compose stack and adding one line to initialize its SDK, it automatically instruments supported LLM providers, frameworks, and vector databases, tracing every LLM call, tool invocation, prompt, agent step, token, cost, error, and evaluation to a local dashboard.

A separate CLI instruments coding agents such as Claude Code, Cursor, and Codex, giving an OpenTelemetry-native view of coding-agent sessions in a dedicated dashboard so a developer can see what an agent actually did during a task. Telemetry is exported over the OTLP protocol to a configurable endpoint, and a doctor command checks that the CLI's instrumentation installed correctly.

It is written in TypeScript and released under the Apache License 2.0.

More in AI evals

See all 269
  • Provenance gate

    github.com

    A deterministic authorization gateway that sits between an AI agent and its tools, tracing where each parameter of a side-effect call came from, and blocking the call or asking a human when the source does not check out.

  • JevBench

    benchmarkheaven.com

    A reproducible benchmark from Benchmark Heaven that ranks Jev-class typed decision models on intelligence, calibration, speed, and cost, with a published harness, scoring rules, and results.

  • A co-evolutionary adversarial red-teaming tool for AI agents that uses quality-diversity evolutionary search to autonomously breed edge-case scenarios, testing whether an agent with tool access can be made to hallucinate, deadlock or leak funds.