Back to directory

ManagePrompt

A local LLM call debugger that captures every API call an app makes to a language model during development, with full request and response detail, token usage, cost and latency, viewed in a local web UI.

ManagePrompt

ManagePrompt has two parts: a Go-binary CLI that runs a local server and web UI for viewing captured requests, and an npm package that instruments an application to capture and forward its LLM API calls to that local server. It works with the Vercel AI SDK through a middleware wrapper, with any OpenAI- or Anthropic-compatible SDK through a capture wrapper that auto-detects the provider, and with manual logging for full control over what is sent.

Captured data includes the full request and response bodies, a visual request flow with tool-call visualization, latency, token usage split into input, output, cache-read and cache-write counts, and a cost estimate based on models.dev pricing. The CLI can be installed with Homebrew, with Go, or built from source, and defaults to serving its dashboard on port 54321.

More in AI evals

See all 269
  • Provenance gate

    github.com

    A deterministic authorization gateway that sits between an AI agent and its tools, tracing where each parameter of a side-effect call came from, and blocking the call or asking a human when the source does not check out.

  • JevBench

    benchmarkheaven.com

    A reproducible benchmark from Benchmark Heaven that ranks Jev-class typed decision models on intelligence, calibration, speed, and cost, with a published harness, scoring rules, and results.

  • A co-evolutionary adversarial red-teaming tool for AI agents that uses quality-diversity evolutionary search to autonomously breed edge-case scenarios, testing whether an agent with tool access can be made to hallucinate, deadlock or leak funds.