jevals
Evaluations and guardrails for AI agents that replace an LLM-as-judge with typed decision models, scoring every metric for a trace in one request that costs a fraction of a cent and returns in a few hundred milliseconds.
jevals runs evaluations and guardrails for AI agents using typed decision models instead of a large-language-model judge, so every metric for a given trace goes out as one request that costs a small fraction of a cent and returns in a few hundred milliseconds, making it practical to run on every trace and inside the agent loop rather than on a small nightly sample. It works with the Jev decision models through TypeSafe's or Vercel's APIs, with two smaller models that can run locally on a Mac, or with an ordinary chat model if that is all that is available, at higher cost and latency.
The project's stated motivation is that a large-language-model judge is the most expensive part of a typical evaluation pipeline, since standard metrics can involve several model calls or one call per retrieved chunk each with few-shot examples and JSON retries, and that on identical traces two frontier-model judges showed far higher run-to-run score variance than the typed decision approach in a cited comparison. Message input accepts the OpenAI chat format directly, or Anthropic content blocks and LangChain message objects.