jev-benchmarks
A benchmark suite that evaluates typed decision models on more than raw accuracy, measuring probability calibration, selective-risk coverage, latency and reproducibility, comparing TypeSafe Jev against a zero-shot classification baseline.
jev-benchmarks evaluates whether a model's reported probabilities are calibrated enough to support automated decisions, not just whether it picks the right label. Its first study compares TypeSafe Jev against fastino/gliner2.5-multi-v1 on zero-shot, single-label text classification with per-label probabilities, using three conditions from BTZSC with fixed model and dataset revisions, identical examples and label descriptions, a uniform negative control, and paired bootstrap confidence intervals.
Measured dimensions include accuracy and macro-F1, multiclass Brier score, negative log likelihood and expected calibration error, the maximum coverage achievable at a fixed error budget, end-to-end latency and resource use, and integrity checks such as probability-vector validation, artifact hashes and pinned revisions.
A pilot run over three held-out datasets found Jev more accurate and better calibrated on two of them, and worse calibrated, though not clearly less accurate, on the third, illustrating the kind of mixed result the benchmark is built to surface rather than hide.