DMB: decision-model benchmark
An independent, reproducible benchmark that compares APIs choosing one option from a list and returning a confidence score, measuring accuracy, latency, cost, failure modes and calibration across a decision model and several constrained LLMs.
DMB tests decision-making APIs, a structured decision model, LLMs constrained to structured output, and deterministic baselines, on typed decision tasks, measuring accuracy, latency, cost, failures, and how well confidence scores track actual correctness.
A pilot run tested 256 decisions per model across five suites with a shared budget cap, covering banking and spam classification among others. Reported results include the decision model's accuracy on banking and spam tasks at a median latency around 0.27 seconds, and comparison figures for several large language models, with per-model cost and coverage reported and the pilot's caveats and exclusions documented alongside the numbers.
The README is explicit that findings apply only to the tested endpoints, settings and datasets, and that small samples and cross-study differences limit broader comparisons.