Decision model benchmark methodology — JevBench v1.6.1

Release and measurement dates

Published release v1.6.1, initially published 2026-10-06. The versioned leaderboard retains its release context. Additions stored in the results file: a6 (2026-10-07), a7 (2026-10-07), a8 (2026-10-07), a11 (2026-10-08), a12 (2026-10-08); amended through 2026-10-08; latest row measurement 2026-10-07. Cite source_sha256 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1 for these exact result bytes. The live board may also include later dated API reruns and presentation revisions.

The release draw contains 1,200 sealed and 300 public decisions. Self-hosted systems answer 1,500 decisions. The v1.6.1 full-set API rows answer the same set. Older subsets and later reruns are explicitly labeled and may be equated. Missing or refused answers are counted by the scorer, rather than silently removed.

Capability and Composite are different scores

Capability: Arithmetic mean of Intelligence and Calibration; ranked models within both official caps. Headline on the open-weights board. The cost cap is USD 0.0646 per 1,000 decisions, and the adjusted median-latency cap is 1.23 seconds. Eligibility is separate from the raw score.

Composite: Weighted harmonic mean of Intelligence, Calibration, Speed and Cost, multiplied by squared penalties when Intelligence is below its configured floor, or Speed or Cost is below 50. Equal 25% weights do not mean an arithmetic average.

Official option A uses weights Calibration 25%, Cost 25%, Intelligence 25%, Speed 25%; Intelligence floor 50. A zero axis gives zero Composite; a missing axis gives no score.

H = Σw / Σ(w / axis). Composite = H × min(1, I / 50)² × min(1, Speed / 50)² × min(1, Cost / 50)².

Intelligence is chance-corrected decision accuracy, with tier weighting and a public/sealed gap adjustment. Calibration evaluates probability distributions; it is not simply accuracy among confident answers. Noul handling, per-type normalization and equating are versioned in the scorer. The four normalized axes are on a 0–100 scale; actual seconds and USD remain separate values.

Latency: what is measured and what is estimated

We time every request and use the median (p50) and the 95th percentile (p95). API models are timed end to end from our measuring host against the vendor endpoint, so their latency includes the network round trip. Self-hosted (open-weights) models run on hardware we rent and operate (a GPU, or a CPU host for the smallest models) and are timed over loopback, where there is no network at all. We tag that value “on-GPU, no network” (or “on-CPU, no network”) wherever it appears, and it is not directly comparable with API latency.

Those two numbers are not comparable as they stand, so for every self-hosted model the ranking and the latency cap use the same estimate that adds the network: 2 × measured p50 + 150 ms (the same adjustment is applied to p95). This is an assumption, not a measurement: it stands in for the extra time an API call to the same model would spend on the network and the serving layer. The Speed axis, the Jev-class latency cap and every rank use the estimate; the measured loopback value is shown next to it on the board, the model pages and in the JSON (p50_s_measured, latency_basis).

Hosted endpoint timing on request. A model author may provide a hosted endpoint for their model. We then time it exactly like an API, network included, with no assumption, and that measured latency replaces the estimate for that row, noted in the revision history. Contact info@productivity-boost.com or note it in your submission.

Calibrating the assumption (planned, not yet applied). The 2 × + 150 ms rule is a fixed assumption. We intend to check it against real end-to-end timings of self-hosted models behind public endpoints, for example endpoints that model authors provide. Any change to the rule will be announced in the revision history and applied to all self-hosted rows at once, never switched silently for single rows. If we use timings from decisionmodels.io, we will say so, because decisionmodels.io hosts some of these models.

Reproducibility and model versions

Scorer and public benchmark repository · Board source · Release manifest and scorer hashes · Published aggregate artifact.

Artifact SHA-256: 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1. The data card documents model pins, measurement dates, price bases and latency adjustments. Unknown pins and dates remain null; hosted service names do not guarantee immutable weights.

Sealed questions, identifiers and predictions are not downloadable. Public-split experiments can be reproduced, and aggregate Composite scores can be recalculated from the published axes. Complete independent reruns of the sealed evaluation require the official evaluation process.

Limitations and historical methods

Coverage is finite and uneven across languages and domains. Cost is modeled from a stated hardware or list-price basis; latency depends on the measured runtime and location. Confidence, coherence, robustness and real-world error costs need additional workload-specific checks. Training on the public split must be disclosed, and a sealed split is not a guarantee of no prior exposure.

Historical releases retain their original scoring definitions and measurement sets. Do not compare scores across method versions as if they were the same experiment. ImageJevBench v0.3.0 has a separate image pool and frozen eligibility envelope; core and computer-use tracks remain separate. Its reserve size is not the number of decisions answered by each model.

Another project named JevBench tests metamorphic coherence. It is independent of Benchmark Heaven and measures consistency rather than this board's accuracy/cost/latency score.