Hosting = where inference runs; company = where the provider or lab is registered.
EU hosting means the route's inference runs inside the EU: an EU region, AWS Bedrock's EU cross-region (geo) profiles, Azure's Europe Data Zone, or a provider whose entire public fleet is documented as EU-hosted, each checked per model against the provider's documentation. Global deployments do not count, and neither does an EU billing region, an EU company or an EU control plane on its own. One disclosed company-policy exception stays in, marked “EU equivalent”.
Data confidentialitywhat the provider may do with your prompts
More settings
Evidence requirements (benchmark evidence, priced provider, measured task tokens) sit above the table they apply to.
“Hide deprecated” removes retired models (every Artificial Analysis configuration retired, on no current DesignArena board) and older retired configurations of current models. Applies to model tables, charts, price views & model offers; benchmark evidence stays unfiltered.
JevBench is Benchmark Heaven's own benchmark for Jev-class decision models: state and a bounded rubric in, a typed answer out.
Frozen release · 1,624 decisions per system (904 open + 720 sealed; sealed decisions are half of Intelligence) · 111 ranked of 117 roster systems · only system-level sealed aggregates are published · aggregate results JSON sha256 8b837b2b760e…
$0.028 per 1,000 decisions = 0.87× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.095 s · est. incl. network (2x + 150 ms): 0.34 s = 0.55× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.095 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.34 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 2 · official #2
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.028 per 1,000 decisions = 0.88× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.040 s · est. incl. network (2x + 150 ms): 0.23 s = 0.37× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.040 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 3 · official #1
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
4Surogate Rune 26B-A4B v3$⏱69.749.079.0$0.050*
Surogate Rune 26B-A4B v3 (RTX PRO 6000) · Open weights · LLM decoder
$0.050 per 1,000 decisions = 1.56× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.100 s · est. incl. network (2x + 150 ms): 0.35 s = 0.57× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.100 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.35 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 4 · official #19
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.029 per 1,000 decisions = 0.90× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.113 s · est. incl. network (2x + 150 ms): 0.38 s = 0.61× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.113 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.38 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 5 · official #7
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.053 per 1,000 decisions = 1.65× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.050 s · est. incl. network (2x + 150 ms): 0.25 s = 0.41× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.050 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.25 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 6 · official #22
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
7Vansa-3.4API$⏱58.061.472.8$0.019*API price · eligibility checked at the developer's list price; at base-model pricing it would still fit within the cost cap (0.53× Jev)
Vansa-3.4 (Vansa, hosted System One API) · Closed API (weights not public)
Base model
Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified.
Capability Score
72.8
Intelligence
58.0
Calibration
87.6
Cost Score
61.4
Cost vs cap
$0.019 per 1,000 decisions = 0.60× Jev → green: as cheap as Jev or better
Latency vs cap
0.14 s = 0.23× Jev → green: as fast as Jev or better
Pricing
Ranked at the developer's own API list price. Base-model reference: Qwen3.5-4B (developer-reported base) at the frozen 25 Sep 2026 DeepInfra list price, USD 0.03 per 1M input and 0.15 per 1M output tokens, on the same measured tokens (921,181 input, 0 output), USD 0.0170 per 1,000 decisions = 0.53× Jev, inside the cost cap (USD 0.0646). Capability Score itself does not depend on price; latency is measured on the developer's endpoint either way.
Rank
Capability Score 7 · official #5
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.53× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.016 s · est. incl. network (2x + 150 ms): 0.18 s = 0.29× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.016 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.18 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 8 · official #4
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.53× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.017 s · est. incl. network (2x + 150 ms): 0.18 s = 0.30× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.017 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.18 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 9 · official #6
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.53× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.016 s · est. incl. network (2x + 150 ms): 0.18 s = 0.30× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.016 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.18 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 10 · official #9
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.52× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.042 s · est. incl. network (2x + 150 ms): 0.23 s = 0.38× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.042 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 11 · official #10
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.015 per 1,000 decisions = 0.47× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.026 s · est. incl. network (2x + 150 ms): 0.20 s = 0.33× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.026 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.20 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 12 · official #8
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.059 per 1,000 decisions = 1.84× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.200 s · est. incl. network (2x + 150 ms): 0.55 s = 0.89× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.200 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.55 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 13 · official #26
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.53× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.016 s · est. incl. network (2x + 150 ms): 0.18 s = 0.30× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.016 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.18 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 14 · official #11
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.018 per 1,000 decisions = 0.56× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.122 s · est. incl. network (2x + 150 ms): 0.39 s = 0.64× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.122 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.39 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 15 · official #17
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.030 per 1,000 decisions = 0.92× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.146 s · est. incl. network (2x + 150 ms): 0.44 s = 0.72× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.146 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.44 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 16 · official #18
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.026 per 1,000 decisions = 0.79× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.068 s · est. incl. network (2x + 150 ms): 0.29 s = 0.46× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.068 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.29 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 17 · official #16
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.015 per 1,000 decisions = 0.48× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.084 s · est. incl. network (2x + 150 ms): 0.32 s = 0.52× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.084 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.32 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 18 · official #13
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.042 per 1,000 decisions = 1.31× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.078 s · est. incl. network (2x + 150 ms): 0.31 s = 0.50× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.078 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.31 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 19 · official #24
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.019 per 1,000 decisions = 0.58× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.126 s · est. incl. network (2x + 150 ms): 0.40 s = 0.65× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.126 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.40 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 20 · official #12
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.53× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.039 s · est. incl. network (2x + 150 ms): 0.23 s = 0.37× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.039 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 21 · official #14
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.024 per 1,000 decisions = 0.73× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.397 s · est. incl. network (2x + 150 ms): 0.94 s = 1.53× Jev → amber: inside the 2× cap, but slower than Jev
Latency basis
Self-hosted model: measured p50 0.397 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.94 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 22 · official #21
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.021 per 1,000 decisions = 0.65× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.128 s · est. incl. network (2x + 150 ms): 0.41 s = 0.66× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.128 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.41 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 23 · official #15
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.53× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.033 s · est. incl. network (2x + 150 ms): 0.22 s = 0.35× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.033 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.22 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 24 · official #25
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.013 per 1,000 decisions = 0.41× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.075 s · est. incl. network (2x + 150 ms): 0.30 s = 0.49× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.075 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.30 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 26 · official #50
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.023 per 1,000 decisions = 0.71× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.138 s · est. incl. network (2x + 150 ms): 0.43 s = 0.69× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.138 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.43 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 27 · official #47
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.52× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.036 s · est. incl. network (2x + 150 ms): 0.22 s = 0.36× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.036 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.22 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 28 · official #42
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.015 per 1,000 decisions = 0.46× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.014 s · est. incl. network (2x + 150 ms): 0.18 s = 0.29× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.014 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.18 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 29 · official #33
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.056 per 1,000 decisions = 1.73× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.020 s · est. incl. network (2x + 150 ms): 0.19 s = 0.31× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.020 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.19 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 31 · official #53
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.014 per 1,000 decisions = 0.44× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.041 s · est. incl. network (2x + 150 ms): 0.23 s = 0.38× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.041 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 32 · official #44
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.030 per 1,000 decisions = 0.94× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.019 s · est. incl. network (2x + 150 ms): 0.19 s = 0.30× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.019 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.19 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 33 · official #48
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.016 per 1,000 decisions = 0.49× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.038 s · est. incl. network (2x + 150 ms): 0.23 s = 0.37× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.038 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 34 · official #52
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
kev 4B (research preview) · Open weights · LLM decoder
Base model
Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base.
Capability Score
53.8
Intelligence
39.9
Calibration
67.6
Cost Score
65.8
Cost vs cap
$0.014 per 1,000 decisions = 0.43× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.172 s · est. incl. network (2x + 150 ms): 0.49 s = 0.80× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.172 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.49 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 35 · official #40
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.52× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.056 s · est. incl. network (2x + 150 ms): 0.26 s = 0.42× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.056 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.26 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 36 · official #23
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.036 per 1,000 decisions = 1.12× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.065 s · est. incl. network (2x + 150 ms): 0.28 s = 0.46× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.065 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.28 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 37 · official #66
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.052 per 1,000 decisions = 1.63× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.181 s · est. incl. network (2x + 150 ms): 0.51 s = 0.83× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.181 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.51 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 38 · official #71
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0013 per 1,000 decisions = 0.04× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.057 s · est. incl. network (2x + 150 ms): 0.26 s = 0.43× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.057 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.26 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 40 · official #94
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.021 per 1,000 decisions = 0.65× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.042 s · est. incl. network (2x + 150 ms): 0.23 s = 0.38× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.042 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 41 · official #101
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.052 per 1,000 decisions = 1.63× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.161 s · est. incl. network (2x + 150 ms): 0.47 s = 0.77× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.161 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.47 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 42 · official #88
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0038 per 1,000 decisions = 0.12× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.386 s · est. incl. network (2x + 150 ms): 0.92 s = 1.49× Jev → amber: inside the 2× cap, but slower than Jev
Latency basis
Self-hosted model: measured p50 0.386 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.92 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 43 · official #110
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.023 per 1,000 decisions = 0.70× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.029 s · est. incl. network (2x + 150 ms): 0.21 s = 0.34× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.029 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.21 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 44 · official #93
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0028 per 1,000 decisions = 0.09× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.310 s · est. incl. network (2x + 150 ms): 0.77 s = 1.25× Jev → amber: inside the 2× cap, but slower than Jev
Latency basis
Self-hosted model: measured p50 0.310 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.77 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 45 · official #107
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0046 per 1,000 decisions = 0.14× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.136 s · est. incl. network (2x + 150 ms): 0.42 s = 0.68× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.136 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.42 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 46 · official #83
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0046 per 1,000 decisions = 0.14× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.022 s · est. incl. network (2x + 150 ms): 0.19 s = 0.31× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.022 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.19 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 47 · official #99
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.049 per 1,000 decisions = 1.52× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.143 s · est. incl. network (2x + 150 ms): 0.44 s = 0.71× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.143 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.44 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 48 · official #105
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.020 per 1,000 decisions = 0.63× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.150 s · est. incl. network (2x + 150 ms): 0.45 s = 0.73× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.150 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.45 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 49 · official #90
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0046 per 1,000 decisions = 0.14× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.058 s · est. incl. network (2x + 150 ms): 0.27 s = 0.43× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.058 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.27 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 50 · official #95
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.011 per 1,000 decisions = 0.33× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.031 s · est. incl. network (2x + 150 ms): 0.21 s = 0.34× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.031 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.21 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 51 · official #96
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0050 per 1,000 decisions = 0.15× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.073 s · est. incl. network (2x + 150 ms): 0.30 s = 0.48× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.073 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.30 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 52 · official #92
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0028 per 1,000 decisions = 0.09× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.161 s · est. incl. network (2x + 150 ms): 0.47 s = 0.77× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.161 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.47 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 54 · official #85
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0046 per 1,000 decisions = 0.14× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.104 s · est. incl. network (2x + 150 ms): 0.36 s = 0.58× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.104 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.36 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 55 · official #97
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.045 per 1,000 decisions = 1.38× Jev → amber: inside the 2× cap, but costlier than Jev
Latency vs cap
measured p50 0.017 s · est. incl. network (2x + 150 ms): 0.18 s = 0.30× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.017 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.18 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 56 · official #82
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.00087 per 1,000 decisions = 0.03× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.034 s · est. incl. network (2x + 150 ms): 0.22 s = 0.35× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.034 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.22 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 57 · official #109
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0028 per 1,000 decisions = 0.09× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.126 s · est. incl. network (2x + 150 ms): 0.40 s = 0.65× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.126 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.40 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 58 · official #106
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0045 per 1,000 decisions = 0.14× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.176 s · est. incl. network (2x + 150 ms): 0.50 s = 0.81× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.176 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.50 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 59 · official #80
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0028 per 1,000 decisions = 0.09× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.424 s · est. incl. network (2x + 150 ms): 1.00 s = 1.62× Jev → amber: inside the 2× cap, but slower than Jev
Latency basis
Self-hosted model: measured p50 0.424 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.00 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 60 · official #79
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0039 per 1,000 decisions = 0.12× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.443 s · est. incl. network (2x + 150 ms): 1.04 s = 1.68× Jev → amber: inside the 2× cap, but slower than Jev
Latency basis
Self-hosted model: measured p50 0.443 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.04 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 61 · official #91
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0056 per 1,000 decisions = 0.17× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.063 s · est. incl. network (2x + 150 ms): 0.28 s = 0.45× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.063 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.28 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 62 · official #84
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.011 per 1,000 decisions = 0.35× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.066 s · est. incl. network (2x + 150 ms): 0.28 s = 0.46× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.066 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.28 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score 63 · official #86
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
Wide coloured bar = Capability Score (0–100). Thin bars: cost above, median latency below; shared log ratio scale 0.25× … 64× Jev, ticks at 1× and 2× (cap). Shorter is cheaper or faster. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ or hover a row or thin bar for details.
Cost and latency are shown separately because they are nearly independent across systems (Spearman ρ = 0.10, n = 111).
Jev — reference (TypeSafe, closed) (1)
Closed API (weights not public) (10)
Open weights · LLM decoder (69)
Open weights · diffusion LM (5)
Open weights · encoder / classifier (16)
Open weights · reranker (5)
Base model control (no decision fine-tune, raw logits) (5)
green: ≤ reference
amber: ≤ cap (2× reference)
red: > cap
Outside the Jev-class limits · 48 systems
Show general-purpose LLMs and other systems outside the limits
Sorted by Capability Score, not numbered. Each row says which limit it misses, measured against Jev 1.13.0 ($0.032 per 1,000 decisions, median 0.62 s).
$0.25 per 1,000 decisions = 7.70× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.666 s · est. incl. network (2x + 150 ms): 1.48 s = 2.40× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.666 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.48 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #62
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.24 per 1,000 decisions = 7.48× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.706 s · est. incl. network (2x + 150 ms): 1.56 s = 2.53× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.706 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.56 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #61
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.24 per 1,000 decisions = 7.38× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.102 s · est. incl. network (2x + 150 ms): 0.35 s = 0.57× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.102 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.35 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #58
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.23 per 1,000 decisions = 7.00× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.101 s · est. incl. network (2x + 150 ms): 0.35 s = 0.57× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.101 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.35 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #55
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.23 per 1,000 decisions = 7.00× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.088 s · est. incl. network (2x + 150 ms): 0.33 s = 0.53× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.088 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.33 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #56
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.085 per 1,000 decisions = 2.63× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.209 s · est. incl. network (2x + 150 ms): 0.57 s = 0.92× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.209 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.57 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #31
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.082 per 1,000 decisions = 2.55× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.080 s · est. incl. network (2x + 150 ms): 0.31 s = 0.50× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.080 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.31 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #29
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.25 per 1,000 decisions = 7.72× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.358 s · est. incl. network (2x + 150 ms): 0.87 s = 1.41× Jev → amber: inside the 2× cap, but slower than Jev
Latency basis
Self-hosted model: measured p50 0.358 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.87 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #63
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.23 per 1,000 decisions = 7.14× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.049 s · est. incl. network (2x + 150 ms): 0.25 s = 0.40× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.049 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.25 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #57
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.30 per 1,000 decisions = 9.21× Jev → red: outside the 2× cap
Latency vs cap
measured p50 1.299 s · est. incl. network (2x + 150 ms): 2.75 s = 4.46× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 1.299 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 2.75 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #68
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.23 per 1,000 decisions = 7.14× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.049 s · est. incl. network (2x + 150 ms): 0.25 s = 0.40× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.049 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.25 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #59
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.17 per 1,000 decisions = 5.28× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.563 s · est. incl. network (2x + 150 ms): 1.28 s = 2.07× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.563 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.28 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #49
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.24 per 1,000 decisions = 7.57× Jev → red: outside the 2× cap
Latency vs cap
measured p50 1.459 s · est. incl. network (2x + 150 ms): 3.07 s = 4.98× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 1.459 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 3.07 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #64
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.078 per 1,000 decisions = 2.41× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.027 s · est. incl. network (2x + 150 ms): 0.20 s = 0.33× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.027 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.20 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #28
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.15 per 1,000 decisions = 4.76× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.045 s · est. incl. network (2x + 150 ms): 0.24 s = 0.39× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.045 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.24 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #46
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.065 per 1,000 decisions = 2.00× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.215 s · est. incl. network (2x + 150 ms): 0.58 s = 0.94× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.215 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.58 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #27
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.13 per 1,000 decisions = 3.97× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.198 s · est. incl. network (2x + 150 ms): 0.55 s = 0.89× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.198 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.55 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #43
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.72 per 1,000 decisions = 22.16× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.700 s · est. incl. network (2x + 150 ms): 1.55 s = 2.51× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.700 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.55 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #76
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.10 per 1,000 decisions = 3.13× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.058 s · est. incl. network (2x + 150 ms): 0.27 s = 0.43× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.058 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.27 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #39
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.017 per 1,000 decisions = 0.52× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 1.358 s · est. incl. network (2x + 150 ms): 2.87 s = 4.65× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 1.358 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 2.87 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #20
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.25 per 1,000 decisions = 7.64× Jev → red: outside the 2× cap
Latency vs cap
measured p50 1.020 s · est. incl. network (2x + 150 ms): 2.19 s = 3.55× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 1.020 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 2.19 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #65
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.097 per 1,000 decisions = 3.00× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.181 s · est. incl. network (2x + 150 ms): 0.51 s = 0.83× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.181 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.51 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #41
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.17 per 1,000 decisions = 5.28× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.429 s · est. incl. network (2x + 150 ms): 1.01 s = 1.64× Jev → amber: inside the 2× cap, but slower than Jev
Latency basis
Self-hosted model: measured p50 0.429 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.01 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #69
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.011 per 1,000 decisions = 0.33× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.620 s · est. incl. network (2x + 150 ms): 1.39 s = 2.25× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.620 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.39 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #67
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.065 per 1,000 decisions = 2.02× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.091 s · est. incl. network (2x + 150 ms): 0.33 s = 0.54× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.091 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.33 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #32
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.068 per 1,000 decisions = 2.11× Jev → red: outside the 2× cap
Latency vs cap
measured p50 0.038 s · est. incl. network (2x + 150 ms): 0.23 s = 0.37× Jev → green: as fast as Jev or better
Latency basis
Self-hosted model: measured p50 0.038 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.23 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #34
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0043 per 1,000 decisions = 0.13× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 3.494 s · est. incl. network (2x + 150 ms): 7.14 s = 11.58× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 3.494 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 7.14 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #89
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0038 per 1,000 decisions = 0.12× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 1.584 s · est. incl. network (2x + 150 ms): 3.32 s = 5.38× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 1.584 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 3.32 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #111
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0056 per 1,000 decisions = 0.17× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 1.944 s · est. incl. network (2x + 150 ms): 4.04 s = 6.55× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 1.944 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 4.04 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #78
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0056 per 1,000 decisions = 0.17× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 1.391 s · est. incl. network (2x + 150 ms): 2.93 s = 4.76× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 1.391 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 2.93 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #104
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0032 per 1,000 decisions = 0.10× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.668 s · est. incl. network (2x + 150 ms): 1.49 s = 2.41× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.668 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.49 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #98
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0048 per 1,000 decisions = 0.15× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.579 s · est. incl. network (2x + 150 ms): 1.31 s = 2.12× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.579 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.31 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #100
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0028 per 1,000 decisions = 0.09× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.610 s · est. incl. network (2x + 150 ms): 1.37 s = 2.22× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.610 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.37 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #77
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0056 per 1,000 decisions = 0.17× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 0.865 s · est. incl. network (2x + 150 ms): 1.88 s = 3.05× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 0.865 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 1.88 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #70
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0098 per 1,000 decisions = 0.30× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 3.610 s · est. incl. network (2x + 150 ms): 7.37 s = 11.96× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 3.610 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 7.37 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #73
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.0023 per 1,000 decisions = 0.07× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 2.214 s · est. incl. network (2x + 150 ms): 4.58 s = 7.43× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 2.214 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 4.58 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #87
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.019 per 1,000 decisions = 0.59× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 67.682 s · est. incl. network (2x + 150 ms): 135.51 s = 219.82× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 67.682 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 135.51 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #102
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
$0.019 per 1,000 decisions = 0.59× Jev → green: as cheap as Jev or better
Latency vs cap
measured p50 29.005 s · est. incl. network (2x + 150 ms): 58.16 s = 94.34× Jev → red: outside the 2× cap
Latency basis
Self-hosted model: measured p50 29.005 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 58.16 s. Model authors can ask us to time a hosted endpoint instead (on request).
Rank
Capability Score outside Jev-class · official #103
Colours
Green ≤ reference; amber ≤ cap; red > cap. Shorter is cheaper or faster.
Jev-class = cost per decision at most 2× Jev 1.13.0's (≤ $0.065 per 1,000 decisions)and median latency at most 2× Jev 1.13.0's (≤ 1.23 s), the adjusted p50 — the same median the speed chart plots, not the four-axis Speed score. 63 of 111 systems qualify; the other 48, including the general-purpose LLMs, are listed below the divider in the ranking. The charts below show speed and cost beside Capability Score; the official JevBench Score weighs all four axes.
Capability against cost and speed
Jev-class systems are shown by default. Bubble size follows the official JevBench Score; official rank stays unchanged. The five most capable Jev-class systems are labelled.
Capability vs cost
Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.
63 systems. Hover or focus a bubbleTap a bubble for its values.
Capability vs speed
Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). Self-hosted systems are plotted at the est. incl. network latency (2x measured + 150 ms); their measured loopback p50 (on-GPU, no network; not directly comparable with API latency) is in the tooltip. The dashed line is 2× Jev’s median latency.
63 systems. Hover or focus a bubbleTap a bubble for its values.
Jev — reference (TypeSafe, closed) (1)
Closed API (weights not public) (10)
Open weights · LLM decoder (69)
Open weights · diffusion LM (5)
Open weights · encoder / classifier (16)
Open weights · reranker (5)
Base model control (no decision fine-tune, raw logits) (5)
faint = outside Jev-class
117 of 117 systems
Model kind
Jev-class
Release
Parameters–
No system in this release reports an exact parameter count.
Developer/API price $/1k–
Base-model price $/1k–
Scored cost $/1k–
Alternative price $/1k–
p50 latency (s)–
p95 latency (s)–
What-If: the weight sliders below re-score every system under other axis weights — only the equal 25/25/25/25 weights give the official option-A ranking. The 3D view of capability, cost and speed loads further down.
JevBench v1.5.7
JevBench Composite Score: 111 ranked systems
Official· four axes 0–100, equal-weight harmonic mean · Method notes ↓
Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).
Whiskers are 95% bootstrap intervals. 67 of 74 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 36 adjacent pairs, so no tie classification is inferred.
All 111 systems
Greener = stronger within its column.
111 of 111 systems, sorted by official rank, #1 first.
*1Vansa-3.4 (base model: Qwen3.5-4B (self-reported)): Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified.
*3kev 4B (base model: Qwen/Qwen3-4B-Base): Evaluated Qwen3 variant; later releases use a different base.
*4Instinct (base model: Qwen3.8-27B): Operator-reported; weights not publicly verifiable.
020406080100
Weights:
Adjust weights ↓
Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; the gates still apply when an axis sits at 0. Only equal weights give the official JevBench Score and rank.
Base model control (no decision fine-tune, raw logits) (5)
Striped bar = same system under the labelled alternative price assumption
I, C, S, K = Intelligence, Calibration, Speed, Cost; the est. pill = estimated cost; ann. = announced price; API = the operator's endpoint saw held-out benchmark inputs, without answers; new = first listed in v1.5.7; $/1k decisions = US dollars per 1,000 decisions (not heat-shaded). Click a column heading to sort. Names link to each project.
Honorable mentions and Jev wrappers — listed separately, not ranked (1)
Eligibility rule: services that run on Jev itself may be measured and shown as honorable mentions, but are not competitors ranked against Jev and do not enter the field median gap (G_med) or tie markers. classifier.dev (TypeSafe) runs on Jev, so it stays unranked under this rule.
classifier.dev (fast tier)APIhonorable mention: runs on Jev (TypeSafe) - listed, not ranked (honorable mention, as in v1.4.2). Official (A) score 74.7.
Compare two systems
Pick any two. Radars for the score axes, capability by subject topic and use cases, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.
0–100, the values in the table. An axis a system has no published value for is left as a gap (it counts as 0 in the composite).
Capability by subject topic
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
What the decision is about — math and dates, coding, rules, money, support work, everyday messages, safety. Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Jev 1.13.0: open+sealed; Cygnet: open+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.What each category means · items per category
Math & numbers — a calculation decides the answer: arithmetic, word problems, probability, dates, units. 235 items (174 open / 61 sealed)
Coding & software — code, SQL, repositories, developer tools and IT systems. 92 items (67 open / 25 sealed)
Rules, policy & law — applying written rules: company policies, contracts, regulations, eligibility. 548 items (260 open / 288 sealed)
Support & operations — support tickets, incidents, logistics, scheduling desks and routing work to a team. 282 items (131 open / 151 sealed)
Everyday language — short everyday messages: intents, assistant requests, reading a detail out of a text. 115 items (98 open / 17 sealed)
Safety & security — untrusted or injected instructions, fraud, moderation, access and security triage. 128 items (40 open / 88 sealed)
Use cases (TypeSafe categories)
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
The real-world application area of each decision, after the TypeSafe use-case map; an item can count in two. Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Jev 1.13.0: open+sealed; Cygnet: open+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.
Low sample, n < 30 — indicative only
Category (items)
A: Jev 1.13.0
B: Cygnet
Graphs and knowledge graphs (15)
100.0 n=15
73.2 n=15
No published value for either system (under 15 answered items, or no per-category values): Scientific discovery (1), Semantic code linting (4), Feature extraction for predictive modeling (7), Recruiting (3), Moderation and trust and safety (12), Advertising (2).
Not drawn on the radar: with so few items a single answer moves a category score by several points, so these values are noise-prone. These values become spokes once the item pool reaches 30 items per category.
What each category means · items per category
Search and retrieval — scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 54 items (18 open / 36 sealed)
Model routing — deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 132 items (131 open / 1 sealed)
Lead generation — matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 98 items (61 open / 37 sealed)
Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 389 items (176 open / 213 sealed)
Financial crime — suspicious transactions, KYC, fraud alerts, entity matching for investigations. 31 items (10 open / 21 sealed)
Legal and compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 508 items (261 open / 247 sealed)
E-commerce marketplaces — product listings, product attributes, orders and deliveries, seller catalog policy. 71 items (48 open / 23 sealed)
Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 78 items (39 open / 39 sealed)
Not drawn: Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 159 items (76 open / 83 sealed) — not a use case of its own, so it is counted but not drawn.
No items in this release: Gaming, Demand forecasting.
Competence per request type, open / sealed
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Chance-corrected competence for Choice, Noul and Score on the 904 open and 720 sealed decisions.
Competence per tier — open set
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Per-tier competence, the three request types pooled by their published decision counts.
Competence per tier — sealed set
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Per-tier competence on the sealed decisions, types pooled the same way; item text stays private.
How the categories were made
Chance-corrected competence (0 = chance, 100 = perfect) over the category's items, open and sealed pooled, all tiers: per request type the method's competence cell (Choice and Noul accuracy above chance, Score 1 - error / chance error), then averaged over types weighted by item count. Not part of the JevBench Score; compare systems within a category, not categories with each other. Values can be negative (below chance).
Every one of the 1,624 decisions was labelled with one subject topic (the v1.2 topic list, unchanged) and one or two TypeSafe use-case categories (docs.typesafe.ai/concepts/use-case-map, plus Other) by Winnow-12B Q8 (the model behind System1 Models s1-pro) on our own GPU pod; 1,118 of the 3,248 calls were first made on s1-pro itself and agree 98.2 % with the pod run. Sealed items stayed on our own infrastructure. About 5 % of public items were checked by hand (Claude Opus 5.5); the item-group rules below fix the systematic misses found.
routing, routing_hard and tool_selection items (choose the model, agent or tool that handles a request) are Model routing first; the labeller tagged many by the request subject.
tool_guardrail, unsafe_action, adequacy, judge_hard and policy_compliance items (check an agent tool call or a drafted answer before it goes out) are LLM guardrails first.
lead_qualification items take the topic Finance & commerce (sales); the labeller read the scoring rubric as rules & law.
A second use case is kept when the labeller gave it probability >= 0.15 and it is not Other.
All values as a table
Spoke
A: Jev 1.13.0
B: Cygnet
The four score axes
Intelligence
72.0
71.1
Calibration
88.0
87.0
Speed
83.8
91.0
Cost
54.7
56.4
Capability by subject topic
Math & numbers
40.5
31.4
Coding & software
80.3
87.8
Rules, policy & law
73.7
76.2
Finance & commerce
83.4
78.5
Support & operations
91.5
88.2
Everyday language
96.1
95.6
Safety & security
67.7
74.4
Use cases (TypeSafe categories)
Search & retrieval
95.9
94.7
Model routing
96.5
99.1
LLM guardrails
67.0
72.3
Lead generation
88.1
82.2
Customer support
91.9
88.9
Insurance claims
67.5
71.7
Financial crime
58.3
71.1
Legal & compliance
74.0
75.3
E-commerce
89.8
91.6
Risk assessment
80.1
73.1
Competence per request type, open / sealed
Choice · open
85.7
82.9
Choice · sealed
87.6
80.3
Noul · open
47.8
57.9
Noul · sealed
48.6
55.7
Score · open
81.2
76.6
Score · sealed
81.1
73.2
Competence per tier — open set
Easy
90.0
89.7
Standard
80.9
82.4
Judge
83.5
82.2
Hard
53.8
58.0
Competence per tier — sealed set
Easy
91.2
88.6
Standard
79.0
73.5
Judge
73.7
75.4
Hard
72.9
65.3
Axes, request types, latency and cost
Compare:
Every measured system, under the four score axes. Intelligence is 50% open (904 decisions) and 50% sealed (720); Gap = I_open − I_sealed, and the penalty applies only above the field median gap (G_med 5.2) plus 8. The per-type competence, latency and price of the same systems are in Types & cost. On a phone the name column stays put while the table scrolls sideways.
Vansa-3.4v1.5 roster addendum A7APIdetailsClosed APIBase model:Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified.
The same systems, by what was asked and what it cost. Per-type columns are chance-corrected competence (CC, 0 = chance) for Choice, Noul and Score, open / sealed; I open and I sealed are the two halves of Intelligence. Latency is adjusted p50 / p95; cost is per 1,000 decisions. On a phone the name column stays put while the table scrolls sideways.
0.02 s / 0.05 son-GPU, no networkest. 0.18 s / 0.24 s*
$0.017
GPU pod (H100)
5
Vansa-3.4v1.5 roster addendum A7APIdetailsClosed APIBase model:Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified.
0.04 s / 0.11 son-GPU, no networkest. 0.23 s / 0.38 s*
$0.014
GPU pod (H100)
Costs are estimates (est.) unless marked tariff.
* Self-hosted rows: the first value is the p50 we measured on our own hardware over loopback (on-GPU, or on-CPU where marked, no network). It is not directly comparable with API latency, which we time end to end from our measuring host, network included. “est.” is the est. incl. network (2x + 150 ms) that the ranking and the latency cap use for every self-hosted row. Model authors can ask us to time a hosted endpoint instead (on request).
API = the operator's endpoint received sealed item text during evaluation, without answers. Sealed item text, answers and item-level results stay private; only system-level aggregates appear here. Hover a cost for its price basis and a latency for its raw values and adjustment.
Official order with 95% intervals (111 systems)
JevBench v1.5.7 · headline option A
JevBench Score: 111 ranked systems
Official (A)weighted harmonic mean of four 0–100 axes, Intelligence · Calibration · Speed · Cost = 25 · 25 · 25 · 25, with the low-axis gates · Method ↓
Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).
Whiskers are 95% bootstrap intervals. 67 of 74 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 36 adjacent pairs, so no tie classification is inferred.
1Cygnet12BBase model:google/gemma-4-12B-itsource73.7I 71 · C 87 · S 91 · K 56 · B#2 · C#1 · $0.028
2Winnow-12B Q8merge · 12B · GGUF Q8_0Base model:google/gemma-4-12B-itsource73.2I 74 · C 84 · S 86 · K 57 · B#1 · C#2 · $0.028
3Jev 1.13.0APIJev referenceBase model:undisclosed72.1I 72 · C 88 · S 84 · K 55 · B#3 · C#3 · $0.032
4JevK5 v0.3v1.5 roster addendum A1adapterBase model:Qwen/Qwen3.5-4Bsource71.9I 56 · C 88 · S 94 · K 63 · B#6 · C#10 · $0.017
5Vansa-3.4v1.5 roster addendum A7APIClosed APIBase model:Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified.71.6I 58 · C 88 · S 91 · K 61 · B#5 · C#6 · $0.019
6Plumb-4Bv1.5 roster addendum A1fine-tune · 4BBase model:alibiserikbay/JevK5source71.6I 56 · C 87 · S 93 · K 63 · B#7 · C#11 · $0.017
7Jev-Omnifine-tuneBase model:google/gemma-4-12B-itsource71.5I 70 · C 83 · S 85 · K 56 · B#4 · C#4 · $0.029
8decider-4b v2adapter · 4BBase model:Qwen/Qwen3.5-4B-Basesource71.3I 56 · C 86 · S 91 · K 65 · B#8 · C#12 · $0.015
9Decision 4B v1.2v1.5 roster addendum A1adapter · 4BBase model:Qwen/Qwen3.5-4Bsource70.8I 54 · C 89 · S 94 · K 63 · B#10 · C#14 · $0.017
10Imajev-4Bv1.5 roster addendum A2adapter · 4BBase model:Qwen/Qwen3.5-4Bsource70.4I 53 · C 88 · S 91 · K 63 · B#13 · C#15 · $0.017
11Decision 4B v1.1v1.5 roster addendum A1adapter · 4BBase model:Qwen/Qwen3.5-4Bsource70.4I 53 · C 87 · S 94 · K 63 · B#14 · C#16 · $0.017
12levv1.5 roster addendum A6adapter · 4BBase model:Qwen/Qwen3.5-4Bsource69.1I 58 · C 78 · S 87 · K 62 · B#12 · C#9 · $0.019
13Manchego v2.1v1.5 roster addendum A5adapter · 4BBase model:Qwen/Qwen3.5-4Bsource68.8I 51 · C 85 · S 89 · K 64 · B#16 · C#22 · $0.015
14SemIf4BBase model:Qwen/Qwen3.5-4Bsource68.7I 51 · C 84 · S 91 · K 63 · B#17 · C#21 · $0.017
15spark-s1-4b-v6adapter · 4BBase model:Qwen3.5-4Bsource68.2I 62 · C 70 · S 86 · K 60 · B#9 · C#5 · $0.021
16metask-jev-4b4BBase model:Qwen3.5-4Bsource67.5I 53 · C 83 · S 89 · K 58 · B#18 · C#18 · $0.026
17Hopperadapter · 4BBase model:Qwen/Qwen3.5-4Bsource67.5I 50 · C 88 · S 87 · K 62 · B#21 · C#26 · $0.018
18Malkuth-4Badapter · 4BBase model:Qwen/Qwen3.5-4B-Basesource66.8I 54 · C 83 · S 86 · K 56 · B#19 · C#17 · $0.030
19Surogate Rune 26B-A4B v3v1.5 roster addendum A226B-A4BBase model:google/gemma-4-26B-A4B-itsource66.5I 70 · C 88 · S 86 · K 49 · B#11 · C#7 · $0.050
20reflex 4Badapter · 4BBase model:Qwen/Qwen3.5-4Bsource65.2I 51 · C 87 · S 69 · K 63 · B#22 · C#23 · $0.017
21jev-local9BBase model:Qwen/Qwen3.5-9Bsource65.2I 56 · C 78 · S 73 · K 59 · B#20 · C#13 · $0.024
22djev (Maisa, diffusion-gemma)26B-A4BBase model:google/diffusiongemma-26B-A4B-itsource64.2I 72 · C 80 · S 91 · K 48 · B#15 · C#8 · $0.053
23Raw Qwen3 4B Instruct 2507 direct logitsoriginal · 4BBase model:Qwen/Qwen3-4B-Instruct-2507 (unmodified)source62.1I 54 · C 53 · S 89 · K 63 · B#23 · C#20 · $0.017
24jqv32BBase model:Qwen3-32Bsource60.8I 49 · C 87 · S 83 · K 51 · B#24 · C#29 · $0.042
25JevK5 v0.2.0adapter · 4BBase model:Qwen3.5-4Bsource58.1I 47 · C 85 · S 91 · K 63 · B#25 · C#32 · $0.017
26Clef-Flashv1.5 roster addendum A6fine-tuneBase model:Qwen/Qwen3.5-9Bsource55.1I 53 · C 88 · S 84 · K 47 · B#26 · C#28 · $0.059
27Qwen3.5-9B Jev-like data-mix v2adapter · 9BBase model:Qwen/Qwen3.5-9Bsource53.0I 60 · C 81 · S 82 · K 46 · B#27 · C#19 · $0.065
28Standard One 8Badapter · 8BBase model:mistralai/Ministral-3-8B-Instruct-2512-BF16source47.8I 60 · C 83 · S 92 · K 43 · B#30 · C#25 · $0.078
29NInfer Qwen3.8-Flash-Next mixedLLM decoderBase model:Qwen3.8-Flash-Nextsource47.5I 67 · C 89 · S 89 · K 43 · B#28 · C#24 · $0.082
30Instinct Dual 4Bv1.5 roster addendum A4APIClosed APIBase model:Qwen3.5-4Bsource · Operator-reported; weights not publicly verifiable.47.0I 43 · C 88 · S 82 · K 60 · B#34 · C#35 · $0.021
31swanOneNVFP4Base model:Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source46.6I 71 · C 87 · S 85 · K 42 · B#29 · C#27 · $0.085
32Raw Qwen3 8B direct logitsoriginal · 8BBase model:Qwen/Qwen3-8B-Basesource45.2I 51 · C 49 · S 87 · K 46 · B#31 · C#34 · $0.065
33decider-2badapter · 2BBase model:Qwen/Qwen3.5-2B-Basesource45.1I 42 · C 72 · S 94 · K 65 · B#37 · C#37 · $0.015
34system-one8BBase model:Qwen3-8Bsource44.1I 50 · C 49 · S 91 · K 45 · B#32 · C#38 · $0.068
35system-one-openAPIadapter · 2BBase model:Gemma 4 E2Bsource42.4I 42 · C 72 · S 78 · K 68 · B#38 · C#40 · $0.011
36Autoloops – Gemma 4 31B ITAPIClosed APIBase model:undisclosed40.5I 77 · C 86 · S 84 · K 40 · B#35 · C#30 · tariff$0.103
37GPT-6 Luna (low reasoning effort)APIClosed APIBase model:undisclosed40.5I 95 · C 95 · S 73 · K 39 · B#33 · C#31 · $0.108
38GPT-6 Luna (default medium reasoning effort)APIClosed APIBase model:undisclosed38.8I 96 · C 96 · S 73 · K 38 · B#36 · C#33 · $0.114
39JevOneadapterBase model:Qwen/Qwen3.6-35B-A3Bsource38.2I 54 · C 85 · S 90 · K 40 · B#39 · C#39 · $0.101
40kev 4Badapter · 4BBase model:Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base.38.1I 40 · C 68 · S 85 · K 66 · B#40 · C#43 · $0.014
41kev 8Badapter · 8BBase model:Qwen/Qwen3-8B-Basesource34.2I 48 · C 71 · S 84 · K 40 · B#41 · C#45 · $0.097
42open-alternative-jev4BBase model:Qwen3.5-4Bsource33.6I 37 · C 77 · S 91 · K 63 · B#43 · C#46 · $0.017
43Bespoke Nimble 9Badapter · 9BBase model:Qwen3.5-9Bsource31.8I 64 · C 77 · S 83 · K 37 · B#42 · C#36 · $0.128
44Malkuth-2Badapter · 2BBase model:empero-ai/Qwen3.8-2B-Distillsource29.9I 36 · C 75 · S 92 · K 66 · B#46 · C#48 · $0.014
45openjev-sglangAPI35B-A3BBase model:Qwen/Qwen3.6-35B-A3Bsource29.0I 59 · C 83 · S 78 · K 36 · B#44 · C#41 · $0.140
46decider-35b-a3bfine-tune · 35B-A3BBase model:Qwen/Qwen3.5-35B-A3B-Basesource27.5I 60 · C 82 · S 91 · K 34 · B#45 · C#42 · $0.154
47local-jev Qwen3.5-4Bfine-tune · 4BBase model:Qwen/Qwen3.5-4Bsource25.8I 34 · C 82 · S 84 · K 59 · B#49 · C#57 · $0.023
48Nemotron Diffusion 8Bv1.5 roster addendum A38BBase model:nvidia/Nemotron-Labs-Diffusion-8Bsource25.7I 34 · C 76 · S 94 · K 55 · B#50 · C#58 · $0.030
49Open-Jev 9Badapter · 9BBase model:Qwen/Qwen3.5-9Bsource24.4I 64 · C 82 · S 74 · K 33 · B#47 · C#44 · $0.170
50Decision 2Badapter · 2BBase model:openbmb/MiniCPM5-2Bsource22.5I 31 · C 86 · S 90 · K 66 · B#56 · C#61 · $0.013
51GPT-5.6 LunaAPIClosed APIBase model:undisclosed22.4I 94 · C 95 · S 74 · K 31 · B#48 · C#47 · tariff$0.205
52typecastlm4BBase model:Qwen3.5-4Bsource21.8I 31 · C 77 · S 92 · K 64 · B#60 · C#63 · $0.016
53JEV Qwen3.5-9B Base NVFP49B · NVFP4Base model:ig1/Qwen3.5-9B-NVFP4source20.1I 32 · C 81 · S 94 · K 48 · B#62 · C#64 · $0.056
54Gemini 3.1 Flash-LiteAPIClosed APIBase model:undisclosed19.6I 78 · C 75 · S 80 · K 30 · B#51 · C#49 · tariff$0.219
55AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1fine-tune · 27BBase model:Qwen/Qwen3.8-27Bsource19.5I 73 · C 88 · S 88 · K 29 · B#52 · C#50 · $0.226
56AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2fine-tune · 27BBase model:Qwen/Qwen3.8-27Bsource19.5I 73 · C 87 · S 87 · K 29 · B#53 · C#51 · $0.226
57NInfer Qwen3.8-27B NVFP427BBase model:Qwen3.8-27Bsource18.7I 65 · C 86 · S 90 · K 29 · B#55 · C#52 · $0.231
58Eikos-27Bv1.5 roster addendum A1adapter · 27BBase model:Qwen/Qwen3.8-27Bsource18.5I 75 · C 86 · S 88 · K 29 · B#54 · C#53 · $0.238
59NInfer Qwen3.8-27B NVFP4 (T=1.5)27BBase model:Qwen3.8-27Bsource18.5I 61 · C 86 · S 90 · K 29 · B#58 · C#54 · $0.231
60InstinctAPIClosed APIBase model:Qwen3.8-27Bsource · Operator-reported; weights not publicly verifiable.18.3I 63 · C 85 · S 82 · K 29 · B#59 · C#55 · $0.230
61OpenJev (thinking, BF16)26B-A4BBase model:google/diffusiongemma-26B-A4B-itsource 1source 217.9I 84 · C 83 · S 74 · K 29 · B#57 · C#56 · $0.241
62djev (thinking)26B-A4BBase model:google/diffusiongemma-26B-A4B-itsource17.4I 77 · C 96 · S 72 · K 28 · B#61 · C#59 · $0.249
63Clefv1.5 roster addendum A6fine-tuneBase model:Qwen/Qwen3.8-27Bsource17.0I 68 · C 87 · S 80 · K 28 · B#63 · C#60 · $0.249
64LitJev27BBase model:Qwen/Qwen3.8-27Bsource16.3I 58 · C 84 · S 68 · K 28 · B#64 · C#62 · $0.244
65Bev / Bonsai 27Bv1.5 roster addendum A427B · GGUF PQ2_0Base model:Qwen/Qwen3.8-27Bsource15.8I 53 · C 78 · S 73 · K 28 · B#65 · C#66 · $0.247
66Raw Phi-4 mini direct logitsoriginalBase model:microsoft/Phi-4-mini-instruct (unmodified)source15.2I 28 · C 71 · S 89 · K 53 · B#67 · C#67 · $0.036
67OpenSourceJevadapter · 4B · GGUF Q4_K_MBase model:Qwen3.5-4Bsource13.2I 26 · C 76 · S 73 · K 69 · B#68 · C#68 · $0.011
68reflex-27b27BBase model:Qwen3.8-27Bsource13.2I 63 · C 86 · S 69 · K 26 · B#66 · C#65 · $0.297
69Open-Jev 2Badapter · 2BBase model:Qwen/Qwen3.5-2Bsource9.1I 34 · C 74 · S 76 · K 33 · B#69 · C#70 · $0.170
70GLiNER2 largefine-tuneBase model:microsoft/deberta-v3-largesource8.4I 22 · C 42 · S 65 · K 78 · B#71 · C#71 · $0.0056
71Qwen3-Reranker-4BRerankerBase model:Qwen/Qwen3-4B-Basesource7.1I 21 · C 76 · S 80 · K 48 · B#72 · C#72 · $0.052
72DeepSeek V4.1 FlashAPILLM decoderBase model:undisclosed6.6I 94 · C 97 · S 69 · K 19 · B#70 · C#69 · $0.498
73SimpleJev0.8BBase model:Qwen/Qwen3.5-0.8Bsource4.0I 17 · C 47 · S 59 · K 70 · B#74 · C#75 · $0.0098
74SimpleJev Qwen3.8-27BAPI27BBase model:Qwen/Qwen3.8-27Bsource3.4I 73 · C 87 · S 75 · K 15 · B#73 · C#73 · $0.687
75decision-machine-1APIClosed APIBase model:undisclosed3.2I 15 · C 81 · S 93 · K 56 · B#76 · C#76 · $0.029
76Open-Jev 27B v1.1v1.5 roster addendum A8adapter · 27BBase model:Qwen/Qwen3.8-27Bsource2.9I 61 · C 80 · S 71 · K 14 · B#75 · C#74 · $0.716
77GLiNER2.5 multifine-tune · 287MBase model:mDeBERTa-v3-basesource2.7I 14 · C 58 · S 67 · K 87 · B#77 · C#77 · $0.0028
78Bosun v3.1 0.6Bv1.5 roster addendum A4adapter · 0.6BBase model:Qwen/Qwen3-0.6Bsource2.5I 14 · C 65 · S 62 · K 78 · B#78 · C#78 · $0.0056
79GLiNER2adapterBase model:DeBERTa-v3-basesource2.3I 13 · C 36 · S 70 · K 87 · B#79 · C#79 · $0.0028
80Deem 0.8B v1v1.5 roster addendum A4LLM decoderBase model:Qwen/Qwen3.5-0.8Bsource2.1I 13 · C 39 · S 84 · K 80 · B#80 · C#80 · $0.0045
81JevActAPIClosed APIBase model:undisclosedsource1.5I 11 · C 63 · S 76 · K 69 · B#82 · C#81 · $0.011
82CLM-8Bfine-tune · 8BBase model:Qwen/Qwen3-8Bsource1.5I 11 · C 48 · S 93 · K 51 · B#81 · C#82 · $0.045
83kev 0.6Badapter · 0.6BBase model:Qwen/Qwen3-0.6B-Basesource1.3I 10 · C 68 · S 87 · K 80 · B#83 · C#83 · $0.0046
84Raw Qwen3 0.6B direct logitsoriginal · 0.6BBase model:Qwen/Qwen3-0.6B-Basesource1.1I 11 · C 21 · S 90 · K 78 · B#84 · C#84 · $0.0056
85GLiNER2.5 smallfine-tune · 74MBase model:microsoft/deberta-v3-xsmallsource0.9I 9 · C 56 · S 77 · K 87 · B#86 · C#85 · $0.0028
86Raw Qwen3 1.7B direct logitsoriginal · 1.7BBase model:Qwen/Qwen3-1.7B-Basesource0.9I 10 · C 22 · S 90 · K 69 · B#85 · C#86 · $0.011
87MirrorEncoder / classifierBase model:undisclosed0.2I 6 · C 43 · S 64 · K 89 · B#87 · C#87 · $0.0023
88ZeroEntropy zerank-2RerankerBase model:Qwen/Qwen3-4Bsource0.1I 5 · C 82 · S 80 · K 48 · B#88 · C#88 · $0.052
89jeffEncoder / classifierBase model:undisclosedsource0.1I 4 · C 80 · S 56 · K 81 · B#89 · C#89 · $0.0043
90smalljev semantic-v9adapter · 2BBase model:openbmb/MiniCPM5-2B-Basesource0.1I 4 · C 73 · S 86 · K 61 · B#90 · C#90 · $0.020
91Laya multilingualv1.5 roster addendum A4fine-tune · 322MBase model:mmBERT-basesource0.0I 2 · C 44 · S 74 · K 82 · B#91 · C#91 · $0.0039
92OpenDecisionEncoder / classifierBase model:MoritzLaurer/ModernBERT-large-zeroshot-v2.0source0.0I 2 · C 73 · S 87 · K 79 · B#92 · C#92 · $0.0050
93BAAI bge-reranker-v2-m3fine-tuneBase model:BAAI/bge-m3source0.0I 0 · C 83 · S 91 · K 59 · B#93 · C#93 · $0.023
94Certo v1Encoder / classifierBase model:ModernBERT-largesource0.0I 0 · C 88 · S 91 · K 97 · B#94 · C#94 · $0.0013
95Decision Fastadapter · 0.6BBase model:Qwen/Qwen3-0.6B-Basesource0.0I 0 · C 76 · S 91 · K 80 · B#95 · C#95 · $0.0046
96Alibaba GTE Reranker ModernBERT-basefine-tuneBase model:answerdotai/ModernBERT-basesource0.0I 0 · C 75 · S 91 · K 69 · B#96 · C#96 · $0.011
97kev 0.5Badapter · 0.5BBase model:Qwen/Qwen2.5-0.5Bsource0.0I 0 · C 65 · S 88 · K 80 · B#97 · C#97 · $0.0046
98Layafine-tune · 421MBase model:ModernBERT-largesource0.0I 0 · C 74 · S 74 · K 85 · B#98 · C#98 · $0.0032
99lev-350madapterBase model:LiquidAI/LFM2.5-350Msource0.0I 0 · C 78 · S 94 · K 80 · B#99 · C#99 · $0.0046
100Qwen3.5-0.8B Decision Modelfine-tune · 0.8BBase model:Qwen/Qwen3.5-0.8B-Basesource0.0I 0 · C 74 · S 72 · K 80 · B#100 · C#100 · $0.0048
101Mixedbread mxbai-rerank-base-v2RerankerBase model:Qwen2.5 (size not stated)source 1source 20.0I 0 · C 87 · S 89 · K 60 · B#101 · C#101 · $0.021
102Needle 3original · 121MBase model:undisclosedsource0.0I 0 · C 0 · S 34 · K 62 · B#102 · C#102 · $0.019
103Needle 3, options as toolsoriginal · 121MBase model:undisclosedsource0.0I 0 · C 0 · S 41 · K 62 · B#103 · C#103 · $0.019
104open-jev-deberta-v3-largeadapterBase model:microsoft/deberta-v3-largesource0.0I 0 · C 77 · S 68 · K 78 · B#104 · C#104 · $0.0056
105Open Jev JSON Canvas26B-A4BBase model:google/diffusiongemma-26B-A4B-itsource0.0I 77 · C 0 · S 86 · K 49 · B#105 · C#105 · $0.049
106openJev Verdictfine-tuneBase model:knowledgator/gliclass-modern-base-v2.0source 1source 20.0I 0 · C 52 · S 84 · K 87 · B#106 · C#106 · $0.0028
107openJev Verdict 1.4Encoder / classifierBase model:knowledgator/gliclass-modern-base-v2.0source0.0I 0 · C 80 · S 81 · K 87 · B#107 · C#107 · $0.0028
108Qwen3.8 27BAPI27BBase model:undisclosed0.0I 96 · C 98 · S 57 · K 0 · B#108 · C#108 · $2.18
109verdict-smallfine-tune · 118MBase model:intfloat/multilingual-e5-smallsource0.0I 0 · C 59 · S 82 · K 100 · B#109 · C#109 · $0.0009
110Vonfine-tune · 395MBase model:answerdotai/ModernBERT-largesource0.0I 0 · C 83 · S 76 · K 83 · B#110 · C#110 · $0.0038
111Laya typed-decisionsv1.5 roster addendum A4fine-tune · 421MBase model:ModernBERT-largesource0.0I 0 · C 83 · S 63 · K 83 · B#111 · C#111 · $0.0038
Costs are estimates (est.) unless marked tariff.
Jev — reference (TypeSafe, closed) (1)Closed API (weights not public) (10)Open weights · LLM decoder (69)Open weights · diffusion LM (5)Open weights · encoder / classifier (16)Open weights · reranker (5)Base model control (no decision fine-tune, raw logits) (5)
All three weight options (111 systems)
All three weight options
A is the official headline: equal 25/25/25/25 axis weights and an Intelligence floor of 50. B remains the secondary 40/20/20/20 axis-weight view; C retains equal axes with an Intelligence floor of 60. All three use equal Choice/Noul/Score weights. The CI column is the paired-bootstrap 95% interval of the A score.
Vansa-3.4v1.5 roster addendum A7APIdetailsClosed APIBase model:Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified.
Among the 63 Jev-class systems, Jev 1.13.0 has the highest JevBench Capability Score, 80.0 (Intelligence 72.0, Calibration 88.0).
Cygnet leads the official JevBench v1.5.7 score (option A) with 73.7: Intelligence 71.1, Calibration 87.0, Speed 91.0, Cost 56.4 ($0.028 per 1,000 decisions).
The best open or open-planned rebuild, Winnow-12B Q8, is #2 at 73.2 — 0.5 points behind.
GPT-6 Luna (default medium reasoning effort) has the highest Intelligence (96.2) but places #38: Speed 73.2, Cost 38.3 — the harmonic mean does not let accuracy buy back a weak axis.
The strongest sealed Intelligence is 98.2 (Qwen3.8 27B, #108); sealed items carry half of Intelligence, and an open-minus-sealed gap beyond the field median plus eight points costs Intelligence.
67 of 74 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other adjacent pairs, so no tie classification is inferred.
All 22 systems joined by separately hashed roster addenda and have official ranks in this revision; their A/B/C ranks and intervals are in the addendum table below.
classifier.dev scores 74.7 but is not ranked: runs on Jev (TypeSafe) - listed, not ranked (honorable mention, as in v1.4.2).
SimpleJev Qwen3.6-35B-A3B, Decision-4B did not complete the full suite; they are listed without a rank.
Jev alternatives, open source and self-hosting
The chart and table above compare the tested systems, not marketing claims. These are the practical answers readers most often need before choosing a Jev-class decision model.
What are open-source alternatives to Jev?
The highest-ranked open entrants in this run are Cygnet (#1, 73.7), Winnow-12B Q8 (#2, 73.2), JevK5 v0.3 (#4, 71.9), Plumb-4B (#6, 71.6). “Open” here means the tested row publishes code or weights; check the licence and exact configuration in the board before adopting one.
Which Jev-class models can I self-host in the EU or use for GDPR-sensitive work?
Open entrants with released code or weights can run on infrastructure you choose, including EU infrastructure. That can support data residency, but neither open source nor an EU server makes a deployment GDPR-compliant by itself. Assess your data, contracts, retention, subprocessors and security for the complete setup. See Benchmark Heaven's broader EU-hosting comparison.
jev-router.com offers self-hosted open decision models. Neutrality disclosure: it is run by the authors of this benchmark; it receives no scoring advantage and is not a ranked entrant.
How is JevBench scored?
The official score (option A) is the equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost — 25/25/25/25 — with an Intelligence floor of 50 and low-axis gates on Speed and Cost. Version v1.5.7 measures 904 open and 720 sealed decisions per system; Choice, Noul and Score each carry a third, sealed items contribute 50% of Intelligence, and an open-minus-sealed gap beyond the field median costs points. Method notes · options B and C.
How do I submit my model?
Open an issue in the JevBench repository with a reproducible endpoint or runnable code, the exact model and licence, and whether public JevBench items were used during development. New entrants use the same frozen harness and appear in a new version or a disclosed roster addendum. For private data, see the custom evaluation options.
What a decision costs
Every price here is US dollars per 1,000 decisions — not per 1,000 tokens. One decision is a whole typed request — state, rubric and options — not a single token.
How costs are estimated
Systems with a public tariff (per token or per request) are priced at that tariff times the tokens we measured — 3 rows carry a tariff. Systems without one — open weights, author demos, models we ran ourselves — are priced as if a large inference provider hosted them: the list price of the same weights, or the nearest larger sibling or size class when the exact weights are not listed. We do not use per-minute GPU rental or our own CPU time — providers buy capacity in bulk or own the hardware, and price accordingly. Price × tokens per decision = $ per 1,000 decisions, marked “est.”.
Price rules (v1.5). Only public, bookable list prices that have been in effect for at least 30 days count; a manufacturer's standard, non-promotional launch list price counts from day one, and promotions, subsidies, credits and free tiers never do. The scoring price is never below the market reference price of the system's base model. A system without any eligible price is listed as unpriced — no Cost axis and no score until a price qualifies. A later price change triggers a re-score with a visible note on the row.
API models with a known base model (from 1 Oct 2026). They are ranked at their developer's own stated API price; a striped second bar shows the score and rank they would have at the base-model reference price, the way we price self-served open weights of the same base. First applied on Image JevBench v0.1.5 (Wity-1). Self-hosted open weights retain the base-model reference price. Cloudflare rows in v1.5.5 show a separate striped Workers AI price scenario with the self-hosted latency held fixed; Workers AI latency has not been measured, and this scenario does not determine official Capability eligibility. Vansa-3.4 (from v1.5.6) is the first JevBench API row under this rule: it is ranked at Vansa's stated USD 0.034 per million input tokens with no output charge, and its striped bar uses the Qwen3.5-4B reference price. Its base model is developer-reported and not independently verified.
Cygnet — ~$0.028 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Winnow-12B Q8 — ~$0.028 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Jev 1.13.0 — ~$0.032 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
Jev-Omni — ~$0.029 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
decider-4b v2 — ~$0.015 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
SemIf — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
spark-s1-4b-v6 — ~$0.021 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
metask-jev-4b — ~$0.026 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Hopper — ~$0.018 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Malkuth-4B — ~$0.030 est. per 1,000 decisions: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
reflex 4B — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
jev-local — ~$0.024 est. per 1,000 decisions: price floor: base-model reference price applied
djev (Maisa, diffusion-gemma) — ~$0.053 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Raw Qwen3 4B Instruct 2507 direct logits — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
jqv — ~$0.042 est. per 1,000 decisions: documented hosted-model estimate
JevK5 v0.2.0 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Qwen3.5-9B Jev-like data-mix v2 — ~$0.065 est. per 1,000 decisions: documented hosted-model estimate
Standard One 8B — ~$0.078 est. per 1,000 decisions: documented hosted-model estimate
NInfer Qwen3.8-Flash-Next mixed — ~$0.082 est. per 1,000 decisions: documented hosted-model estimate
swanOne — ~$0.085 est. per 1,000 decisions: price floor: base-model reference price applied
Raw Qwen3 8B direct logits — ~$0.065 est. per 1,000 decisions: documented hosted-model estimate
decider-2b — ~$0.015 est. per 1,000 decisions: documented hosted-model estimate
system-one — ~$0.068 est. per 1,000 decisions: documented hosted-model estimate
system-one-open — ~$0.011 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
GPT-6 Luna (low reasoning effort) — ~$0.108 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
GPT-6 Luna (default medium reasoning effort) — ~$0.114 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
JevOne — ~$0.101 est. per 1,000 decisions: documented hosted-model estimate
kev 4B — ~$0.014 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
kev 8B — ~$0.097 est. per 1,000 decisions: price floor: base-model reference price applied
open-alternative-jev — ~$0.017 est. per 1,000 decisions: ESTIMATE (proxy tokens, I-2): exact base-model market reference; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Bespoke Nimble 9B — ~$0.128 est. per 1,000 decisions: documented hosted-model estimate
Malkuth-2B — ~$0.014 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
openjev-sglang — ~$0.140 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
decider-35b-a3b — ~$0.154 est. per 1,000 decisions: price floor: base-model reference price applied
local-jev Qwen3.5-4B — ~$0.023 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Open-Jev 9B — ~$0.170 est. per 1,000 decisions: documented hosted-model estimate
Decision 2B — ~$0.013 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
typecastlm — ~$0.016 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
JEV Qwen3.5-9B Base NVFP4 — ~$0.056 est. per 1,000 decisions: ESTIMATE (proxy tokens, I-2): exact base-model market reference
NInfer Qwen3.8-27B NVFP4 — ~$0.231 est. per 1,000 decisions: price floor: base-model reference price applied
NInfer Qwen3.8-27B NVFP4 (T=1.5) — ~$0.231 est. per 1,000 decisions: price floor: base-model reference price applied
Instinct — ~$0.230 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
OpenJev (thinking, BF16) — ~$0.241 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
djev (thinking) — ~$0.249 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
LitJev — ~$0.244 est. per 1,000 decisions: price floor: base-model reference price applied
Raw Phi-4 mini direct logits — ~$0.036 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
OpenSourceJev — ~$0.011 est. per 1,000 decisions: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
reflex-27b — ~$0.297 est. per 1,000 decisions: price floor: base-model reference price applied
Open-Jev 2B — ~$0.170 est. per 1,000 decisions: documented hosted-model estimate
GLiNER2 large — ~$0.0056 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
Qwen3-Reranker-4B — ~$0.052 est. per 1,000 decisions: ESTIMATE: hosted exact-model reference (deepinfra:Qwen/Qwen3-Reranker-4B); no exact base-model floor applies
DeepSeek V4.1 Flash — ~$0.498 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
SimpleJev — ~$0.0098 est. per 1,000 decisions: documented hosted-model estimate
SimpleJev Qwen3.8-27B — ~$0.687 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
decision-machine-1 — ~$0.029 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
GLiNER2.5 multi — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
GLiNER2 — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
JevAct — ~$0.011 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
CLM-8B — ~$0.045 est. per 1,000 decisions: price floor: base-model reference price applied
kev 0.6B — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Raw Qwen3 0.6B direct logits — ~$0.0056 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
GLiNER2.5 small — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
Raw Qwen3 1.7B direct logits — ~$0.011 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Mirror — ~$0.0023 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
ZeroEntropy zerank-2 — ~$0.052 est. per 1,000 decisions: ESTIMATE: size-class proxy (deepinfra:Qwen/Qwen3-Reranker-4B); no exact base-model floor applies
jeff — ~$0.0043 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
smalljev semantic-v9 — ~$0.020 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
OpenDecision — ~$0.0050 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
BAAI bge-reranker-v2-m3 — ~$0.023 est. per 1,000 decisions: ESTIMATE: base-model market reference (deepinfra:BAAI/bge-m3)
Certo v1 — ~$0.0013 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Decision Fast — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Alibaba GTE Reranker ModernBERT-base — ~$0.011 est. per 1,000 decisions: ESTIMATE: size-class proxy (deepinfra:thenlper/gte-base); no exact base-model floor applies
kev 0.5B — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Laya — ~$0.0032 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
lev-350m — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Qwen3.5-0.8B Decision Model — ~$0.0048 est. per 1,000 decisions: documented hosted-model estimate
Mixedbread mxbai-rerank-base-v2 — ~$0.021 est. per 1,000 decisions: ESTIMATE: size-class proxy (deepinfra:Qwen/Qwen3-Reranker-0.6B); no exact base-model floor applies
Needle 3 — ~$0.019 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
Needle 3, options as tools — ~$0.019 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
open-jev-deberta-v3-large — ~$0.0056 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
Open Jev JSON Canvas — ~$0.049 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
openJev Verdict — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
openJev Verdict 1.4 — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
Qwen3.8 27B — ~$2.18 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
verdict-small — ~$0.0009 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Von — ~$0.0038 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
classifier.dev — ~$0.023 est. per 1,000 decisions: ESTIMATE (proxy tokens, I-2): operator list price: higher of 19 Sep plan cost and 26 Sep usage tariff USD 0.042/M input (rule 1.2); no exact base-model floor applies
JevK5 v0.3 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Plumb-4B — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Decision 4B v1.2 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Imajev-4B — ~$0.017 est. per 1,000 decisions: ESTIMATE (I-2): measured input tokens; zero generated output tokens for signed logits readout. M2 floor uses the 25 Sep 2026 DeepInfra Qwen3.5-4B snapshot rates (USD 0.03/M input, USD 0.15/M output); the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen3.5-9B.
Decision 4B v1.1 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
Surogate Rune 26B-A4B v3 — ~$0.050 est. per 1,000 decisions: ESTIMATE: M2 base-model market reference from the frozen 25 Sep 2026 OpenRouter snapshot applied to measured tokens.
AutoJev-27B (denis-pplx, Qwen3.8-27B) — ~$0.226 est. per 1,000 decisions: documented hosted-model estimate
AutoJev-27B (RTX PRO 6000) — ~$0.226 est. per 1,000 decisions: ESTIMATE: M2 base-model market reference from the frozen 25 Sep 2026 OpenRouter snapshot applied to measured tokens.
Eikos-27B — ~$0.238 est. per 1,000 decisions: documented hosted-model estimate
SimpleJev Qwen3.6-35B-A3B — ~$0.145 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
Nemotron Diffusion 8B — ~$0.030 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
Bev / Bonsai 27B — ~$0.247 est. per 1,000 decisions: ESTIMATE: documented hosted-model estimate. Frozen market reference qwen/qwen3.8-27b.
Deem 0.8B v1 — ~$0.0045 est. per 1,000 decisions: ESTIMATE: documented hosted-model estimate. Frozen market reference deepinfra:Qwen/Qwen3.5-0.8B.
Decision-4B — ~$0.014 est. per 1,000 decisions: ESTIMATE: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off, already marked deprecated at that cutoff (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). Frozen market reference deepinfra:Qwen/Qwen3.5-4B.
Instinct Dual 4B — ~$0.021 est. per 1,000 decisions: ESTIMATE: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule); reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off, already marked deprecated at that cutoff (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). Frozen market reference deepinfra:Qwen/Qwen3.5-4B.
Bosun v3.1 0.6B — ~$0.0056 est. per 1,000 decisions: ESTIMATE: documented hosted-model estimate; no exact base-model floor applies. $0.01/M input, $0/M output; same-base frozen raw-qwen3-0.6b estimate.
Laya multilingual — ~$0.0039 est. per 1,000 decisions: ESTIMATE: documented hosted-model estimate; no exact base-model floor applies. $0.01/M input, $0/M output; same-class hosted-encoder estimate; no exact-base market floor.
Laya typed-decisions — ~$0.0038 est. per 1,000 decisions: ESTIMATE: documented hosted-model estimate; no exact base-model floor applies. $0.01/M input, $0/M output; same-class hosted-encoder estimate; no exact-base market floor.
Manchego v2.1 — ~$0.015 est. per 1,000 decisions: ESTIMATE: frozen 25 Sep DeepInfra Qwen/Qwen3.5-4B market reference, already marked deprecated at that cutoff (replaced by Qwen/Qwen3.5-9B). $0.03/M input, zero generated output; 836,500 measured input tokens across 1,624 decisions. Frozen v1.5 base-model floor applied; no bookable Manchego tariff claimed. Re-score if the basis changes.
Clef — ~$0.249 est. per 1,000 decisions: price floor: base-model reference price applied; self-hosted estimate, not measured hosted billing
Clef-Flash — ~$0.059 est. per 1,000 decisions: price floor: base-model reference price applied; self-hosted estimate, not measured hosted billing
lev — ~$0.019 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3); self-hosted estimate, not measured hosted billing
Vansa-3.4 — ~$0.019 est. per 1,000 decisions: Developer's own stated API price: USD 0.034 per 1M input tokens, no output-token charge (docs.vansa.org/pricing, read 1 Oct 2026), x measured input tokens (921,181 over 1,624 decisions). Standard launch list price counted from day 1 (interpretation I-1); the closed-beta 'no charges today' is a promotion and is not used. API model with a developer-reported base: ranked at its own API price (1 Oct 2026 rule); the striped bar shows the Qwen3.5-4B base-model reference.
Open-Jev 27B v1.1 — ~$0.716 est. per 1,000 decisions: Frozen BASE_REFERENCES Qwen/Qwen3.8-27B market price (USD 0.42/M input, 3.00/M output) applied to 2,767,510 input tokens and zero output tokens over 1,624 decisions; self-hosted estimate, not measured hosted billing. Lineage verified by architecture; no pricing gate. The author method encodes each candidate independently.
Roster addendum: newcomers scored on the same frozen protocol (22)
All 22 A1/A2/A3/A4/A5/A6/A7/A8 systems below completed all 1,624 decisions and now have official ranks in A, B, and C; the interactive score presets also include them. Their scores, individual 95% intervals, and the frozen v1.5.0 G_med are unchanged. No new paired-bootstrap comparisons were computed for addendum systems. Existing tie markers are retained only for base-system pairs that remain adjacent; no tie or separation is inferred for the other pairs.
Ranks follow option scores; score intervals are per system, not pairwise rank comparisons. Every slider preset sorts these same ranked systems using its selected weights.
Not ranked: partial, unpriced and unmeasured systems
These systems are part of the 117-system v1.5 roster but have no rank. Their numbers are never shown as zero or free.
Partial runs (2)
SimpleJev Qwen3.6-35B-A3BAPIpartial run: Partial run: 677 of 1,624 decisions answered; the missing ones count wrong and the row is not ranked.
Decision-4B (Eval Engine / Chromia)v1.5 roster addendum A4not ranked: PARTIAL / UNRANKED: 1,550 of 1,624 valid answers; 74 context overflows in our evaluator at 2,048 tokens. No official score or rank.
Incomplete or not measured in v1.5 (3)
Jobe Qwen3.5-4B (frozen): not measured. 4BBase model:undisclosed · No verified public base-model disclosure recorded for this unmeasured evaluated variant.
mica-v01-4bv1.5 roster addendum A1: The frozen refusal policy stopped the run after 1,088 of 1,624 rows: 27 documented refusals were mapped to HTTP 422, then three consecutive passthrough HTTP 400 refusals triggered exit 6. The remaining 536 rows have no scores, so this system is not eligible for an official rank. (1,088/1,624 rows; 536 missing). fine-tune · 4BBase model:undisclosed · No verified public base-model disclosure recorded for this unmeasured evaluated variant.
OpenJev (DiffusionGemma 26B-A4B NVFP4, razorback16): not measured. 26B-A4B · NVFP4Base model:undisclosed · No verified public base-model disclosure recorded for this unmeasured evaluated variant.
Incomplete and unmeasured systems receive no official rank. Existing results from earlier benchmark versions remain on their frozen version pages.
Method notes: what changed in v1.5
The four axes
Intelligence · 25%
How often answers are right above chance: each type is normalized against its task-specific random baseline (chance = 0, perfect = 100; below-chance tiers can be negative). Choice, Noul and Score count one third each. Easy, Standard, Judge and Hard items count 10%, 20%, 30% and 40%; the open and sealed sets count equally.
Calibration · 25%
How closely stated probabilities match what happens. It uses ECE and TVD for Choice, ECE and Brier for Noul, and normalized RPS plus top-level ECE for Score. The three types count equally; open and sealed items are pooled.
Speed · 25%
Serial response latency on open Standard and Judge items. The p50 and p95 each get a log score: 100 − 20 × log₁₀(seconds ÷ 0.1), then are averaged. Self-hosted and demo endpoints get the published ×2 plus 0.15-second adjustment.
Cost · 25%
Estimated or billed US dollars per 1,000 decisions, using pooled token use across 1,624 decisions and the documented price rules. The log score is 100 − 30 × log₁₀(cost ÷ $0.001). The price reference is $0.001 per 1,000 decisions.
The official score is the weighted harmonic mean of the four axes. Option A gives each axis 25%; option B (40/20/20/20) is a secondary view, while option C keeps equal axes and sets the Intelligence gate at 60. In A and B, Intelligence, Speed and Cost each have a quadratic gate below 50. To limit benchmaxxing on the public items, the open-minus-sealed Intelligence gap may be up to 8 points above the field median (G_med) before a penalty applies. Each further point lowers the multiplier on unpenalized Intelligence by one percentage point.
The method owner chose equal axis weights and equal weights for Choice, Noul and Score after reviewing the What-If Lab, preserving continuity with v1.4 and treating the three decision types equally. Disclosed headline amendment: equal-axis, equal-type A, SHA-256 752ddccc4e19…. B remains a secondary view.
Re-evaluation policy. Every release re-evaluates the current top 10 on the composite score. Models ranked #11 and below are re-evaluated on a slower cadence — at least monthly, or with every third scheduled refresh release, whichever comes first — and their score is shown as last measured on its release. A material method change re-evaluates every model. Paid fast-lane runs are evaluated within 48 hours of payment, and new submissions are evaluated in the order received.
1,624 decisions per system: 904 open (601 published) and 720 sealed, drawn fresh from a private pool with the same tier mix as the open set. Sealed counts for 50% of Intelligence: base = 0.5 × I_open + 0.5 × I_sealed.
Three request types are scored natively and chance-corrected per item: Choice, Noul and Score each receive one third. Tier weights easy / standard / judge / hard = 10 / 20 / 30 / 40. A type a system does not support is excluded, never scored zero; only full-coverage systems are ranked.
Overfit penalty relative to the field: excess = gap − G_med, penalty = max(0, 1 − max(0, excess − 8) / 100). G_med for this batch is 5.2 CC points.
Calibration is typed (Choice ECE/TVD, Noul ECE with Brier, Score normalised RPS and top-level ECE), pooled over open and sealed. Speed and Cost formulas are unchanged from v1.4; self-hosted and demo endpoints carry the ×2 + 0.15 s adjustment. A manufacturer's standard, non-promotional launch list price counts from day one, but a newer price cut younger than 30 days does not. Rows without token counts use the measured proxy-token basis. A system without any eligible public, bookable price is listed as unpriced.
The frozen 25 Sep DeepInfra snapshot records Qwen3.5-4B as deprecated on 11 Jun 2026 and replaced by Qwen3.5-9B. Its frozen snapshot rates remain the v1.5 M2 reference; price basis tooltips and the correction note disclose this. Pricing disclosure correction SHA-256: 1b660648bd49….
Composite: weighted harmonic mean with the Intelligence, Speed and Cost gates below 50 (Intelligence below 60 in option C). The official headline A uses equal 25 / 25 / 25 / 25 axis weights and Intelligence floor 50. B remains the secondary 40 / 20 / 20 / 20 view; C keeps equal axes and Intelligence floor 60. Ties come from the paired bootstrap. The gates belong to the score, not to the weights: in the custom-weight views of the score chart they still apply to an axis set to 0, as in the published views — so “Intelligence only” follows the Intelligence column except for systems with Cost, Speed or Intelligence below 50 (a general-purpose LLM with Cost 39 keeps only (39/50)², about 0.61, of its score). Every gated row carries a “gate ×…” tag naming the axis and factor, and the weights panel lists the gates that fire.
All 22 full-coverage systems marked with a v1.5 roster addendum label are officially ranked in A, B, C, and every score preset. Their scores, individual intervals, method, pricing rules and frozen G_med are unchanged. No new paired-bootstrap comparison is inferred for addendum pairs.
Before every release we review the leaderboard for anomalies and close loopholes with general, documented rules. The page and Git repository provide transparent data and method details; Benchmark Heaven owns its rules.
Data file SHA-256 8b837b2b760ef238e2a465632a48aa95b53b9ce5d797dfa79b2c24b0a638911e · scorer output SHA-256 452885de2a84cd5b9ed393d541fd8d6c6a540f9d7f762f9d2df9349383f26173 · run kind official.
Limits
1,624 decisions per system (904 open, 720 sealed) is a measurement, not a census, and it is English-only.
The weights are a choice. Option A weights the four axes equally and uses a harmonic mean, so the weakest axis dominates; options B and C are published alternatives and the weight sliders re-score the same axes for exploration — only the official option gives the official score and rank. If a wrong decision costs you more than a slow or expensive one, read the Intelligence column and the per-type competence rather than the score alone.
The latency adjustment (×2, +0.15 s on our own servers and demo endpoints) is an assumption, not a measurement. We ran the self-hosted and demo endpoints one request at a time (parallelism 1, no other load), so their latency is likely better than the same model on a busy production server. Serving under load trades per-user speed for throughput. The +0.15 s stands for infrastructure our self-hosted tests lacked: authentication, load balancing, logging, billing and an API gateway. Both numbers are assumptions; raw p50/p95 latencies are in the table and the repo.
Held-out decisions are sent to the evaluated services to get predictions. Not public is not the same as not seen.
Latency is one origin at one time of day; hosted endpoints, public demos and our own pods are different kinds of latency. Public demo endpoints are shared with everyone else using them.
Estimated costs describe what a large inference provider would charge for a model of that size, not what the author pays; a system on a tariff pays its tariff.
Credit
Harness, public tasks and every scoring rule: github.com/fstandhartinger/jevbench (MIT). Each project links its author's repository or vendor page.
Decision 2B (FlyMy.AI, v59) — FlyMy.AI (@denti), Apache-2.0 notices on the included code and the pinned base; the weights are an evaluation preview under EVALUATION-PERMISSION.md, not a cleared commercial release — huggingface.co/flymy-ai/decision-2b-preview
Decision 4B v1.1 (FlyMyJev, Qwen3.5-4B + LoRA) — unknown, not recorded
Decision 4B v1.2 (FlyMyJev, Qwen3.5-4B + LoRA) — unknown, not recorded
Decision Fast (FlyMy.AI, v53a) — FlyMy.AI (@denti), Apache-2.0 notices on the included code and the pinned base; the weights are an evaluation preview under EVALUATION-PERMISSION.md, not a cleared commercial release — huggingface.co/flymy-ai/decision-fast-preview
GPT-5.6 Luna (low reasoning effort) — OpenAI, proprietary API
GPT-6 Luna (default medium reasoning effort) — OpenAI, proprietary API
GPT-6 Luna (low reasoning effort) — OpenAI, proprietary API
Hopper — HopitAI, Component-specific terms recorded in RESULT.md; submitted adapter release and Qwen base retain their respective terms — huggingface.co/HopitAI/hopper
jev-local (Qwen3.5-9B) — us (GitHub), no licence stated in the repository (public code); Apache-2.0 base weights — github.com/us/jev-local
Jev-Omni (akhilaaa3, Gemma-4-12B merged) — akhilaaa3, Apache-2.0, following Gemma 4; dataset rights stated separately by the author — huggingface.co/akhilaaa3/Jev-Omni
Malkuth-2B (newfull5, Kev post-train) — newfull5 (dhtocks), CC-BY-NC-4.0, research use only (XNLI and RACE in the training mix) — github.com/newfull5/malkuth
Malkuth-4B (newfull5, Kev post-train) — newfull5 (dhtocks), CC-BY-NC-4.0, research use only (XNLI and RACE in the training mix) — github.com/newfull5/malkuth
Open-Jev 2B (Zefan Cai) — Zefan Cai (@Zefan_Cai), MIT (loader); Apache-2.0 (adapter and pinned Qwen base); CC0-1.0 public training projection — github.com/Zefan-Cai/Open-Jev
Open-Jev 9B (Zefan Cai) — Zefan Cai (@Zefan_Cai), MIT (loader); Apache-2.0 (adapter and pinned Qwen base); CC0-1.0 public training projection — github.com/Zefan-Cai/Open-Jev
openjev-sglang (Qwen3.6-35B-A3B on SGLang) — ekzhang, no licence file in the repository as of 2026-09-19; Qwen3.6 weights keep their own terms — github.com/ekzhang/openjev-sglang
OpenSourceJev (Qwen3.5-4B Q4_K_M, native llama.cpp) — sabeel111, MIT (repository code); Apache-2.0 (Qwen/Qwen3.5-4B base and unsloth/Qwen3.5-4B-GGUF Q4_K_M conversion) — github.com/sabeel111/OpenSourceJev
Plumb-4B (crh225, JevK5 v0.2 + LoRA) — unknown, not recorded
spark-s1-4b-v6 (Open Spark Jev, abhishek085) — Abhishek Rai (abhishek085), Apache-2.0 (code and weights); base Qwen/Qwen3.5-4B Apache-2.0 — github.com/abhishek085/open-spark-jev
Standard One 8B (Standard Thinking) — Standard Thinking (myeongho12), Apache-2.0 (jev-adapter server, LoRA and merged weights; base Ministral 3 8B Apache-2.0) — huggingface.co/StandardThinking/StandardOne-8B
Surogate Rune 26B-A4B v3 (RTX PRO 6000) — unknown, not recorded
swanOne (blockbrain, Qwen3.8-Flash-Next NVFP4) — blockbrain, patches Apache-2.0, shim MIT; weights under the Qwen licence (LICENSE-NOTICE.md) — github.com/blockbrain-ai/swanone-recipe
system-one (Qwen3-8B, Sean Goedecke) — Sean Goedecke, no licence file in the repository as of 19 Sep; Qwen3 weights Apache-2.0 — github.com/sgoedecke/system-one
system-one-open (Gemma 4 E2B LoRA on an L4) — mithalouni, MIT (repository LICENSE; Gemma weights keep Google’s terms) — github.com/mithalouni/system-one-open
Authors: if we tested the wrong configuration, tell us and we will rerun it. New entrants become a new version or a disclosed roster addendum rather than silently changing this one.
Measured 2026-10-02 through 2026-10-04 (UTC); published 2026-10-09. Historical v1.5 pool: 1,624 decisions per model; scored with the frozen headline-A method and original G_med. These results are independent of v1.6.1 and do not enter its rankings.
21 complete measurements join this historical table. The frozen v1.5.7 ranking and its top five remain unchanged. Prices are estimates from the original declared references (*est.); latency includes the original self-hosted adjustment. Decision 2.0 Vega is the version measured in this pool.
Historical model measurements, ordered by original composite A score
Category radars use stored outcomes and the existing historical subject and use-case labels. Chance-corrected competence (0 = chance, 100 = perfect) over the category's items, open and sealed pooled, all tiers: per request type the method's competence cell (Choice and Noul accuracy above chance, Score 1 - error / chance error), then averaged over types weighted by item count. Not part of the JevBench Score; compare systems within a category, not categories with each other. Values can be negative (below chance). Samples below 15 remain gaps.
Xor 26B-A4B NVFP4 (Juspay): category radars and measurement details
Measured 2026-10-04 (UTC) on RTX PRO 6000 Blackwell. Licence: Apache-2.0 as stated in the model card; base model is Google Gemma 4.. Model or runtime repository.
Measured on the frozen v1.5.7 pool (1,624 items), self-hosted on RTX PRO 6000 Blackwell. The author reports public-set results; prefix caching does not improve this distinct-state evaluation input.
Base model: google/gemma-4-26b-a4b-it. Cost estimate: price floor: base-model reference price applied. Median latency: 0.084 s raw; 0.317 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Wald-Q4B v1.1, effort none (org2ai): category radars and measurement details
Measured 2026-10-04 (UTC) on RTX PRO 6000 Blackwell. Licence: Weights and bundled server are declared Apache-2.0; the author’s provenance file notes some training sources have unknown or restrictive licences.. Model or runtime repository.
Measured on the frozen v1.5.7 pool (1,624 items), self-hosted on RTX PRO 6000 Blackwell. The author disclosed that public items influenced checkpoint selection as a scoreboard, not as training data; calibration tables used held-out development data.
Base model: Qwen/Qwen3.5-4B-Base. Cost estimate: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off, deprecated afterwards (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). Median latency: 0.058 s raw; 0.265 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Measured 2026-10-03 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Merged rank-128 LoRA on all text projections of Qwen3.5-4B-Base plus a new candidate head; eight-seed weight soup. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.5-4B-Base. Cost estimate: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off, deprecated afterwards (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). Median latency: 0.041 s raw; 0.232 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Measured 2026-10-03 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Full fine-tune of Decision-1.0-Lux-9B with soft targets; Qwen3.5-9B lineage. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
First 18 of 32 Qwen3.5-4B-Base layers with a merged LoRA and four bootstrap-trained scorer heads. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.5-4B-Base. Cost estimate: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off, deprecated afterwards (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). Median latency: 0.024 s raw; 0.199 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
fastsem-jev l16r25 (KoshiBebop): category radars and measurement details
Measured 2026-10-04 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: MIT inference code; Apache-2.0 base weights. Model or runtime repository.
Training-free inference method on frozen Qwen3.5-4B weights, using the author l16r25 configuration in process. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.5-4B. Cost estimate: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off, deprecated afterwards (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). Median latency: 0.101 s raw; 0.351 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Rank-16 LoRA and pointer head; author runtime with KEV_FUSED=0 because the published serving lock omits the fused-kernel dependency. Stored head temperature retained. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.5-9B-Base. Cost estimate: price floor: base-model reference price applied. Median latency: 0.047 s raw; 0.243 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
janus 35B-A3B (Icarus AI / cmxu, NVFP4 MoE + LoRA r64 and pointer decision head): category radars and measurement details
Measured 2026-10-04 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
NVFP4 Qwen3.6-35B-A3B backbone with rank-64 LoRA and pointer head, measured on Blackwell hardware required by the routed-expert kernel. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.6-35B-A3B. Cost estimate: price floor: base-model reference price applied. Median latency: 0.050 s raw; 0.250 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
CoCo-Decision-4B-Ko (corners.ai): category radars and measurement details
Measured 2026-10-04 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Korean/English rank-16 LoRA served unmerged by the author oh-my-jev engine. The 96 GB card used bf16; the engine's optional nf4 fallback did not trigger. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.5-4B. Cost estimate: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off, deprecated afterwards (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). Median latency: 0.127 s raw; 0.403 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Pinned Qwen3-VL-2B-Instruct adapter and decision head, served by the author runtime. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3-VL-2B-Instruct. Cost estimate: documented hosted-model estimate; no exact base-model floor applies. Median latency: 2.656 s raw; 5.462 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Measured 2026-10-03 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Rank-64 LoRA and candidate head on the pinned Qwen3.8-27B backbone; base is loaded separately. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Measured 2026-10-03 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Full fine-tune of Decision-1.0-Sol-2B with soft targets; Qwen3.5-2B lineage. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.5-2B. Cost estimate: price floor: base-model reference price applied. Median latency: 0.029 s raw; 0.208 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Sev-2B-preview (sora42y): category radars and measurement details
Measured 2026-10-03 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: CC BY-NC 4.0 weights (non-commercial); Apache-2.0 code and base. Model or runtime repository.
Tree Forcing full fine-tune of Qwen3.5-2B-Base, interpolated with base weights using the author WiSE-FT alpha 0.7. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3.5-2B-Base. Cost estimate: price floor: base-model reference price applied. Median latency: 0.060 s raw; 0.269 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Measured 2026-10-03 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Full fine-tune of Decision-1.0-Eos-0.8B, itself a Qwen3.5-0.8B fine-tune. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Full merged bf16 checkpoint and pointer head; author runtime with KEV_FUSED=0. Stored head temperature retained. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Native option-probability readout without text generation, GGUF Q8_0, CPU with six threads. All 1,624 decisions complete. Original input-token estimate $0.03/M; generated output tokens zero (*est.). Historical v1.5 measurement; fresh v1.6 re-run pending. Probabilities are from the original 1–2 October run; latency is from the admitted 2 October quiet-host Speed-only run. The author reports using Jev API only for comparisons, with no Jev outputs used in training; teacher-written scenarios came from Apache-2.0 Qwen3.8-27B. Public JevBench results informed data design and public items were used for reporting; checkpoint selection used the author’s own held-out development loss.
Base model: Qwen/Qwen3.5-0.8B. Cost estimate: *est. Cost uses measured input tokens at the original declared size-class reference ($0.03 per million input tokens); output tokens are 0 because the pinned native classifier produces probabilities without text generation. Missing output count accepted under the 9 October decision. Historical v1.5 method; do not combine into current v1.6 ranking.. Median latency: 2.199 s raw; 4.549 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
janus 0.8B (Icarus AI / cmxu, Qwen3.5-0.8B + LoRA r64 and pointer decision head), GPU: category radars and measurement details
Measured 2026-10-04 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
GPU measurement of the pinned Qwen3.5-0.8B checkpoint with rank-64 LoRA and pointer head. The author image lacks causal-conv1d; its reference PyTorch convolution path was retained. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Measured 2026-10-03 (UTC) on RTX PRO 6000 Blackwell 96 GB. Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Full fine-tune of Qwen3-0.6B-Base with a candidate head trained from scratch; six-checkpoint weight average. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3-0.6B-Base. Cost estimate: documented hosted-model estimate; no exact base-model floor applies. Median latency: 0.013 s raw; 0.175 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
198M DeBERTa-v3-base encoder with two candidate-set attention layers and an MLP scoring head. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: microsoft/deberta-v3-base. Cost estimate: documented hosted-model estimate; no exact base-model floor applies. Median latency: 0.014 s raw; 0.177 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Pinned Qwen3-0.6B adapter and decision head, served by the author runtime. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: Qwen/Qwen3-0.6B. Cost estimate: documented hosted-model estimate; no exact base-model floor applies. Median latency: 0.015 s raw; 0.180 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Tachyone (munod, ModernBERT-large 396M + LoRA r16, choice head bank): category radars and measurement details
Measured 2026-10-03 (UTC) on CPU: AMD Zen 2, 6 cores / 12 threads (shared). Licence: Apache-2.0 weights and code; base retains its own Apache-2.0 licence. Model or runtime repository.
Author CPU backend on shared AMD Zen 2 hardware (6 cores / 12 threads, AVX2). Recorded CPU load makes Speed a disclosed lower bound; no CUDA fast path was enabled. Self-hosted; serial, one decision per request. Original ×2 + 0.15 s latency adjustment retained.
Base model: answerdotai/ModernBERT-large. Cost estimate: documented hosted-model estimate; no exact base-model floor applies. Median latency: 1.426 s raw; 3.003 s after ×2 + 0.15 s.
Subject topics
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Use cases
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.