Hosting = where inference runs; company = where the provider or lab is registered.
EU hosting means the route's inference runs inside the EU: an EU region, AWS Bedrock's EU cross-region (geo) profiles, Azure's Europe Data Zone, or a provider whose entire public fleet is documented as EU-hosted, each checked per model against the provider's documentation. Global deployments do not count, and neither does an EU billing region, an EU company or an EU control plane on its own. One disclosed company-policy exception stays in, marked “EU equivalent”.
Evidence requirements (benchmark evidence, priced provider, measured task tokens) sit above the table they apply to.
“Hide deprecated” removes retired models (every Artificial Analysis configuration retired, on no current DesignArena board) and older retired configurations of current models. Applies to model tables, charts, price views & model offers; benchmark evidence stays unfiltered.
JevBench is Benchmark Heaven's own benchmark for Jev-class decision models: state and a bounded rubric in, a typed answer out.
Frozen release · 1,624 decisions per system (904 open + 720 sealed; sealed decisions are half of Intelligence) · 110 ranked of 116 roster systems · only system-level sealed aggregates are published · aggregate results JSON sha256 2a0adbfa3b78…
Making decisions from images? Explore Image JevBench v0.1.5 and compare its systems.
Share this version · View live board · Previous release: JevBench v1.5.5
JevBench v1.5.6 · headline
Capability ranking of Jev-class systems
Capability Score averages Intelligence and Calibration. Jev 1.13.0 leads the Jev-class systems with 80.0.
Jev-class means at most 2× the cost and median latency of Jev. How we choose ↘
Wide coloured bar = Capability Score (0–100). Thin bars: cost above, median latency below; shared log ratio scale 0.25× … 64× Jev, ticks at 1× and 2× (cap). Shorter is cheaper or faster. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ or hover a row or thin bar for details.
Cost and latency are shown separately because they are nearly independent across systems (Spearman ρ = 0.08, n = 110).
Sorted by Capability Score, not numbered. Each row says which limit it misses, measured against Jev 1.13.0 ($0.032 per 1,000 decisions, median 0.62 s).
Jev-class = cost per decision at most 2× Jev 1.13.0's (≤ $0.065 per 1,000 decisions) and median latency at most 2× Jev 1.13.0's (≤ 1.23 s), the adjusted p50 — the same median the speed chart plots, not the four-axis Speed score. 63 of 110 systems qualify; the other 47, including the general-purpose LLMs, are listed below the divider in the ranking. The charts below show speed and cost beside Capability Score; the official JevBench Score weighs all four axes.
Jev-class systems are shown by default. Bubble size follows the official JevBench Score; official rank stays unchanged. The five most capable Jev-class systems are labelled.
Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.
Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). Self-hosted systems are plotted at the est. incl. network latency (2x measured + 150 ms); their measured loopback p50 (on-GPU, no network; not directly comparable with API latency) is in the tooltip. The dashed line is 2× Jev’s median latency.
Model kind
Jev-class
Release
No system in this release reports an exact parameter count.
What-If: the weight sliders below re-score every system under other axis weights — only the equal 25/25/25/25 weights give the official option-A ranking. The 3D view of capability, cost and speed loads further down.
JevBench v1.5.6
Official· four axes 0–100, equal-weight harmonic mean · Method notes ↓
Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).
Whiskers are 95% bootstrap intervals. 68 of 75 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 34 adjacent pairs, so no tie classification is inferred.
Greener = stronger within its column.
110 of 110 systems, sorted by official rank, #1 first.
Base-model notes
Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; the gates still apply when an axis sits at 0. Only equal weights give the official JevBench Score and rank.
Score = 4 / (1/I + 1/C + 1/S + 1/K) (each 0–100; × (axis / 50)² for Intelligence, Speed or Cost below 50)
Eligibility rule: services that run on Jev itself may be measured and shown as honorable mentions, but are not competitors ranked against Jev and do not enter the field median gap (G_med) or tie markers. classifier.dev (TypeSafe) runs on Jev, so it stays unranked under this rule.
Pick any two. Radars for the score axes, capability by subject topic and use cases, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Low sample, n < 30 — indicative only
| Category (items) | A: Jev 1.13.0 | B: Cygnet |
|---|---|---|
| Graphs and knowledge graphs (15) | 100.0 n=15 | 73.2 n=15 |
No published value for either system (under 15 answered items, or no per-category values): Scientific discovery (1), Semantic code linting (4), Feature extraction for predictive modeling (7), Recruiting (3), Moderation and trust and safety (12), Advertising (2).
Not drawn on the radar: with so few items a single answer moves a category score by several points, so these values are noise-prone. These values become spokes once the item pool reaches 30 items per category.
Not drawn: Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 159 items (76 open / 83 sealed) — not a use case of its own, so it is counted but not drawn.
No items in this release: Gaming, Demand forecasting.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.
Chance-corrected competence (0 = chance, 100 = perfect) over the category's items, open and sealed pooled, all tiers: per request type the method's competence cell (Choice and Noul accuracy above chance, Score 1 - error / chance error), then averaged over types weighted by item count. Not part of the JevBench Score; compare systems within a category, not categories with each other. Values can be negative (below chance).
Every one of the 1,624 decisions was labelled with one subject topic (the v1.2 topic list, unchanged) and one or two TypeSafe use-case categories (docs.typesafe.ai/concepts/use-case-map, plus Other) by Winnow-12B Q8 (the model behind System1 Models s1-pro) on our own GPU pod; 1,118 of the 3,248 calls were first made on s1-pro itself and agree 98.2 % with the pod run. Sealed items stayed on our own infrastructure. About 5 % of public items were checked by hand (Claude Opus 5.5); the item-group rules below fix the systematic misses found.
| Spoke | A: Jev 1.13.0 | B: Cygnet |
|---|---|---|
| The four score axes | ||
| Intelligence | 72.0 | 71.1 |
| Calibration | 88.0 | 87.0 |
| Speed | 83.8 | 91.0 |
| Cost | 54.7 | 56.4 |
| Capability by subject topic | ||
| Math & numbers | 40.5 | 31.4 |
| Coding & software | 80.3 | 87.8 |
| Rules, policy & law | 73.7 | 76.2 |
| Finance & commerce | 83.4 | 78.5 |
| Support & operations | 91.5 | 88.2 |
| Everyday language | 96.1 | 95.6 |
| Safety & security | 67.7 | 74.4 |
| Use cases (TypeSafe categories) | ||
| Search & retrieval | 95.9 | 94.7 |
| Model routing | 96.5 | 99.1 |
| LLM guardrails | 67.0 | 72.3 |
| Lead generation | 88.1 | 82.2 |
| Customer support | 91.9 | 88.9 |
| Insurance claims | 67.5 | 71.7 |
| Financial crime | 58.3 | 71.1 |
| Legal & compliance | 74.0 | 75.3 |
| E-commerce | 89.8 | 91.6 |
| Risk assessment | 80.1 | 73.1 |
| Competence per request type, open / sealed | ||
| Choice · open | 85.7 | 82.9 |
| Choice · sealed | 87.6 | 80.3 |
| Noul · open | 47.8 | 57.9 |
| Noul · sealed | 48.6 | 55.7 |
| Score · open | 81.2 | 76.6 |
| Score · sealed | 81.1 | 73.2 |
| Competence per tier — open set | ||
| Easy | 90.0 | 89.7 |
| Standard | 80.9 | 82.4 |
| Judge | 83.5 | 82.2 |
| Hard | 53.8 | 58.0 |
| Competence per tier — sealed set | ||
| Easy | 91.2 | 88.6 |
| Standard | 79.0 | 73.5 |
| Judge | 73.7 | 75.4 |
| Hard | 72.9 | 65.3 |
Every measured system, under the four score axes. Intelligence is 50% open (904 decisions) and 50% sealed (720); Gap = I_open − I_sealed, and the penalty applies only above the field median gap (G_med 5.2) plus 8. The per-type competence, latency and price of the same systems are in Types & cost. On a phone the name column stays put while the table scrolls sideways.
| #A | System | Score | Intel. | Calib. | Speed | Cost | Gap | Penalty |
|---|---|---|---|---|---|---|---|---|
| 1 | Cygnetdetails12BBase model: google/gemma-4-12B-itsource | 73.7 | 71.1 | 87.0 | 91.0 | 56.4 | +2.7 | ×1.000 |
| 2 | Winnow-12B Q8detailsmerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsource | 73.2 | 74.4 | 84.1 | 86.1 | 56.6 | -3.9 | ×1.000 |
| 3 | Jev 1.13.0APIdetailsJev referenceBase model: undisclosed | 72.1 | 72.0 | 88.0 | 83.8 | 54.7 | -0.9 | ×1.000 |
| 4 | JevK5 v0.3v1.5 roster addendum A1adapterBase model: Qwen/Qwen3.5-4Bsource | 71.9 | 56.3 | 88.3 | 93.6 | 63.1 | +10.7 | ×1.000 |
| 5 | Vansa-3.4v1.5 roster addendum A7APIdetailsClosed APIBase model: Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified. | 71.6 | 58.0 | 87.6 | 91.3 | 61.4 | +6.9 | ×1.000 |
| 6 | Plumb-4Bv1.5 roster addendum A1fine-tune · 4BBase model: alibiserikbay/JevK5source | 71.6 | 55.8 | 87.4 | 93.5 | 63.1 | +10.1 | ×1.000 |
| 7 | Jev-Omnidetailsfine-tuneBase model: google/gemma-4-12B-itsource | 71.5 | 70.5 | 82.6 | 84.7 | 56.1 | -0.0 | ×1.000 |
| 8 | decider-4b v2detailsadapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource | 71.3 | 55.8 | 85.6 | 90.9 | 64.5 | +11.9 | ×1.000 |
| 9 | Decision 4B v1.2v1.5 roster addendum A1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 70.8 | 53.7 | 88.6 | 93.5 | 63.1 | +5.2 | ×1.000 |
| 10 | Imajev-4Bv1.5 roster addendum A2adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 70.4 | 53.5 | 88.1 | 91.1 | 63.3 | +5.5 | ×1.000 |
| 11 | Decision 4B v1.1v1.5 roster addendum A1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 70.4 | 53.1 | 87.3 | 93.5 | 63.1 | +3.1 | ×1.000 |
| 12 | levv1.5 roster addendum A6detailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 69.1 | 57.7 | 77.9 | 86.5 | 61.8 | +1.1 | ×1.000 |
| 13 | Manchego v2.1v1.5 roster addendum A5detailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 68.8 | 51.2 | 84.9 | 88.6 | 64.3 | +6.4 | ×1.000 |
| 14 | SemIfdetails4BBase model: Qwen/Qwen3.5-4Bsource | 68.7 | 51.3 | 84.0 | 90.9 | 63.1 | +5.4 | ×1.000 |
| 15 | spark-s1-4b-v6detailsadapter · 4BBase model: Qwen3.5-4Bsource | 68.2 | 62.1 | 69.7 | 85.5 | 60.4 | +0.7 | ×1.000 |
| 16 | metask-jev-4bdetails4BBase model: Qwen3.5-4Bsource | 67.5 | 53.5 | 82.7 | 89.5 | 57.7 | +6.5 | ×1.000 |
| 17 | Hopperdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 67.5 | 49.9 | 87.9 | 87.2 | 62.3 | -2.3 | ×1.000 |
| 18 | Malkuth-4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource | 66.8 | 54.5 | 83.2 | 86.2 | 55.8 | +7.8 | ×1.000 |
| 19 | Surogate Rune 26B-A4B v3v1.5 roster addendum A226B-A4BBase model: google/gemma-4-26B-A4B-itsource | 66.5 | 69.7 | 88.3 | 86.0 | 49.0 | -1.6 | ×1.000 |
| 20 | reflex 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 65.2 | 51.5 | 86.8 | 68.8 | 63.1 | +7.4 | ×1.000 |
| 21 | jev-localdetails9BBase model: Qwen/Qwen3.5-9Bsource | 65.2 | 56.3 | 77.8 | 73.0 | 58.9 | +3.8 | ×1.000 |
| 22 | djev (Maisa, diffusion-gemma)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 64.2 | 72.3 | 80.4 | 91.0 | 48.2 | +0.8 | ×1.000 |
| 23 | Raw Qwen3 4B Instruct 2507 direct logitsdetailsoriginal · 4BBase model: Qwen/Qwen3-4B-Instruct-2507 (unmodified)source | 62.1 | 54.1 | 52.9 | 89.2 | 63.4 | +6.7 | ×1.000 |
| 24 | jqvdetails32BBase model: Qwen3-32Bsource | 60.8 | 49.1 | 86.8 | 83.4 | 51.2 | +4.5 | ×1.000 |
| 25 | JevK5 v0.2.0detailsadapter · 4BBase model: Qwen3.5-4Bsource | 58.1 | 46.7 | 84.9 | 90.9 | 63.1 | +6.6 | ×1.000 |
| 26 | Clef-Flashv1.5 roster addendum A6detailsfine-tuneBase model: Qwen/Qwen3.5-9Bsource | 55.1 | 53.1 | 87.6 | 83.6 | 46.8 | +11.5 | ×1.000 |
| 27 | Qwen3.5-9B Jev-like data-mix v2detailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource | 53.0 | 60.4 | 80.9 | 82.0 | 45.7 | +4.3 | ×1.000 |
| 28 | Standard One 8Bdetailsadapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source | 47.8 | 59.6 | 83.2 | 92.5 | 43.3 | +0.7 | ×1.000 |
| 29 | NInfer Qwen3.8-Flash-Next mixeddetailsLLM decoderBase model: Qwen3.8-Flash-Nextsource | 47.5 | 67.2 | 88.5 | 88.6 | 42.5 | +6.1 | ×1.000 |
| 30 | Instinct Dual 4Bv1.5 roster addendum A4APIdetailsClosed APIBase model: Qwen3.5-4Bsource · Operator-reported; weights not publicly verifiable. | 47.0 | 43.1 | 88.3 | 82.0 | 60.2 | +6.3 | ×1.000 |
| 31 | swanOnedetailsNVFP4Base model: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source | 46.6 | 71.2 | 87.1 | 84.7 | 42.2 | -1.5 | ×1.000 |
| 32 | Raw Qwen3 8B direct logitsdetailsoriginal · 8BBase model: Qwen/Qwen3-8B-Basesource | 45.2 | 51.1 | 49.2 | 86.8 | 45.5 | +11.3 | ×1.000 |
| 33 | decider-2bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2B-Basesource | 45.1 | 42.3 | 71.5 | 94.4 | 64.9 | +12.9 | ×1.000 |
| 34 | system-onedetails8BBase model: Qwen3-8Bsource | 44.1 | 50.5 | 49.4 | 90.9 | 45.0 | +6.1 | ×1.000 |
| 35 | system-one-openAPIdetailsadapter · 2BBase model: Gemma 4 E2Bsource | 42.4 | 41.6 | 71.6 | 78.3 | 68.3 | +5.6 | ×1.000 |
| 36 | Autoloops – Gemma 4 31B ITAPIdetailsClosed APIBase model: undisclosed | 40.5 | 76.7 | 85.8 | 83.9 | 39.6 | +0.4 | ×1.000 |
| 37 | GPT-6 Luna (low reasoning effort)APIdetailsClosed APIBase model: undisclosed | 40.5 | 95.3 | 94.9 | 73.2 | 39.1 | -1.1 | ×1.000 |
| 38 | GPT-6 Luna (default medium reasoning effort)APIdetailsClosed APIBase model: undisclosed | 38.8 | 96.2 | 95.6 | 73.2 | 38.3 | +0.2 | ×1.000 |
| 39 | JevOnedetailsadapterBase model: Qwen/Qwen3.6-35B-A3Bsource | 38.2 | 54.1 | 84.9 | 90.4 | 39.8 | -0.5 | ×1.000 |
| 40 | kev 4Bdetailsadapter · 4BBase model: Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base. | 38.1 | 39.9 | 67.6 | 85.5 | 65.8 | +13.9 | ×0.993 |
| 41 | kev 8Bdetailsadapter · 8BBase model: Qwen/Qwen3-8B-Basesource | 34.2 | 48.3 | 71.0 | 84.1 | 40.4 | +12.0 | ×1.000 |
| 42 | open-alternative-jevdetails4BBase model: Qwen3.5-4Bsource | 33.6 | 37.4 | 76.9 | 91.3 | 63.3 | +10.5 | ×1.000 |
| 43 | Bespoke Nimble 9Bdetailsadapter · 9BBase model: Qwen3.5-9Bsource | 31.8 | 63.7 | 77.2 | 83.0 | 36.8 | +2.8 | ×1.000 |
| 44 | Malkuth-2Bdetailsadapter · 2BBase model: empero-ai/Qwen3.8-2B-Distillsource | 29.9 | 35.5 | 75.2 | 91.8 | 65.6 | +15.6 | ×0.976 |
| 45 | openjev-sglangAPIdetails35B-A3BBase model: Qwen/Qwen3.6-35B-A3Bsource | 29.0 | 58.6 | 82.9 | 78.1 | 35.6 | +5.4 | ×1.000 |
| 46 | decider-35b-a3bdetailsfine-tune · 35B-A3BBase model: Qwen/Qwen3.5-35B-A3B-Basesource | 27.5 | 60.5 | 82.0 | 91.0 | 34.4 | +7.4 | ×1.000 |
| 47 | local-jev Qwen3.5-4Bdetailsfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource | 25.8 | 33.7 | 82.4 | 83.7 | 59.3 | +5.9 | ×1.000 |
| 48 | Nemotron Diffusion 8Bv1.5 roster addendum A3details8BBase model: nvidia/Nemotron-Labs-Diffusion-8Bsource | 25.7 | 33.8 | 76.2 | 93.7 | 55.5 | +11.4 | ×1.000 |
| 49 | Open-Jev 9Bdetailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource | 24.4 | 63.8 | 81.5 | 73.6 | 33.1 | -4.1 | ×1.000 |
| 50 | Decision 2Bdetailsadapter · 2BBase model: openbmb/MiniCPM5-2Bsource | 22.5 | 31.3 | 86.2 | 90.4 | 66.4 | +6.7 | ×1.000 |
| 51 | GPT-5.6 LunaAPIdetailsClosed APIBase model: undisclosed | 22.4 | 94.3 | 94.7 | 74.1 | 30.7 | -2.3 | ×1.000 |
| 52 | typecastlmdetails4BBase model: Qwen3.5-4Bsource | 21.8 | 31.2 | 76.7 | 92.0 | 64.0 | -15.1 | ×1.000 |
| 53 | JEV Qwen3.5-9B Base NVFP4details9B · NVFP4Base model: ig1/Qwen3.5-9B-NVFP4source | 20.1 | 32.2 | 80.9 | 93.5 | 47.6 | +1.3 | ×1.000 |
| 54 | Gemini 3.1 Flash-LiteAPIdetailsClosed APIBase model: undisclosed | 19.6 | 77.6 | 74.7 | 80.1 | 29.8 | -1.9 | ×1.000 |
| 55 | AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1fine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource | 19.5 | 72.8 | 87.7 | 87.6 | 29.4 | -1.7 | ×1.000 |
| 56 | AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2fine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource | 19.5 | 72.8 | 86.7 | 87.1 | 29.4 | -1.5 | ×1.000 |
| 57 | NInfer Qwen3.8-27B NVFP4details27BBase model: Qwen3.8-27Bsource | 18.7 | 65.5 | 85.9 | 89.9 | 29.1 | +3.0 | ×1.000 |
| 58 | Eikos-27Bv1.5 roster addendum A1adapter · 27BBase model: Qwen/Qwen3.8-27Bsource | 18.5 | 75.1 | 86.3 | 87.6 | 28.7 | -1.9 | ×1.000 |
| 59 | NInfer Qwen3.8-27B NVFP4 (T=1.5)details27BBase model: Qwen3.8-27Bsource | 18.5 | 61.1 | 86.5 | 89.9 | 29.1 | +2.7 | ×1.000 |
| 60 | InstinctAPIdetailsClosed APIBase model: Qwen3.8-27Bsource · Operator-reported; weights not publicly verifiable. | 18.3 | 62.7 | 85.0 | 81.7 | 29.2 | -1.7 | ×1.000 |
| 61 | OpenJev (thinking, BF16)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource 1source 2 | 17.9 | 84.2 | 83.1 | 73.9 | 28.5 | -3.4 | ×1.000 |
| 62 | djev (thinking)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 17.4 | 77.3 | 95.7 | 72.3 | 28.1 | -1.6 | ×1.000 |
| 63 | Clefv1.5 roster addendum A6detailsfine-tuneBase model: Qwen/Qwen3.8-27Bsource | 17.0 | 67.9 | 86.7 | 79.9 | 28.1 | +0.4 | ×1.000 |
| 64 | LitJevdetails27BBase model: Qwen/Qwen3.8-27Bsource | 16.3 | 58.3 | 84.5 | 68.2 | 28.4 | -0.5 | ×1.000 |
| 65 | Bev / Bonsai 27Bv1.5 roster addendum A4details27B · GGUF PQ2_0Base model: Qwen/Qwen3.8-27Bsource | 15.8 | 53.1 | 77.7 | 72.7 | 28.2 | -8.8 | ×1.000 |
| 66 | Raw Phi-4 mini direct logitsdetailsoriginalBase model: microsoft/Phi-4-mini-instruct (unmodified)source | 15.2 | 27.6 | 71.4 | 89.2 | 53.2 | +18.3 | ×0.949 |
| 67 | OpenSourceJevdetailsadapter · 4B · GGUF Q4_K_MBase model: Qwen3.5-4Bsource | 13.2 | 25.8 | 76.0 | 72.5 | 69.1 | +4.2 | ×1.000 |
| 68 | reflex-27bdetails27BBase model: Qwen3.8-27Bsource | 13.2 | 62.8 | 85.9 | 69.2 | 25.8 | +0.6 | ×1.000 |
| 69 | Open-Jev 2Bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2Bsource | 9.1 | 33.6 | 73.6 | 75.8 | 33.1 | +10.4 | ×1.000 |
| 70 | GLiNER2 largedetailsfine-tuneBase model: microsoft/deberta-v3-largesource | 8.4 | 22.5 | 42.4 | 64.8 | 77.6 | +7.1 | ×1.000 |
| 71 | Qwen3-Reranker-4BdetailsRerankerBase model: Qwen/Qwen3-4B-Basesource | 7.1 | 21.0 | 76.3 | 79.6 | 48.4 | +17.0 | ×0.962 |
| 72 | DeepSeek V4.1 FlashAPIdetailsLLM decoderBase model: undisclosed | 6.6 | 93.7 | 96.9 | 69.4 | 19.1 | -3.7 | ×1.000 |
| 73 | SimpleJevdetails0.8BBase model: Qwen/Qwen3.5-0.8Bsource | 4.0 | 16.7 | 46.7 | 59.1 | 70.3 | -6.8 | ×1.000 |
| 74 | SimpleJev Qwen3.8-27BAPIdetails27BBase model: Qwen/Qwen3.8-27Bsource | 3.4 | 72.8 | 87.2 | 74.9 | 14.9 | -1.7 | ×1.000 |
| 75 | decision-machine-1APIdetailsClosed APIBase model: undisclosed | 3.2 | 14.8 | 81.2 | 92.8 | 56.3 | +22.4 | ×0.908 |
| 76 | GLiNER2.5 multidetailsfine-tune · 287MBase model: mDeBERTa-v3-basesource | 2.7 | 14.0 | 58.0 | 66.6 | 86.6 | +3.6 | ×1.000 |
| 77 | Bosun v3.1 0.6Bv1.5 roster addendum A4detailsadapter · 0.6BBase model: Qwen/Qwen3-0.6Bsource | 2.5 | 13.6 | 65.2 | 61.8 | 77.5 | +17.8 | ×0.954 |
| 78 | GLiNER2detailsadapterBase model: DeBERTa-v3-basesource | 2.3 | 13.5 | 35.6 | 70.3 | 86.6 | +7.5 | ×1.000 |
| 79 | Deem 0.8B v1v1.5 roster addendum A4detailsLLM decoderBase model: Qwen/Qwen3.5-0.8Bsource | 2.1 | 13.0 | 38.9 | 84.3 | 80.3 | +21.1 | ×0.921 |
| 80 | JevActAPIdetailsClosed APIBase model: undisclosedsource | 1.5 | 11.2 | 62.5 | 76.4 | 68.6 | +16.3 | ×0.969 |
| 81 | CLM-8Bdetailsfine-tune · 8BBase model: Qwen/Qwen3-8Bsource | 1.5 | 11.4 | 48.5 | 93.3 | 50.5 | +11.3 | ×1.000 |
| 82 | kev 0.6Bdetailsadapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 1.3 | 10.3 | 67.7 | 87.2 | 80.1 | +27.0 | ×0.862 |
| 83 | Raw Qwen3 0.6B direct logitsdetailsoriginal · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 1.1 | 10.6 | 21.5 | 90.4 | 77.6 | -1.1 | ×1.000 |
| 84 | GLiNER2.5 smalldetailsfine-tune · 74MBase model: microsoft/deberta-v3-xsmallsource | 0.9 | 9.2 | 55.9 | 77.4 | 86.6 | +15.5 | ×0.977 |
| 85 | Raw Qwen3 1.7B direct logitsdetailsoriginal · 1.7BBase model: Qwen/Qwen3-1.7B-Basesource | 0.9 | 9.7 | 21.6 | 90.2 | 68.5 | +5.2 | ×1.000 |
| 86 | MirrordetailsEncoder / classifierBase model: undisclosed | 0.2 | 5.6 | 43.2 | 63.6 | 89.3 | +2.3 | ×1.000 |
| 87 | ZeroEntropy zerank-2detailsRerankerBase model: Qwen/Qwen3-4Bsource | 0.1 | 4.8 | 81.7 | 80.3 | 48.4 | +10.4 | ×1.000 |
| 88 | jeffdetailsEncoder / classifierBase model: undisclosedsource | 0.1 | 4.4 | 80.3 | 55.8 | 81.1 | +16.3 | ×0.969 |
| 89 | smalljev semantic-v9detailsadapter · 2BBase model: openbmb/MiniCPM5-2B-Basesource | 0.1 | 3.7 | 73.1 | 86.5 | 60.8 | +14.5 | ×0.987 |
| 90 | Laya multilingualv1.5 roster addendum A4detailsfine-tune · 322MBase model: mmBERT-basesource | 0.0 | 2.4 | 43.6 | 73.7 | 82.2 | +0.9 | ×1.000 |
| 91 | OpenDecisiondetailsEncoder / classifierBase model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0source | 0.0 | 2.1 | 72.5 | 86.6 | 79.1 | +14.7 | ×0.985 |
| 92 | BAAI bge-reranker-v2-m3detailsfine-tuneBase model: BAAI/bge-m3source | 0.0 | 0.0 | 83.3 | 90.5 | 59.3 | +1.7 | ×1.000 |
| 93 | Certo v1detailsEncoder / classifierBase model: ModernBERT-largesource | 0.0 | 0.0 | 87.5 | 91.4 | 96.9 | +0.8 | ×1.000 |
| 94 | Decision Fastdetailsadapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 0.0 | 0.0 | 76.1 | 91.4 | 80.1 | +19.0 | ×0.942 |
| 95 | Alibaba GTE Reranker ModernBERT-basedetailsfine-tuneBase model: answerdotai/ModernBERT-basesource | 0.0 | 0.0 | 75.2 | 91.5 | 69.3 | +4.5 | ×1.000 |
| 96 | kev 0.5Bdetailsadapter · 0.5BBase model: Qwen/Qwen2.5-0.5Bsource | 0.0 | 0.0 | 65.0 | 88.5 | 80.1 | +25.7 | ×0.875 |
| 97 | Layadetailsfine-tune · 421MBase model: ModernBERT-largesource | 0.0 | 0.0 | 73.7 | 73.9 | 84.9 | +12.7 | ×1.000 |
| 98 | lev-350mdetailsadapterBase model: LiquidAI/LFM2.5-350Msource | 0.0 | 0.0 | 77.7 | 93.6 | 80.0 | +23.5 | ×0.896 |
| 99 | Qwen3.5-0.8B Decision Modeldetailsfine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource | 0.0 | 0.0 | 73.6 | 71.8 | 79.5 | +7.1 | ×1.000 |
| 100 | Mixedbread mxbai-rerank-base-v2detailsRerankerBase model: Qwen2.5 (size not stated)source 1source 2 | 0.0 | 0.0 | 86.6 | 89.3 | 60.3 | +3.0 | ×1.000 |
| 101 | Needle 3detailsoriginal · 121MBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 34.1 | 61.6 | +10.0 | ×1.000 |
| 102 | Needle 3, options as toolsdetailsoriginal · 121MBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 41.0 | 61.6 | +14.1 | ×0.991 |
| 103 | open-jev-deberta-v3-largedetailsadapterBase model: microsoft/deberta-v3-largesource | 0.0 | 0.0 | 77.1 | 68.3 | 77.6 | +5.3 | ×1.000 |
| 104 | Open Jev JSON Canvasdetails26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 0.0 | 77.1 | 0.0 | 85.6 | 49.3 | +4.0 | ×1.000 |
| 105 | openJev Verdictdetailsfine-tuneBase model: knowledgator/gliclass-modern-base-v2.0source 1source 2 | 0.0 | 0.0 | 52.2 | 83.9 | 86.6 | +26.7 | ×0.865 |
| 106 | openJev Verdict 1.4detailsEncoder / classifierBase model: knowledgator/gliclass-modern-base-v2.0source | 0.0 | 0.0 | 80.3 | 80.6 | 86.6 | +3.5 | ×1.000 |
| 107 | Qwen3.8 27BAPIdetails27BBase model: undisclosed | 0.0 | 95.6 | 98.1 | 56.8 | 0.0 | -5.3 | ×1.000 |
| 108 | verdict-smalldetailsfine-tune · 118MBase model: intfloat/multilingual-e5-smallsource | 0.0 | 0.0 | 59.5 | 82.1 | 100.0 | +2.0 | ×1.000 |
| 109 | Vondetailsfine-tune · 395MBase model: answerdotai/ModernBERT-largesource | 0.0 | 0.0 | 83.5 | 75.7 | 82.7 | +14.7 | ×0.985 |
| 110 | Laya typed-decisionsv1.5 roster addendum A4detailsfine-tune · 421MBase model: ModernBERT-largesource | 0.0 | 0.0 | 83.3 | 63.4 | 82.5 | +8.5 | ×1.000 |
| – | classifier.devAPIhonorable mentiondetailsClosed APIBase model: undisclosed | — | 75.8 | 89.2 | 82.0 | 58.9 | -2.7 | ×1.000 |
| – | SimpleJev Qwen3.6-35B-A3BAPIpartial rundetails35B-A3BBase model: undisclosedsource | — | 0.0 | 78.5 | 75.2 | 35.1 | +5.6 | ×1.000 |
| – | Decision-4Bv1.5 roster addendum A4not rankeddetailsadapterBase model: Qwen/Qwen3.5-4Bsource | — | — | — | — | — | — | — |
The same systems, by what was asked and what it cost. Per-type columns are chance-corrected competence (CC, 0 = chance) for Choice, Noul and Score, open / sealed; I open and I sealed are the two halves of Intelligence. Latency is adjusted p50 / p95; cost is per 1,000 decisions. On a phone the name column stays put while the table scrolls sideways.
| #A | System | I open | I sealed | Choice o / s | Noul o / s | Score o / s | p50 / p95 | $/1k decisions | Endpoint |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Cygnetdetails12BBase model: google/gemma-4-12B-itsource | 72.5 | 69.7 | 83 / 80 | 58 / 56 | 77 / 73 | 0.04 s / 0.10 son-GPU, no networkest. 0.23 s / 0.35 s* | $0.028 | GPU pod (RTXPRO6000) |
| 2 | Winnow-12B Q8detailsmerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsource | 72.5 | 76.4 | 78 / 82 | 59 / 66 | 81 / 81 | 0.10 s / 0.28 son-GPU, no networkest. 0.34 s / 0.72 s* | $0.028 | GPU pod (RTX6000) |
| 3 | Jev 1.13.0APIdetailsJev referenceBase model: undisclosed | 71.5 | 72.5 | 86 / 88 | 48 / 49 | 81 / 81 | 0.62 s / 0.67 s | $0.032 | hosted API |
| 4 | JevK5 v0.3v1.5 roster addendum A1adapterBase model: Qwen/Qwen3.5-4Bsource | 61.6 | 50.9 | 80 / 74 | 42 / 15 | 62 / 64 | 0.02 s / 0.05 son-GPU, no networkest. 0.18 s / 0.24 s* | $0.017 | GPU pod (H100) |
| 5 | Vansa-3.4v1.5 roster addendum A7APIdetailsClosed APIBase model: Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified. | 61.5 | 54.6 | 79 / 75 | 42 / 24 | 64 / 65 | 0.14 s / 0.52 s | $0.019 | hosted API |
| 6 | Plumb-4Bv1.5 roster addendum A1fine-tune · 4BBase model: alibiserikbay/JevK5source | 60.9 | 50.8 | 80 / 73 | 43 / 17 | 59 / 63 | 0.02 s / 0.05 son-GPU, no networkest. 0.18 s / 0.24 s* | $0.017 | GPU pod (H100) |
| 7 | Jev-Omnidetailsfine-tuneBase model: google/gemma-4-12B-itsource | 70.5 | 70.5 | 83 / 79 | 55 / 57 | 73 / 76 | 0.11 s / 0.38 son-GPU, no networkest. 0.38 s / 0.91 s* | $0.029 | GPU pod (RTX6000) |
| 8 | decider-4b v2detailsadapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource | 61.7 | 49.8 | 80 / 77 | 43 / 19 | 62 / 53 | 0.03 s / 0.13 son-GPU, no networkest. 0.20 s / 0.40 s* | $0.015 | GPU pod (RTX5090) |
| 9 | Decision 4B v1.2v1.5 roster addendum A1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 56.3 | 51.1 | 81 / 74 | 20 / 16 | 68 / 63 | 0.02 s / 0.05 son-GPU, no networkest. 0.18 s / 0.24 s* | $0.017 | GPU pod (H100) |
| 10 | Imajev-4Bv1.5 roster addendum A2adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 56.2 | 50.7 | 80 / 82 | 22 / 8 | 67 / 62 | 0.04 s / 0.09 son-GPU, no networkest. 0.23 s / 0.33 s* | $0.017 | GPU pod (RTX 5090) |
| 11 | Decision 4B v1.1v1.5 roster addendum A1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 54.7 | 51.6 | 80 / 75 | 18 / 16 | 66 / 64 | 0.02 s / 0.05 son-GPU, no networkest. 0.18 s / 0.24 s* | $0.017 | GPU pod (H100) |
| 12 | levv1.5 roster addendum A6detailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 58.2 | 57.1 | 67 / 71 | 49 / 45 | 59 / 55 | 0.13 s / 0.20 son-GPU, no networkest. 0.40 s / 0.55 s* | $0.019 | GPU pod (H100) |
| 13 | Manchego v2.1v1.5 roster addendum A5detailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 54.4 | 48.0 | 67 / 65 | 34 / 17 | 62 / 62 | 0.08 s / 0.14 son-GPU, no networkest. 0.32 s / 0.43 s* | $0.015 | GPU pod (RTX6000) |
| 14 | SemIfdetails4BBase model: Qwen/Qwen3.5-4Bsource | 54.0 | 48.6 | 74 / 71 | 30 / 14 | 58 / 61 | 0.04 s / 0.10 son-GPU, no networkest. 0.23 s / 0.36 s* | $0.017 | GPU pod (RTX5090) |
| 15 | spark-s1-4b-v6detailsadapter · 4BBase model: Qwen3.5-4Bsource | 62.5 | 61.8 | 73 / 73 | 50 / 46 | 64 / 66 | 0.13 s / 0.27 son-GPU, no networkest. 0.41 s / 0.69 s* | $0.021 | GPU pod (RTX6000) |
| 16 | metask-jev-4bdetails4BBase model: Qwen3.5-4Bsource | 56.7 | 50.2 | 70 / 72 | 43 / 25 | 57 / 53 | 0.07 s / 0.12 son-GPU, no networkest. 0.29 s / 0.39 s* | $0.026 | GPU pod (RTX5090) |
| 17 | Hopperdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 48.7 | 51.0 | 70 / 75 | 10 / 13 | 66 / 65 | 0.12 s / 0.16 son-GPU, no networkest. 0.39 s / 0.48 s* | $0.018 | GPU pod (A6000) |
| 18 | Malkuth-4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource | 58.4 | 50.5 | 71 / 70 | 39 / 23 | 64 / 59 | 0.15 s / 0.20 son-GPU, no networkest. 0.44 s / 0.55 s* | $0.030 | GPU pod (RTX5090) |
| 19 | Surogate Rune 26B-A4B v3v1.5 roster addendum A226B-A4BBase model: google/gemma-4-26B-A4B-itsource | 69.0 | 70.5 | 82 / 82 | 46 / 50 | 79 / 80 | 0.10 s / 0.29 son-GPU, no networkest. 0.35 s / 0.72 s* | $0.050 | GPU pod (RTX PRO 6000) |
| 20 | reflex 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 55.2 | 47.8 | 74 / 69 | 33 / 19 | 59 / 55 | 1.4 s / 2.2 son-GPU, no networkest. 2.9 s / 4.6 s* | $0.017 | GPU pod (H100) |
| 21 | jev-localdetails9BBase model: Qwen/Qwen3.5-9Bsource | 58.2 | 54.4 | 67 / 65 | 47 / 37 | 61 / 61 | 0.40 s / 2.6 son-GPU, no networkest. 0.94 s / 5.3 s* | $0.024 | GPU pod (H100) |
| 22 | djev (Maisa, diffusion-gemma)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 72.7 | 71.9 | 77 / 73 | 66 / 70 | 75 / 73 | 0.05 s / 0.08 son-GPU, no networkest. 0.25 s / 0.32 s* | $0.053 | GPU pod (H100) |
| 23 | Raw Qwen3 4B Instruct 2507 direct logitsdetailsoriginal · 4BBase model: Qwen/Qwen3-4B-Instruct-2507 (unmodified)source | 57.4 | 50.7 | 65 / 63 | 50 / 38 | 57 / 51 | 0.06 s / 0.15 son-GPU, no networkest. 0.26 s / 0.46 s* | $0.017 | GPU pod (A6000) |
| 24 | jqvdetails32BBase model: Qwen3-32Bsource | 51.3 | 46.8 | 73 / 73 | 19 / 10 | 62 / 57 | 0.08 s / 0.68 son-GPU, no networkest. 0.31 s / 1.5 s* | $0.042 | GPU pod (H100) |
| 25 | JevK5 v0.2.0detailsadapter · 4BBase model: Qwen3.5-4Bsource | 50.0 | 43.4 | 78 / 72 | 12 / -4 | 60 / 62 | 0.03 s / 0.11 son-GPU, no networkest. 0.22 s / 0.38 s* | $0.017 | GPU pod (RTX5090) |
| 26 | Clef-Flashv1.5 roster addendum A6detailsfine-tuneBase model: Qwen/Qwen3.5-9Bsource | 58.9 | 47.3 | 77 / 76 | 34 / 5 | 65 / 61 | 0.20 s / 0.32 son-GPU, no networkest. 0.55 s / 0.79 s* | $0.059 | GPU pod (H100) |
| 27 | Qwen3.5-9B Jev-like data-mix v2detailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource | 62.6 | 58.3 | 73 / 75 | 42 / 32 | 73 / 68 | 0.22 s / 0.47 son-GPU, no networkest. 0.58 s / 1.1 s* | $0.065 | GPU pod (A6000) |
| 28 | Standard One 8Bdetailsadapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source | 60.0 | 59.2 | 74 / 72 | 45 / 43 | 61 / 63 | 0.03 s / 0.06 son-GPU, no networkest. 0.20 s / 0.28 s* | $0.078 | GPU pod (RTX5090) |
| 29 | NInfer Qwen3.8-Flash-Next mixeddetailsLLM decoderBase model: Qwen3.8-Flash-Nextsource | 70.3 | 64.1 | 85 / 79 | 51 / 43 | 74 / 70 | 0.08 s / 0.15 son-GPU, no networkest. 0.31 s / 0.44 s* | $0.082 | GPU pod (RTXPRO6000) |
| 30 | Instinct Dual 4Bv1.5 roster addendum A4APIdetailsClosed APIBase model: Qwen3.5-4Bsource · Operator-reported; weights not publicly verifiable. | 46.3 | 39.9 | 75 / 74 | 4 / -13 | 59 / 58 | 0.51 s / 1.3 s | $0.021 | author demo endpoint |
| 31 | swanOnedetailsNVFP4Base model: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source | 70.5 | 71.9 | 88 / 83 | 46 / 60 | 78 / 73 | 0.21 s / 0.22 son-GPU, no networkest. 0.57 s / 0.59 s* | $0.085 | GPU pod (RTXPRO6000) |
| 32 | Raw Qwen3 8B direct logitsdetailsoriginal · 8BBase model: Qwen/Qwen3-8B-Basesource | 56.8 | 45.5 | 65 / 57 | 48 / 38 | 57 / 42 | 0.09 s / 0.24 son-GPU, no networkest. 0.33 s / 0.62 s* | $0.065 | GPU pod (A6000) |
| 33 | decider-2bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2B-Basesource | 48.8 | 35.9 | 52 / 53 | 35 / 6 | 59 / 48 | 0.01 s / 0.03 son-GPU, no networkest. 0.18 s / 0.20 s* | $0.015 | GPU pod (H100) |
| 34 | system-onedetails8BBase model: Qwen3-8Bsource | 53.5 | 47.4 | 63 / 57 | 50 / 40 | 48 / 46 | 0.04 s / 0.11 son-GPU, no networkest. 0.23 s / 0.36 s* | $0.068 | GPU pod (RTX5090) |
| 35 | system-one-openAPIdetailsadapter · 2BBase model: Gemma 4 E2Bsource | 44.4 | 38.8 | 60 / 52 | 15 / 15 | 58 / 50 | 1.2 s / 1.3 s | $0.011 | author demo endpoint |
| 36 | Autoloops – Gemma 4 31B ITAPIdetailsClosed APIBase model: undisclosed | 76.9 | 76.5 | 86 / 88 | 69 / 67 | 76 / 75 | 0.61 s / 0.67 s | tariff$0.103 | hosted API |
| 37 | GPT-6 Luna (low reasoning effort)APIdetailsClosed APIBase model: undisclosed | 94.7 | 95.9 | 99 / 98 | 85 / 90 | 100 / 100 | 1.6 s / 3.0 s | $0.108 | hosted API |
| 38 | GPT-6 Luna (default medium reasoning effort)APIdetailsClosed APIBase model: undisclosed | 96.3 | 96.2 | 99 / 100 | 91 / 90 | 99 / 99 | 1.6 s / 3.1 s | $0.114 | hosted API |
| 39 | JevOnedetailsadapterBase model: Qwen/Qwen3.6-35B-A3Bsource | 53.9 | 54.4 | 82 / 77 | 9 / 17 | 70 / 69 | 0.06 s / 0.09 son-GPU, no networkest. 0.27 s / 0.34 s* | $0.101 | GPU pod (RTXPRO6000) |
| 40 | kev 4Bdetailsadapter · 4BBase model: Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base. | 47.1 | 33.2 | 54 / 47 | 29 / 4 | 58 / 49 | 0.17 s / 0.21 son-GPU, no networkest. 0.49 s / 0.57 s* | $0.014 | GPU pod (A6000) |
| 41 | kev 8Bdetailsadapter · 8BBase model: Qwen/Qwen3-8B-Basesource | 54.3 | 42.3 | 61 / 59 | 38 / 14 | 64 / 54 | 0.18 s / 0.30 son-GPU, no networkest. 0.51 s / 0.75 s* | $0.097 | GPU pod (A6000) |
| 42 | open-alternative-jevdetails4BBase model: Qwen3.5-4Bsource | 42.6 | 32.1 | 63 / 65 | 5 / -17 | 59 / 48 | 0.04 s / 0.09 son-GPU, no networkest. 0.22 s / 0.33 s* | $0.017 | GPU pod (RTX5090) |
| 43 | Bespoke Nimble 9Bdetailsadapter · 9BBase model: Qwen3.5-9Bsource | 65.1 | 62.3 | 72 / 73 | 53 / 45 | 71 / 70 | 0.20 s / 0.39 son-GPU, no networkest. 0.55 s / 0.93 s* | $0.128 | GPU pod (A6000) |
| 44 | Malkuth-2Bdetailsadapter · 2BBase model: empero-ai/Qwen3.8-2B-Distillsource | 44.2 | 28.6 | 55 / 48 | 26 / -6 | 52 / 43 | 0.04 s / 0.07 son-GPU, no networkest. 0.23 s / 0.29 s* | $0.014 | GPU pod (RTX5090) |
| 45 | openjev-sglangAPIdetails35B-A3BBase model: Qwen/Qwen3.6-35B-A3Bsource | 61.4 | 55.9 | 81 / 75 | 36 / 27 | 67 / 66 | 1.2 s / 1.3 s | $0.140 | author demo endpoint |
| 46 | decider-35b-a3bdetailsfine-tune · 35B-A3BBase model: Qwen/Qwen3.5-35B-A3B-Basesource | 64.1 | 56.8 | 74 / 76 | 51 / 31 | 67 / 63 | 0.05 s / 0.09 son-GPU, no networkest. 0.24 s / 0.33 s* | $0.154 | GPU pod (H100) |
| 47 | local-jev Qwen3.5-4Bdetailsfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource | 36.7 | 30.8 | 74 / 69 | -23 / -38 | 60 / 62 | 0.14 s / 0.42 son-GPU, no networkest. 0.43 s / 0.99 s* | $0.023 | GPU pod (A6000) |
| 48 | Nemotron Diffusion 8Bv1.5 roster addendum A3details8BBase model: nvidia/Nemotron-Labs-Diffusion-8Bsource | 39.5 | 28.1 | 57 / 59 | 11 / -17 | 51 / 43 | 0.02 s / 0.04 son-GPU, no networkest. 0.19 s / 0.23 s* | $0.030 | GPU pod (H100) |
| 49 | Open-Jev 9Bdetailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource | 61.8 | 65.9 | 66 / 71 | 49 / 59 | 70 / 67 | 0.56 s / 1.6 son-GPU, no networkest. 1.3 s / 3.4 s* | $0.170 | GPU pod (H100) |
| 50 | Decision 2Bdetailsadapter · 2BBase model: openbmb/MiniCPM5-2Bsource | 34.6 | 28.0 | 58 / 68 | -16 / -35 | 62 / 51 | 0.07 s / 0.08 son-GPU, no networkest. 0.30 s / 0.31 s* | $0.013 | GPU pod (RTX6000) |
| 51 | GPT-5.6 LunaAPIdetailsClosed APIBase model: undisclosed | 93.2 | 95.5 | 96 / 94 | 87 / 94 | 96 / 99 | 1.3 s / 3.0 s | tariff$0.205 | hosted API |
| 52 | typecastlmdetails4BBase model: Qwen3.5-4Bsource | 23.7 | 38.8 | 66 / 72 | -52 / -7 | 57 / 52 | 0.04 s / 0.06 son-GPU, no networkest. 0.23 s / 0.28 s* | $0.016 | GPU pod (RTX5090) |
| 53 | JEV Qwen3.5-9B Base NVFP4details9B · NVFP4Base model: ig1/Qwen3.5-9B-NVFP4source | 32.9 | 31.6 | 71 / 60 | -35 / -22 | 62 / 57 | 0.02 s / 0.04 son-GPU, no networkest. 0.19 s / 0.23 s* | $0.056 | GPU pod (RTX5090) |
| 54 | Gemini 3.1 Flash-LiteAPIdetailsClosed APIBase model: undisclosed | 76.6 | 78.6 | 84 / 84 | 74 / 75 | 72 / 77 | 0.86 s / 1.1 s | tariff$0.219 | hosted API |
| 55 | AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1fine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource | 71.9 | 73.6 | 85 / 82 | 50 / 62 | 80 / 77 | 0.10 s / 0.17 son-GPU, no networkest. 0.35 s / 0.49 s* | $0.226 | GPU pod (H100) |
| 56 | AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2fine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource | 72.0 | 73.5 | 85 / 82 | 51 / 62 | 80 / 77 | 0.09 s / 0.23 son-GPU, no networkest. 0.33 s / 0.60 s* | $0.226 | GPU pod (RTX PRO 6000) |
| 57 | NInfer Qwen3.8-27B NVFP4details27BBase model: Qwen3.8-27Bsource | 67.0 | 64.0 | 81 / 80 | 48 / 45 | 72 / 67 | 0.05 s / 0.13 son-GPU, no networkest. 0.25 s / 0.42 s* | $0.231 | GPU pod (RTX5090) |
| 58 | Eikos-27Bv1.5 roster addendum A1adapter · 27BBase model: Qwen/Qwen3.8-27Bsource | 74.1 | 76.0 | 88 / 88 | 56 / 64 | 78 / 76 | 0.10 s / 0.17 son-GPU, no networkest. 0.35 s / 0.49 s* | $0.238 | GPU pod (H100) |
| 59 | NInfer Qwen3.8-27B NVFP4 (T=1.5)details27BBase model: Qwen3.8-27Bsource | 62.4 | 59.7 | 81 / 80 | 37 / 36 | 70 / 63 | 0.05 s / 0.13 son-GPU, no networkest. 0.25 s / 0.42 s* | $0.231 | GPU pod (RTX5090) |
| 60 | InstinctAPIdetailsClosed APIBase model: Qwen3.8-27Bsource · Operator-reported; weights not publicly verifiable. | 61.9 | 63.6 | 84 / 80 | 29 / 40 | 73 / 72 | 0.52 s / 1.3 s | $0.230 | author demo endpoint |
| 61 | OpenJev (thinking, BF16)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource 1source 2 | 82.5 | 85.9 | 82 / 79 | 81 / 89 | 85 / 91 | 0.71 s / 1.2 son-GPU, no networkest. 1.6 s / 2.6 s* | $0.241 | GPU pod (H100) |
| 62 | djev (thinking)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 76.6 | 78.1 | 65 / 48 | 75 / 96 | 90 / 90 | 0.67 s / 1.9 son-GPU, no networkest. 1.5 s / 4.0 s* | $0.249 | GPU pod (H100) |
| 63 | Clefv1.5 roster addendum A6detailsfine-tuneBase model: Qwen/Qwen3.8-27Bsource | 68.1 | 67.7 | 85 / 82 | 47 / 52 | 72 / 69 | 0.36 s / 0.52 son-GPU, no networkest. 0.87 s / 1.2 s* | $0.249 | GPU pod (H100) |
| 64 | LitJevdetails27BBase model: Qwen/Qwen3.8-27Bsource | 58.1 | 58.6 | 80 / 79 | 23 / 34 | 70 / 63 | 1.5 s / 2.4 son-GPU, no networkest. 3.1 s / 4.9 s* | $0.244 | GPU pod (H100) |
| 65 | Bev / Bonsai 27Bv1.5 roster addendum A4details27B · GGUF PQ2_0Base model: Qwen/Qwen3.8-27Bsource | 48.7 | 57.5 | 74 / 76 | 4 / 29 | 68 / 67 | 1.0 s / 1.1 son-GPU, no networkest. 2.2 s / 2.4 s* | $0.247 | GPU pod (H100) |
| 66 | Raw Phi-4 mini direct logitsdetailsoriginalBase model: microsoft/Phi-4-mini-instruct (unmodified)source | 38.3 | 20.0 | 54 / 57 | 6 / -44 | 55 / 47 | 0.07 s / 0.14 son-GPU, no networkest. 0.28 s / 0.43 s* | $0.036 | GPU pod (A6000) |
| 67 | OpenSourceJevdetailsadapter · 4B · GGUF Q4_K_MBase model: Qwen3.5-4Bsource | 27.9 | 23.7 | 64 / 51 | -36 / -37 | 55 / 57 | 0.62 s / 1.9 son-GPU, no networkest. 1.4 s / 4.0 s* | $0.011 | GPU pod (A6000) |
| 68 | reflex-27bdetails27BBase model: Qwen3.8-27Bsource | 63.1 | 62.5 | 85 / 80 | 31 / 36 | 74 / 71 | 1.3 s / 2.1 son-GPU, no networkest. 2.7 s / 4.4 s* | $0.297 | GPU pod (H100) |
| 69 | Open-Jev 2Bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2Bsource | 38.8 | 28.3 | 52 / 45 | 17 / -8 | 48 / 48 | 0.43 s / 1.2 son-GPU, no networkest. 1.0 s / 2.6 s* | $0.170 | GPU pod (H100) |
| 70 | GLiNER2 largedetailsfine-tuneBase model: microsoft/deberta-v3-largesource | 26.0 | 19.0 | 27 / 28 | 16 / -1 | 35 / 29 | 0.86 s / 8.8 son-CPU, no networkest. 1.9 s / 17.7 s* | $0.0056 | CPU container |
| 71 | Qwen3-Reranker-4BdetailsRerankerBase model: Qwen/Qwen3-4B-Basesource | 30.3 | 13.4 | 57 / 46 | -10 / -46 | 44 / 40 | 0.18 s / 1.00 son-GPU, no networkest. 0.51 s / 2.1 s* | $0.052 | GPU pod (A6000) |
| 72 | DeepSeek V4.1 FlashAPIdetailsLLM decoderBase model: undisclosed | 91.8 | 95.5 | 96 / 98 | 86 / 97 | 93 / 92 | 1.8 s / 6.5 s | $0.498 | hosted API |
| 73 | SimpleJevdetails0.8BBase model: Qwen/Qwen3.5-0.8Bsource | 13.3 | 20.2 | 28 / 28 | -13 / 4 | 25 / 28 | 3.6 s / 8.3 son-CPU, no networkest. 7.4 s / 16.8 s* | $0.0098 | CPU container |
| 74 | SimpleJev Qwen3.8-27BAPIdetails27BBase model: Qwen/Qwen3.8-27Bsource | 71.9 | 73.6 | 85 / 81 | 51 / 61 | 79 / 79 | 1.7 s / 1.9 s | $0.687 | author demo endpoint |
| 75 | decision-machine-1APIdetailsClosed APIBase model: undisclosed | 27.5 | 5.1 | 53 / 45 | -18 / -68 | 47 / 38 | 0.18 s / 0.29 s | $0.029 | hosted API |
| 76 | GLiNER2.5 multidetailsfine-tune · 287MBase model: mDeBERTa-v3-basesource | 15.8 | 12.2 | 19 / 17 | 0 / -3 | 28 / 22 | 0.61 s / 8.0 son-CPU, no networkest. 1.4 s / 16.1 s* | $0.0028 | CPU container |
| 77 | Bosun v3.1 0.6Bv1.5 roster addendum A4detailsadapter · 0.6BBase model: Qwen/Qwen3-0.6Bsource | 23.2 | 5.3 | 33 / 37 | -4 / -58 | 40 / 37 | 1.9 s / 8.2 son-CPU, no networkest. 4.0 s / 16.5 s* | $0.0056 | CPU (shared host) |
| 78 | GLiNER2detailsadapterBase model: DeBERTa-v3-basesource | 17.2 | 9.7 | 25 / 21 | 2 / -10 | 25 / 18 | 0.42 s / 4.6 son-CPU, no networkest. 1.00 s / 9.3 s* | $0.0028 | CPU container |
| 79 | Deem 0.8B v1v1.5 roster addendum A4detailsLLM decoderBase model: Qwen/Qwen3.5-0.8Bsource | 24.7 | 3.6 | 28 / 20 | 8 / -29 | 38 / 20 | 0.18 s / 0.29 son-GPU, no networkest. 0.50 s / 0.74 s* | $0.0045 | GPU pod (H100) |
| 80 | JevActAPIdetailsClosed APIBase model: undisclosedsource | 19.7 | 3.5 | 31 / 35 | -11 / -55 | 39 / 30 | 0.80 s / 2.8 s | $0.011 | author demo endpoint |
| 81 | CLM-8Bdetailsfine-tune · 8BBase model: Qwen/Qwen3-8Bsource | 17.1 | 5.8 | 19 / 8 | 7 / -13 | 25 / 23 | 0.02 s / 0.05 son-GPU, no networkest. 0.18 s / 0.26 s* | $0.045 | GPU pod (RTXPRO6000) |
| 82 | kev 0.6Bdetailsadapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 25.5 | -1.5 | 33 / 21 | 2 / -58 | 41 / 32 | 0.14 s / 0.15 son-GPU, no networkest. 0.42 s / 0.45 s* | $0.0046 | GPU pod (A6000) |
| 83 | Raw Qwen3 0.6B direct logitsdetailsoriginal · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 10.0 | 11.1 | 17 / 24 | 13 / -4 | -1 / 14 | 0.06 s / 0.09 son-GPU, no networkest. 0.28 s / 0.33 s* | $0.0056 | GPU pod (A6000) |
| 84 | GLiNER2.5 smalldetailsfine-tune · 74MBase model: microsoft/deberta-v3-xsmallsource | 17.2 | 1.6 | 16 / 12 | 4 / -34 | 31 / 27 | 0.16 s / 1.9 son-CPU, no networkest. 0.47 s / 3.9 s* | $0.0028 | CPU container |
| 85 | Raw Qwen3 1.7B direct logitsdetailsoriginal · 1.7BBase model: Qwen/Qwen3-1.7B-Basesource | 12.3 | 7.1 | 41 / 23 | 12 / -3 | -16 / 1 | 0.07 s / 0.09 son-GPU, no networkest. 0.28 s / 0.34 s* | $0.011 | GPU pod (A6000) |
| 86 | MirrordetailsEncoder / classifierBase model: undisclosed | 6.7 | 4.4 | 2 / -9 | -1 / -5 | 19 / 27 | 2.2 s / 4.6 son-CPU, no networkest. 4.6 s / 9.4 s* | $0.0023 | CPU container |
| 87 | ZeroEntropy zerank-2detailsRerankerBase model: Qwen/Qwen3-4Bsource | 10.0 | -0.4 | 57 / 47 | -68 / -86 | 41 / 37 | 0.16 s / 0.92 son-GPU, no networkest. 0.47 s / 2.0 s* | $0.052 | GPU pod (A6000) |
| 88 | jeffdetailsEncoder / classifierBase model: undisclosedsource | 12.7 | -3.6 | 33 / 22 | -28 / -61 | 33 / 28 | 3.5 s / 18.5 son-CPU, no networkest. 7.1 s / 37.2 s* | $0.0043 | CPU container |
| 89 | smalljev semantic-v9detailsadapter · 2BBase model: openbmb/MiniCPM5-2B-Basesource | 10.9 | -3.5 | 31 / 18 | -40 / -64 | 42 / 35 | 0.15 s / 0.18 son-GPU, no networkest. 0.45 s / 0.50 s* | $0.020 | GPU pod (A6000) |
| 90 | Laya multilingualv1.5 roster addendum A4detailsfine-tune · 322MBase model: mmBERT-basesource | 2.8 | 1.9 | 20 / 9 | -36 / -31 | 24 / 28 | 0.44 s / 2.0 son-CPU, no networkest. 1.0 s / 4.1 s* | $0.0039 | CPU (shared host) |
| 91 | OpenDecisiondetailsEncoder / classifierBase model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0source | 9.5 | -5.3 | 34 / 20 | -47 / -71 | 41 / 35 | 0.07 s / 0.29 son-GPU, no networkest. 0.30 s / 0.74 s* | $0.0050 | GPU pod (H100) |
| 92 | BAAI bge-reranker-v2-m3detailsfine-tuneBase model: BAAI/bge-m3source | -21.1 | -22.8 | 4 / 4 | -100 / -100 | 32 / 27 | 0.03 s / 0.14 son-GPU, no networkest. 0.21 s / 0.42 s* | $0.023 | GPU pod (A6000) |
| 93 | Certo v1detailsEncoder / classifierBase model: ModernBERT-largesource | -24.1 | -24.9 | -2 / -2 | -100 / -100 | 29 / 27 | 0.06 s / 0.06 son-GPU, no networkest. 0.26 s / 0.27 s* | $0.0013 | GPU pod (A6000) |
| 94 | Decision Fastdetailsadapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 7.3 | -11.7 | 38 / 18 | -59 / -87 | 43 / 33 | 0.06 s / 0.06 son-GPU, no networkest. 0.27 s / 0.27 s* | $0.0046 | GPU pod (RTX6000) |
| 95 | Alibaba GTE Reranker ModernBERT-basedetailsfine-tuneBase model: answerdotai/ModernBERT-basesource | -18.6 | -23.2 | 13 / 2 | -100 / -100 | 31 / 28 | 0.03 s / 0.09 son-GPU, no networkest. 0.21 s / 0.34 s* | $0.011 | GPU pod (A6000) |
| 96 | kev 0.5Bdetailsadapter · 0.5BBase model: Qwen/Qwen2.5-0.5Bsource | 3.4 | -22.3 | 25 / 11 | -48 / -95 | 33 / 17 | 0.10 s / 0.12 son-GPU, no networkest. 0.36 s / 0.40 s* | $0.0046 | GPU pod (A6000) |
| 97 | Layadetailsfine-tune · 421MBase model: ModernBERT-largesource | -0.5 | -13.2 | 37 / 13 | -76 / -82 | 37 / 29 | 0.67 s / 1.3 son-CPU, no networkest. 1.5 s / 2.7 s* | $0.0032 | CPU container |
| 98 | lev-350mdetailsadapterBase model: LiquidAI/LFM2.5-350Msource | 5.5 | -18.0 | 33 / 15 | -52 / -100 | 36 / 31 | 0.02 s / 0.04 son-GPU, no networkest. 0.19 s / 0.22 s* | $0.0046 | GPU pod (RTX6000) |
| 99 | Qwen3.5-0.8B Decision Modeldetailsfine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource | 3.4 | -3.7 | 35 / 40 | -70 / -86 | 45 / 34 | 0.58 s / 2.4 son-CPU, no networkest. 1.3 s / 5.0 s* | $0.0048 | CPU container |
| 100 | Mixedbread mxbai-rerank-base-v2detailsRerankerBase model: Qwen2.5 (size not stated)source 1source 2 | -21.0 | -24.0 | 6 / 1 | -100 / -100 | 31 / 27 | 0.04 s / 0.17 son-GPU, no networkest. 0.23 s / 0.49 s* | $0.021 | GPU pod (A6000) |
| 101 | Needle 3detailsoriginal · 121MBase model: undisclosedsource | -11.1 | -21.1 | -4 / -8 | -15 / -24 | -15 / -32 | 67.7 s / 142.6 son-CPU, no networkest. 135.5 s / 285.4 s* | $0.019 | CPU container |
| 102 | Needle 3, options as toolsdetailsoriginal · 121MBase model: undisclosedsource | -5.8 | -19.9 | 11 / 2 | -12 / -30 | -16 / -32 | 29.0 s / 68.3 son-CPU, no networkest. 58.2 s / 136.7 s* | $0.019 | CPU container |
| 103 | open-jev-deberta-v3-largedetailsadapterBase model: microsoft/deberta-v3-largesource | -9.3 | -14.6 | 21 / 17 | -79 / -87 | 30 / 27 | 1.4 s / 2.5 son-CPU, no networkest. 2.9 s / 5.1 s* | $0.0056 | CPU container |
| 104 | Open Jev JSON Canvasdetails26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 79.2 | 75.1 | 81 / 76 | 75 / 74 | 81 / 75 | 0.14 s / 0.24 son-GPU, no networkest. 0.44 s / 0.63 s* | $0.049 | GPU pod (H100) |
| 105 | openJev Verdictdetailsfine-tuneBase model: knowledgator/gliclass-modern-base-v2.0source 1source 2 | 12.6 | -14.1 | 27 / 17 | -15 / -79 | 26 / 20 | 0.13 s / 0.43 son-CPU, no networkest. 0.40 s / 1.0 s* | $0.0028 | CPU container |
| 106 | openJev Verdict 1.4detailsEncoder / classifierBase model: knowledgator/gliclass-modern-base-v2.0source | -14.7 | -18.3 | 24 / 18 | -100 / -100 | 31 / 27 | 0.31 s / 0.49 son-CPU, no networkest. 0.77 s / 1.1 s* | $0.0028 | CPU container |
| 107 | Qwen3.8 27BAPIdetails27BBase model: undisclosed | 92.9 | 98.2 | 97 / 98 | 87 / 99 | 95 / 98 | 6.5 s / 31.9 s | $2.18 | hosted API |
| 108 | verdict-smalldetailsfine-tune · 118MBase model: intfloat/multilingual-e5-smallsource | -0.9 | -2.9 | 26 / 13 | -55 / -44 | 26 / 22 | 0.03 s / 1.4 son-CPU, no networkest. 0.22 s / 2.9 s* | $0.0009 | CPU container |
| 109 | Vondetailsfine-tune · 395MBase model: answerdotai/ModernBERT-largesource | -1.9 | -16.6 | 34 / 18 | -79 / -98 | 38 / 30 | 0.39 s / 1.4 son-CPU, no networkest. 0.92 s / 2.9 s* | $0.0038 | CPU container |
| 110 | Laya typed-decisionsv1.5 roster addendum A4detailsfine-tune · 421MBase model: ModernBERT-largesource | -8.5 | -16.9 | 31 / 18 | -92 / -99 | 35 / 31 | 1.6 s / 6.8 son-CPU, no networkest. 3.3 s / 13.8 s* | $0.0038 | CPU (shared host) |
| – | classifier.devAPIhonorable mentiondetailsClosed APIBase model: undisclosed | 74.5 | 77.1 | 84 / 85 | 57 / 64 | 82 / 82 | 0.52 s / 1.2 s | $0.023 | hosted API |
| – | SimpleJev Qwen3.6-35B-A3BAPIpartial rundetails35B-A3BBase model: undisclosedsource | -17.4 | -23.0 | 16 / 9 | -36 / -43 | -32 / -35 | 1.7 s / 1.8 s | $0.145 | author demo endpoint |
| – | Decision-4Bv1.5 roster addendum A4not rankeddetailsadapterBase model: Qwen/Qwen3.5-4Bsource | — | — | – / – | – / – | – / – | 0.04 s / 0.11 son-GPU, no networkest. 0.23 s / 0.38 s* | $0.014 | GPU pod (H100) |
Costs are estimates (est.) unless marked tariff.
* Self-hosted rows: the first value is the p50 we measured on our own hardware over loopback (on-GPU, or on-CPU where marked, no network). It is not directly comparable with API latency, which we time end to end from our measuring host, network included. “est.” is the est. incl. network (2x + 150 ms) that the ranking and the latency cap use for every self-hosted row. Model authors can ask us to time a hosted endpoint instead (on request).
API = the operator's endpoint received sealed item text during evaluation, without answers. Sealed item text, answers and item-level results stay private; only system-level aggregates appear here. Hover a cost for its price basis and a latency for its raw values and adjustment.
JevBench v1.5.6 · headline option A
Official (A)weighted harmonic mean of four 0–100 axes, Intelligence · Calibration · Speed · Cost = 25 · 25 · 25 · 25, with the low-axis gates · Method ↓
Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).
Whiskers are 95% bootstrap intervals. 68 of 75 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 34 adjacent pairs, so no tie classification is inferred.
Costs are estimates (est.) unless marked tariff.
Jev — reference (TypeSafe, closed) (1)Closed API (weights not public) (10)Open weights · LLM decoder (68)Open weights · diffusion LM (5)Open weights · encoder / classifier (16)Open weights · reranker (5)Base model control (no decision fine-tune, raw logits) (5)
A is the official headline: equal 25/25/25/25 axis weights and an Intelligence floor of 50. B remains the secondary 40/20/20/20 axis-weight view; C retains equal axes with an Intelligence floor of 60. All three use equal Choice/Noul/Score weights. The CI column is the paired-bootstrap 95% interval of the A score.
| #A | System | A · equal (headline) | B · 40/20/20/20 | C · equal, I floor 60 | #B | #C | A 95% CI |
|---|---|---|---|---|---|---|---|
| 1 | Cygnetdetails12BBase model: google/gemma-4-12B-itsource | 73.7 | 73.2 | 73.7 | 2 | 1 | 72.4–74.5 |
| 2 | Winnow-12B Q8detailsmerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsource | 73.2 | 73.5 | 73.2 | 1 | 2 | 72.0–74.0 |
| 3 | Jev 1.13.0APIdetailsJev referenceBase model: undisclosed | 72.1 | 72.1 | 72.1 | 3 | 3 | 71.0–72.6 |
| 4 | JevK5 v0.3v1.5 roster addendum A1adapterBase model: Qwen/Qwen3.5-4Bsource | 71.9 | 68.1 | 63.2 | 6 | 10 | 69.4–72.9 |
| 5 | Vansa-3.4v1.5 roster addendum A7APIdetailsClosed APIBase model: Qwen3.5-4B (self-reported) · Developer-reported to Benchmark Heaven (private correspondence, 1 Oct 2026): built on Qwen3.5-4B via an open decision fine-tune, with Vansa's own adapters; the intermediate fine-tune is not named. Not publicly documented or independently verified. | 71.6 | 68.4 | 67.0 | 5 | 6 | 69.7–72.6 |
| 6 | Plumb-4Bv1.5 roster addendum A1fine-tune · 4BBase model: alibiserikbay/JevK5source | 71.6 | 67.7 | 62.0 | 7 | 11 | 69.2–72.7 |
| 7 | Jev-Omnidetailsfine-tuneBase model: google/gemma-4-12B-itsource | 71.5 | 71.3 | 71.5 | 4 | 4 | 70.2–72.4 |
| 8 | decider-4b v2detailsadapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource | 71.3 | 67.5 | 61.6 | 8 | 12 | 69.1–72.3 |
| 9 | Decision 4B v1.2v1.5 roster addendum A1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 70.8 | 66.6 | 56.7 | 10 | 14 | 68.5–72.0 |
| 10 | Imajev-4Bv1.5 roster addendum A2adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 70.4 | 66.2 | 55.9 | 13 | 15 | 67.8–71.6 |
| 11 | Decision 4B v1.1v1.5 roster addendum A1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 70.4 | 66.1 | 55.1 | 14 | 16 | 66.9–71.6 |
| 12 | levv1.5 roster addendum A6detailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 69.1 | 66.4 | 63.8 | 12 | 9 | 67.2–70.4 |
| 13 | Manchego v2.1v1.5 roster addendum A5detailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 68.8 | 64.4 | 50.1 | 16 | 22 | 59.6–70.3 |
| 14 | SemIfdetails4BBase model: Qwen/Qwen3.5-4Bsource | 68.7 | 64.3 | 50.2 | 17 | 21 | 60.1–70.2 |
| 15 | spark-s1-4b-v6detailsadapter · 4BBase model: Qwen3.5-4Bsource | 68.2 | 66.9 | 68.2 | 9 | 5 | 66.3–69.7 |
| 16 | metask-jev-4bdetails4BBase model: Qwen3.5-4Bsource | 67.5 | 64.1 | 53.6 | 18 | 18 | 65.4–68.7 |
| 17 | Hopperdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 67.5 | 62.9 | 46.9 | 21 | 26 | 56.3–69.2 |
| 18 | Malkuth-4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource | 66.8 | 63.9 | 55.0 | 19 | 17 | 64.9–67.9 |
| 19 | Surogate Rune 26B-A4B v3v1.5 roster addendum A226B-A4BBase model: google/gemma-4-26B-A4B-itsource | 66.5 | 66.6 | 66.5 | 11 | 7 | 65.4–67.0 |
| 20 | reflex 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource | 65.2 | 61.9 | 48.0 | 22 | 23 | 58.6–66.3 |
| 21 | jev-localdetails9BBase model: Qwen/Qwen3.5-9Bsource | 65.2 | 63.2 | 57.4 | 20 | 13 | 63.5–66.5 |
| 22 | djev (Maisa, diffusion-gemma)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 64.2 | 64.8 | 64.2 | 15 | 8 | 63.1–65.0 |
| 23 | Raw Qwen3 4B Instruct 2507 direct logitsdetailsoriginal · 4BBase model: Qwen/Qwen3-4B-Instruct-2507 (unmodified)source | 62.1 | 60.3 | 50.4 | 23 | 20 | 59.4–64.2 |
| 24 | jqvdetails32BBase model: Qwen3-32Bsource | 60.8 | 57.5 | 42.2 | 24 | 29 | 51.3–64.3 |
| 25 | JevK5 v0.2.0detailsadapter · 4BBase model: Qwen3.5-4Bsource | 58.1 | 53.6 | 40.4 | 25 | 32 | 47.6–68.2 |
| 26 | Clef-Flashv1.5 roster addendum A6detailsfine-tuneBase model: Qwen/Qwen3.5-9Bsource | 55.1 | 53.1 | 43.1 | 26 | 28 | 49.6–55.9 |
| 27 | Qwen3.5-9B Jev-like data-mix v2detailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource | 53.0 | 52.5 | 53.0 | 27 | 19 | 51.9–53.9 |
| 28 | Standard One 8Bdetailsadapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source | 47.8 | 47.1 | 47.2 | 30 | 25 | 46.7–48.5 |
| 29 | NInfer Qwen3.8-Flash-Next mixeddetailsLLM decoderBase model: Qwen3.8-Flash-Nextsource | 47.5 | 47.7 | 47.5 | 28 | 24 | 46.7–47.9 |
| 30 | Instinct Dual 4Bv1.5 roster addendum A4APIdetailsClosed APIBase model: Qwen3.5-4Bsource · Operator-reported; weights not publicly verifiable. | 47.0 | 43.0 | 32.6 | 34 | 35 | 37.9–56.8 |
| 31 | swanOnedetailsNVFP4Base model: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source | 46.6 | 47.3 | 46.6 | 29 | 27 | 45.8–46.9 |
| 32 | Raw Qwen3 8B direct logitsdetailsoriginal · 8BBase model: Qwen/Qwen3-8B-Basesource | 45.2 | 44.6 | 32.9 | 31 | 34 | 36.9–46.7 |
| 33 | decider-2bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2B-Basesource | 45.1 | 41.1 | 31.3 | 37 | 37 | 33.8–54.3 |
| 34 | system-onedetails8BBase model: Qwen3-8Bsource | 44.1 | 43.4 | 31.2 | 32 | 38 | 36.8–45.7 |
| 35 | system-one-openAPIdetailsadapter · 2BBase model: Gemma 4 E2Bsource | 42.4 | 38.8 | 29.5 | 38 | 40 | 33.8–51.9 |
| 36 | Autoloops – Gemma 4 31B ITAPIdetailsClosed APIBase model: undisclosed | 40.5 | 41.8 | 40.5 | 35 | 30 | 40.1–40.8 |
| 37 | GPT-6 Luna (low reasoning effort)APIdetailsClosed APIBase model: undisclosed | 40.5 | 43.1 | 40.5 | 33 | 31 | 40.3–40.7 |
| 38 | GPT-6 Luna (default medium reasoning effort)APIdetailsClosed APIBase model: undisclosed | 38.8 | 41.4 | 38.8 | 36 | 33 | 38.6–38.9 |
| 39 | JevOnedetailsadapterBase model: Qwen/Qwen3.6-35B-A3Bsource | 38.2 | 37.4 | 31.1 | 39 | 39 | 37.4–38.8 |
| 40 | kev 4Bdetailsadapter · 4BBase model: Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base. | 38.1 | 34.6 | 26.4 | 40 | 43 | 27.5–46.8 |
| 41 | kev 8Bdetailsadapter · 8BBase model: Qwen/Qwen3-8B-Basesource | 34.2 | 33.1 | 23.7 | 41 | 45 | 26.8–37.5 |
| 42 | open-alternative-jevdetails4BBase model: Qwen3.5-4Bsource | 33.6 | 29.9 | 23.3 | 43 | 46 | 25.8–41.7 |
| 43 | Bespoke Nimble 9Bdetailsadapter · 9BBase model: Qwen3.5-9Bsource | 31.8 | 32.3 | 31.8 | 42 | 36 | 31.2–32.3 |
| 44 | Malkuth-2Bdetailsadapter · 2BBase model: empero-ai/Qwen3.8-2B-Distillsource | 29.9 | 26.4 | 20.8 | 46 | 48 | 21.0–39.0 |
| 45 | openjev-sglangAPIdetails35B-A3BBase model: Qwen/Qwen3.6-35B-A3Bsource | 29.0 | 29.2 | 27.7 | 44 | 41 | 28.4–29.4 |
| 46 | decider-35b-a3bdetailsfine-tune · 35B-A3BBase model: Qwen/Qwen3.5-35B-A3B-Basesource | 27.5 | 27.7 | 27.5 | 45 | 42 | 26.9–27.9 |
| 47 | local-jev Qwen3.5-4Bdetailsfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource | 25.8 | 22.7 | 17.9 | 49 | 57 | 19.9–33.0 |
| 48 | Nemotron Diffusion 8Bv1.5 roster addendum A3details8BBase model: nvidia/Nemotron-Labs-Diffusion-8Bsource | 25.7 | 22.7 | 17.8 | 50 | 58 | 18.6–32.8 |
| 49 | Open-Jev 9Bdetailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource | 24.4 | 25.0 | 24.4 | 47 | 44 | 23.9–24.6 |
| 50 | Decision 2Bdetailsadapter · 2BBase model: openbmb/MiniCPM5-2Bsource | 22.5 | 19.3 | 15.6 | 56 | 61 | 16.7–28.8 |
| 51 | GPT-5.6 LunaAPIdetailsClosed APIBase model: undisclosed | 22.4 | 24.2 | 22.4 | 48 | 47 | 22.3–22.5 |
| 52 | typecastlmdetails4BBase model: Qwen3.5-4Bsource | 21.8 | 18.9 | 15.2 | 60 | 63 | 16.2–28.6 |
| 53 | JEV Qwen3.5-9B Base NVFP4details9B · NVFP4Base model: ig1/Qwen3.5-9B-NVFP4source | 20.1 | 17.8 | 13.9 | 62 | 64 | 15.2–25.5 |
| 54 | Gemini 3.1 Flash-LiteAPIdetailsClosed APIBase model: undisclosed | 19.6 | 20.8 | 19.6 | 51 | 49 | 19.3–19.8 |
| 55 | AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1fine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource | 19.5 | 20.4 | 19.5 | 52 | 50 | 19.3–19.6 |
| 56 | AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2fine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource | 19.5 | 20.4 | 19.5 | 53 | 51 | 19.3–19.6 |
| 57 | NInfer Qwen3.8-27B NVFP4details27BBase model: Qwen3.8-27Bsource | 18.7 | 19.3 | 18.7 | 55 | 52 | 18.5–18.9 |
| 58 | Eikos-27Bv1.5 roster addendum A1adapter · 27BBase model: Qwen/Qwen3.8-27Bsource | 18.5 | 19.5 | 18.5 | 54 | 53 | 18.3–18.6 |
| 59 | NInfer Qwen3.8-27B NVFP4 (T=1.5)details27BBase model: Qwen3.8-27Bsource | 18.5 | 18.9 | 18.5 | 58 | 54 | 18.2–18.6 |
| 60 | InstinctAPIdetailsClosed APIBase model: Qwen3.8-27Bsource · Operator-reported; weights not publicly verifiable. | 18.3 | 18.9 | 18.3 | 59 | 55 | 18.0–18.5 |
| 61 | OpenJev (thinking, BF16)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource 1source 2 | 17.9 | 19.3 | 17.9 | 57 | 56 | 17.8–18.1 |
| 62 | djev (thinking)details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 17.4 | 18.5 | 17.4 | 61 | 59 | 17.3–17.5 |
| 63 | Clefv1.5 roster addendum A6detailsfine-tuneBase model: Qwen/Qwen3.8-27Bsource | 17.0 | 17.7 | 17.0 | 63 | 60 | 16.8–17.1 |
| 64 | LitJevdetails27BBase model: Qwen/Qwen3.8-27Bsource | 16.3 | 16.7 | 15.4 | 64 | 62 | 16.0–16.5 |
| 65 | Bev / Bonsai 27Bv1.5 roster addendum A4details27B · GGUF PQ2_0Base model: Qwen/Qwen3.8-27Bsource | 15.8 | 16.0 | 12.3 | 65 | 66 | 15.1–16.0 |
| 66 | Raw Phi-4 mini direct logitsdetailsoriginalBase model: microsoft/Phi-4-mini-instruct (unmodified)source | 15.2 | 13.1 | 10.6 | 67 | 67 | 10.3–20.7 |
| 67 | OpenSourceJevdetailsadapter · 4B · GGUF Q4_K_MBase model: Qwen3.5-4Bsource | 13.2 | 11.2 | 9.2 | 68 | 68 | 9.0–18.3 |
| 68 | reflex-27bdetails27BBase model: Qwen3.8-27Bsource | 13.2 | 13.8 | 13.2 | 66 | 65 | 13.0–13.3 |
| 69 | Open-Jev 2Bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2Bsource | 9.1 | 8.4 | 6.3 | 69 | 70 | 6.6–11.5 |
| 70 | GLiNER2 largedetailsfine-tuneBase model: microsoft/deberta-v3-largesource | 8.4 | 7.2 | 5.8 | 71 | 71 | 5.2–12.3 |
| 71 | Qwen3-Reranker-4BdetailsRerankerBase model: Qwen/Qwen3-4B-Basesource | 7.1 | 5.9 | 4.9 | 72 | 72 | 4.3–10.7 |
| 72 | DeepSeek V4.1 FlashAPIdetailsLLM decoderBase model: undisclosed | 6.6 | 7.4 | 6.6 | 70 | 69 | 6.6–6.7 |
| 73 | SimpleJevdetails0.8BBase model: Qwen/Qwen3.5-0.8Bsource | 4.0 | 3.3 | 2.8 | 74 | 74 | 2.1–6.7 |
| 74 | SimpleJev Qwen3.8-27BAPIdetails27BBase model: Qwen/Qwen3.8-27Bsource | 3.4 | 3.7 | 3.4 | 73 | 73 | 3.3–3.4 |
| 75 | decision-machine-1APIdetailsClosed APIBase model: undisclosed | 3.2 | 2.5 | 2.3 | 75 | 75 | 1.7–5.4 |
| 76 | GLiNER2.5 multidetailsfine-tune · 287MBase model: mDeBERTa-v3-basesource | 2.7 | 2.1 | 1.9 | 76 | 76 | 1.2–5.1 |
| 77 | Bosun v3.1 0.6Bv1.5 roster addendum A4detailsadapter · 0.6BBase model: Qwen/Qwen3-0.6Bsource | 2.5 | 1.9 | 1.7 | 77 | 77 | 1.2–4.6 |
| 78 | GLiNER2detailsadapterBase model: DeBERTa-v3-basesource | 2.3 | 1.8 | 1.6 | 78 | 78 | 0.9–4.5 |
| 79 | Deem 0.8B v1v1.5 roster addendum A4detailsLLM decoderBase model: Qwen/Qwen3.5-0.8Bsource | 2.1 | 1.7 | 1.5 | 79 | 79 | 0.9–4.3 |
| 80 | JevActAPIdetailsClosed APIBase model: undisclosedsource | 1.5 | 1.1 | 1.1 | 81 | 80 | 0.5–3.2 |
| 81 | CLM-8Bdetailsfine-tune · 8BBase model: Qwen/Qwen3-8Bsource | 1.5 | 1.2 | 1.0 | 80 | 81 | 0.5–3.2 |
| 82 | kev 0.6Bdetailsadapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 1.3 | 0.9 | 0.9 | 82 | 82 | 0.5–2.5 |
| 83 | Raw Qwen3 0.6B direct logitsdetailsoriginal · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 1.1 | 0.9 | 0.7 | 83 | 83 | 0.3–2.6 |
| 84 | GLiNER2.5 smalldetailsfine-tune · 74MBase model: microsoft/deberta-v3-xsmallsource | 0.9 | 0.6 | 0.6 | 85 | 84 | 0.2–2.2 |
| 85 | Raw Qwen3 1.7B direct logitsdetailsoriginal · 1.7BBase model: Qwen/Qwen3-1.7B-Basesource | 0.9 | 0.7 | 0.6 | 84 | 85 | 0.1–2.3 |
| 86 | MirrordetailsEncoder / classifierBase model: undisclosed | 0.2 | 0.2 | 0.2 | 86 | 86 | 0.0–0.9 |
| 87 | ZeroEntropy zerank-2detailsRerankerBase model: Qwen/Qwen3-4Bsource | 0.1 | 0.1 | 0.1 | 87 | 87 | 0.0–0.5 |
| 88 | jeffdetailsEncoder / classifierBase model: undisclosedsource | 0.1 | 0.1 | 0.1 | 88 | 88 | 0.0–0.5 |
| 89 | smalljev semantic-v9detailsadapter · 2BBase model: openbmb/MiniCPM5-2B-Basesource | 0.1 | 0.0 | 0.0 | 89 | 89 | 0.0–0.4 |
| 90 | Laya multilingualv1.5 roster addendum A4detailsfine-tune · 322MBase model: mmBERT-basesource | 0.0 | 0.0 | 0.0 | 90 | 90 | 0.0–0.3 |
| 91 | OpenDecisiondetailsEncoder / classifierBase model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0source | 0.0 | 0.0 | 0.0 | 91 | 91 | 0.0–0.2 |
| 92 | BAAI bge-reranker-v2-m3detailsfine-tuneBase model: BAAI/bge-m3source | 0.0 | 0.0 | 0.0 | 92 | 92 | 0.0–0.0 |
| 93 | Certo v1detailsEncoder / classifierBase model: ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 93 | 93 | 0.0–0.0 |
| 94 | Decision Fastdetailsadapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource | 0.0 | 0.0 | 0.0 | 94 | 94 | 0.0–0.0 |
| 95 | Alibaba GTE Reranker ModernBERT-basedetailsfine-tuneBase model: answerdotai/ModernBERT-basesource | 0.0 | 0.0 | 0.0 | 95 | 95 | 0.0–0.0 |
| 96 | kev 0.5Bdetailsadapter · 0.5BBase model: Qwen/Qwen2.5-0.5Bsource | 0.0 | 0.0 | 0.0 | 96 | 96 | 0.0–0.0 |
| 97 | Layadetailsfine-tune · 421MBase model: ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 97 | 97 | 0.0–0.0 |
| 98 | lev-350mdetailsadapterBase model: LiquidAI/LFM2.5-350Msource | 0.0 | 0.0 | 0.0 | 98 | 98 | 0.0–0.0 |
| 99 | Qwen3.5-0.8B Decision Modeldetailsfine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource | 0.0 | 0.0 | 0.0 | 99 | 99 | 0.0–0.0 |
| 100 | Mixedbread mxbai-rerank-base-v2detailsRerankerBase model: Qwen2.5 (size not stated)source 1source 2 | 0.0 | 0.0 | 0.0 | 100 | 100 | 0.0–0.0 |
| 101 | Needle 3detailsoriginal · 121MBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 101 | 101 | 0.0–0.0 |
| 102 | Needle 3, options as toolsdetailsoriginal · 121MBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 102 | 102 | 0.0–0.0 |
| 103 | open-jev-deberta-v3-largedetailsadapterBase model: microsoft/deberta-v3-largesource | 0.0 | 0.0 | 0.0 | 103 | 103 | 0.0–0.0 |
| 104 | Open Jev JSON Canvasdetails26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource | 0.0 | 0.0 | 0.0 | 104 | 104 | 0.0–0.0 |
| 105 | openJev Verdictdetailsfine-tuneBase model: knowledgator/gliclass-modern-base-v2.0source 1source 2 | 0.0 | 0.0 | 0.0 | 105 | 105 | 0.0–0.0 |
| 106 | openJev Verdict 1.4detailsEncoder / classifierBase model: knowledgator/gliclass-modern-base-v2.0source | 0.0 | 0.0 | 0.0 | 106 | 106 | 0.0–0.0 |
| 107 | Qwen3.8 27BAPIdetails27BBase model: undisclosed | 0.0 | 0.0 | 0.0 | 107 | 107 | 0.0–0.0 |
| 108 | verdict-smalldetailsfine-tune · 118MBase model: intfloat/multilingual-e5-smallsource | 0.0 | 0.0 | 0.0 | 108 | 108 | 0.0–0.0 |
| 109 | Vondetailsfine-tune · 395MBase model: answerdotai/ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 109 | 109 | 0.0–0.0 |
| 110 | Laya typed-decisionsv1.5 roster addendum A4detailsfine-tune · 421MBase model: ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 110 | 110 | 0.0–0.0 |
The chart and table above compare the tested systems, not marketing claims. These are the practical answers readers most often need before choosing a Jev-class decision model.
The highest-ranked open entrants in this run are Cygnet (#1, 73.7), Winnow-12B Q8 (#2, 73.2), JevK5 v0.3 (#4, 71.9), Plumb-4B (#6, 71.6). “Open” here means the tested row publishes code or weights; check the licence and exact configuration in the board before adopting one.
Open entrants with released code or weights can run on infrastructure you choose, including EU infrastructure. That can support data residency, but neither open source nor an EU server makes a deployment GDPR-compliant by itself. Assess your data, contracts, retention, subprocessors and security for the complete setup. See Benchmark Heaven's broader EU-hosting comparison.
jev-router.com offers self-hosted open decision models. Neutrality disclosure: it is run by the authors of this benchmark; it receives no scoring advantage and is not a ranked entrant.
The official score (option A) is the equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost — 25/25/25/25 — with an Intelligence floor of 50 and low-axis gates on Speed and Cost. Version v1.5.6 measures 904 open and 720 sealed decisions per system; Choice, Noul and Score each carry a third, sealed items contribute 50% of Intelligence, and an open-minus-sealed gap beyond the field median costs points. Method notes · options B and C.
Open an issue in the JevBench repository with a reproducible endpoint or runnable code, the exact model and licence, and whether public JevBench items were used during development. New entrants use the same frozen harness and appear in a new version or a disclosed roster addendum. For private data, see the custom evaluation options.
Every price here is US dollars per 1,000 decisions — not per 1,000 tokens. One decision is a whole typed request — state, rubric and options — not a single token.
Systems with a public tariff (per token or per request) are priced at that tariff times the tokens we measured — 3 rows carry a tariff. Systems without one — open weights, author demos, models we ran ourselves — are priced as if a large inference provider hosted them: the list price of the same weights, or the nearest larger sibling or size class when the exact weights are not listed. We do not use per-minute GPU rental or our own CPU time — providers buy capacity in bulk or own the hardware, and price accordingly. Price × tokens per decision = $ per 1,000 decisions, marked “est.”.
Price rules (v1.5). Only public, bookable list prices that have been in effect for at least 30 days count; a manufacturer's standard, non-promotional launch list price counts from day one, and promotions, subsidies, credits and free tiers never do. The scoring price is never below the market reference price of the system's base model. A system without any eligible price is listed as unpriced — no Cost axis and no score until a price qualifies. A later price change triggers a re-score with a visible note on the row.
API models with a known base model (from 1 Oct 2026). They are ranked at their developer's own stated API price; a striped second bar shows the score and rank they would have at the base-model reference price, the way we price self-served open weights of the same base. First applied on Image JevBench v0.1.5 (Wity-1). Self-hosted open weights retain the base-model reference price. Cloudflare rows in v1.5.5 show a separate striped Workers AI price scenario with the self-hosted latency held fixed; Workers AI latency has not been measured, and this scenario does not determine official Capability eligibility. Vansa-3.4 (from v1.5.6) is the first JevBench API row under this rule: it is ranked at Vansa's stated USD 0.034 per million input tokens with no output charge, and its striped bar uses the Qwen3.5-4B reference price. Its base model is developer-reported and not independently verified.
All 21 A1/A2/A3/A4/A5/A6/A7 systems below completed all 1,624 decisions and now have official ranks in A, B, and C; the interactive score presets also include them. Their scores, individual 95% intervals, and the frozen v1.5.0 G_med are unchanged. No new paired-bootstrap comparisons were computed for addendum systems. Existing tie markers are retained only for base-system pairs that remain adjacent; no tie or separation is inferred for the other pairs.
| System | Rank (A) | A score · 95% CI | Rank (B) | B score · 95% CI | Rank (C) | C score · 95% CI |
|---|---|---|---|---|---|---|
| JevK5 v0.3v1.5 roster addendum A1 | #4 | 71.9 69.4–72.9 | #6 | 68.1 64.9–69.8 | #10 | 63.2 51.3–72.1 |
| Vansa-3.4v1.5 roster addendum A7API | #5 | 71.6 69.7–72.6 | #5 | 68.4 66.1–70.1 | #6 | 67.0 57.6–72.6 |
| Plumb-4Bv1.5 roster addendum A1 | #6 | 71.6 69.2–72.7 | #7 | 67.7 64.7–69.6 | #11 | 62.0 50.7–70.9 |
| Decision 4B v1.2v1.5 roster addendum A1 | #9 | 70.8 68.5–72.0 | #10 | 66.6 63.7–68.4 | #14 | 56.7 47.6–65.1 |
| Imajev-4Bv1.5 roster addendum A2 | #10 | 70.4 67.8–71.6 | #13 | 66.2 63.1–68.1 | #15 | 55.9 47.1–64.5 |
| Decision 4B v1.1v1.5 roster addendum A1 | #11 | 70.4 66.9–71.6 | #14 | 66.1 62.1–67.9 | #16 | 55.1 46.4–63.5 |
| levv1.5 roster addendum A6 | #12 | 69.1 67.2–70.4 | #12 | 66.4 64.3–68.2 | #9 | 63.8 55.6–70.4 |
| Manchego v2.1v1.5 roster addendum A5 | #13 | 68.8 59.6–70.3 | #16 | 64.4 55.2–66.5 | #22 | 50.1 41.4–58.1 |
| Surogate Rune 26B-A4B v3v1.5 roster addendum A2 | #19 | 66.5 65.4–67.0 | #11 | 66.6 65.1–67.5 | #7 | 66.5 65.4–67.0 |
| Clef-Flashv1.5 roster addendum A6 | #26 | 55.1 49.6–55.9 | #26 | 53.1 47.1–54.5 | #28 | 43.1 34.5–49.4 |
| Instinct Dual 4Bv1.5 roster addendum A4API | #30 | 47.0 37.9–56.8 | #34 | 43.0 34.2–52.8 | #35 | 32.6 26.3–39.4 |
| Nemotron Diffusion 8Bv1.5 roster addendum A3 | #48 | 25.7 18.6–32.8 | #50 | 22.7 16.1–29.5 | #58 | 17.8 12.9–22.8 |
| AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1 | #55 | 19.5 19.3–19.6 | #52 | 20.4 20.1–20.6 | #50 | 19.5 19.3–19.6 |
| AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2 | #56 | 19.5 19.3–19.6 | #53 | 20.4 20.1–20.6 | #51 | 19.5 19.3–19.6 |
| Eikos-27Bv1.5 roster addendum A1 | #58 | 18.5 18.3–18.6 | #54 | 19.5 19.2–19.7 | #53 | 18.5 18.3–18.6 |
| Clefv1.5 roster addendum A6 | #63 | 17.0 16.8–17.1 | #63 | 17.7 17.4–18.0 | #60 | 17.0 16.8–17.1 |
| Bev / Bonsai 27Bv1.5 roster addendum A4 | #65 | 15.8 15.1–16.0 | #65 | 16.0 15.2–16.4 | #66 | 12.3 10.5–14.3 |
| Bosun v3.1 0.6Bv1.5 roster addendum A4 | #77 | 2.5 1.2–4.6 | #77 | 1.9 0.9–3.6 | #77 | 1.7 0.8–3.2 |
| Deem 0.8B v1v1.5 roster addendum A4 | #79 | 2.1 0.9–4.3 | #79 | 1.7 0.7–3.5 | #79 | 1.5 0.6–3.0 |
| Laya multilingualv1.5 roster addendum A4 | #90 | 0.0 0.0–0.3 | #90 | 0.0 0.0–0.2 | #90 | 0.0 0.0–0.2 |
| Laya typed-decisionsv1.5 roster addendum A4 | #110 | 0.0 0.0–0.0 | #110 | 0.0 0.0–0.0 | #110 | 0.0 0.0–0.0 |
Ranks follow option scores; score intervals are per system, not pairwise rank comparisons. Every slider preset sorts these same ranked systems using its selected weights.
These systems are part of the 116-system v1.5 roster but have no rank. Their numbers are never shown as zero or free.
Incomplete and unmeasured systems receive no official rank. Existing results from earlier benchmark versions remain on their frozen version pages.
The official score is the weighted harmonic mean of the four axes. Option A gives each axis 25%; option B (40/20/20/20) is a secondary view, while option C keeps equal axes and sets the Intelligence gate at 60. In A and B, Intelligence, Speed and Cost each have a quadratic gate below 50. To limit benchmaxxing on the public items, the open-minus-sealed Intelligence gap may be up to 8 points above the field median (G_med) before a penalty applies. Each further point lowers the multiplier on unpenalized Intelligence by one percentage point.
Frozen method METHOD-v1.5, SHA-256 c25d3d8b8512…; pricing addendum v1.5-M2, SHA-256 2fc44459ef80….
The method owner chose equal axis weights and equal weights for Choice, Noul and Score after reviewing the What-If Lab, preserving continuity with v1.4 and treating the three decision types equally. Disclosed headline amendment: equal-axis, equal-type A, SHA-256 752ddccc4e19…. B remains a secondary view.
Re-evaluation policy. Every release re-evaluates the current top 10 on the composite score. Models ranked #11 and below are re-evaluated on a slower cadence — at least monthly, or with every third scheduled refresh release, whichever comes first — and their score is shown as last measured on its release. A material method change re-evaluates every model. Paid fast-lane runs are evaluated within 48 hours of payment, and new submissions are evaluated in the order received.
base = 0.5 × I_open + 0.5 × I_sealed.excess = gap − G_med, penalty = max(0, 1 − max(0, excess − 8) / 100). G_med for this batch is 5.2 CC points.1b660648bd49….Data file SHA-256 2a0adbfa3b78996042cbc8a4fe9a140af68934951f13088396b9c33a36a63583 · scorer output SHA-256 452885de2a84cd5b9ed393d541fd8d6c6a540f9d7f762f9d2df9349383f26173 · run kind official.
Harness, public tasks and every scoring rule: github.com/fstandhartinger/jevbench (MIT). Each project links its author's repository or vendor page.
Authors: if we tested the wrong configuration, tell us and we will rerun it. New entrants become a new version or a disclosed roster addendum rather than silently changing this one.
3D view: three.js r128 (MIT).
Loading capability views…
Loading context-length views…
Previous release: JevBench v1.5.5 (frozen results).
Model author? Submit a model for evaluation (free queue or fast lane) →
Open this section to load the earlier public-only board and diagnostics.