Benchmarks
More
OverviewBenchmaxxingCompareChartsCost vs CapabilityProviders per ModelProvider explorerGatewaysEU & SovereignJev-class modelsImage JevBenchAudioJevBenchSubmit a modelAbout
Official JevBench v1.6.1

JevBench v1.6.1 — Jev alternatives ranking

JevBench by Benchmark Heaven is a benchmark for AI decision models, including Jev-compatible systems: state and a bounded rubric in, a typed answer out. Compare accuracy, probability calibration, latency and cost. Maintained independently of TypeSafe AI.

Release v1.6.1 · 1,500 decisions per self-hosted system and 1,500 per hosted API · 135 ranked systems · 28 systems retain a separately dated v1.5.x score · only system-level aggregates are published · aggregate results JSON · SHA-256 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1

Share this version · View live board · API leaderboard · Previous release: JevBench v1.6.0

Making decisions from images? Explore ImageJevBench v0.3.0 and compare its systems.

JevBench v1.6.1 · headline

JevBench Capability Score

Capability ranking of Jev-class systems

Capability Score averages Intelligence and Calibration. Quyet-1.0-Large leads the Jev-class systems with 81.7.

Jev-class means at most 2× the cost and median latency of Jev (v1.5 reference). How we choose ↘

Adjust cost / latency caps · 2× official
Official
#SystemICCap.$/1k
  1. 1Quyet-1.0-Large73.450.581.7$0.045*
  2. 2decisio v0.8.0 on gemma-4-31B-it70.451.779.6$0.041*
  3. 3Sage 1.3.0API65.658.278.6$0.025*
  4. 4deck-31B73.049.777.6$0.048*
  5. 5Jev 1.13.0API63.654.777.1$0.032
  6. 6René-1 31B FP861.746.076.0$0.063*
  7. 7H2O-Lightning-4B v1.160.060.375.0$0.021*
  8. 8SPX-CD FlashAPI58.452.174.3$0.039*
  9. 9Mercury DecideAPI65.962.173.6$0.018*
  10. 10Bobcat Flash 1.265.849.773.5$0.048*
Show all 97 Jev-class systems (87 more)
  1. 11Surogate Rune 26B-A4B v356.149.673.5$0.048*
  2. 12decider-12b v263.057.872.5$0.026*
  3. 13Xor 26B-A4B58.750.772.4$0.044*
  4. 14decider-12b v1, stock Gemma-4-12B-it60.657.872.2$0.026*
  5. 15torchcast-decision-12b60.556.471.7$0.028*
  6. 16Jev-Omni55.556.171.3$0.029*
  7. 17Winnow-12B Q859.556.671.2$0.028*
  8. 18Cygnet54.856.470.9$0.028*
  9. 19decisio v0.8.0 on gemma-4-12B-it52.359.370.7$0.023*
  10. 20decider chat on Gemma-4-31B-it58.550.870.6$0.044*
  11. 21GEV-26B-Decide49.851.170.1$0.043*
  12. 22decisio v0.9.0 on gemma-4-12B-it52.359.370.0$0.023*
  13. 23Hopper 12B trained51.756.268.1$0.029*
  14. 24SPX-CD-Omni48.858.368.1$0.025*
  15. 25Aplomb 145.857.967.5$0.025*
  16. 26Bespoke Nimble 9B v356.546.166.8$0.063*
  17. 27Plumb-4B43.063.165.4$0.017*
  18. 28decider-4b v240.164.564.9$0.015*
  19. 29JevK5 v0.338.563.164.2$0.017*
  20. 30Clef-Flash42.246.863.8$0.059*
  21. 31deck-4B v1.038.663.563.6$0.017*
  22. 32janus 4B37.664.462.9$0.015*
  23. 33JevK5 v0.2.036.363.162.8$0.017*
  24. 34Hopper34.762.362.4$0.018*
  25. 35Imajev-4B34.663.361.9$0.017*
  26. 36jqv33.951.261.9$0.042*
  27. 37metask-jev-4b39.257.761.8$0.026*
  28. 38Kahn1 4B39.746.461.7$0.061*
  29. 39lev41.161.861.7$0.019*
  30. 40Malkuth-4B33.855.861.1$0.030*
  31. 41Decision 4B v1.232.463.160.4$0.017*
  32. 42Quyet-1.0-Medium41.763.459.2$0.017*
  33. 43Decision-4B37.965.959.1$0.014*
  34. 44Manchego v2.132.564.358.3$0.015*
  35. 45Messier One v0.242.464.858.1$0.015
  36. 46Diffusion Jev54.949.658.1$0.048*
  37. 47jev-local41.958.957.5$0.024*
  38. 48Decision 4B v1.129.163.157.2$0.017*
  39. 49JEV Qwen3.5-9B Base NVFP429.347.657.1$0.056*
  40. 50CoCo-Decision-4B32.765.256.9$0.014*
  41. 51APUS-OpenJev-v1-9B47.248.756.9$0.051*
  42. 52typecastlm29.864.056.5$0.016*
  43. 53TypeCastLM 1.4.035.464.856.4$0.015*
  44. 54SemIf27.163.155.4$0.017*
  45. 55local-jev Qwen3.5-4B24.159.355.0$0.023*
  46. 56Decision 2B22.966.454.8$0.013*
  47. 57spark-s1-4b-v643.260.454.0$0.021*
  48. 58ClassOne Qwen 3.5 9B42.249.350.5$0.049*
  49. 59ZeroEntropy zerank-215.648.449.4$0.052*
  50. 60open-alternative-jev22.363.349.1$0.017*
  51. 61OpenSourceJev23.269.148.6$0.011*
  52. 62EXAONE-4.0-1.2B-JEV v0.314.857.947.7$0.025*
  53. 63Malkuth-2B15.965.647.6$0.014*
  54. 64Nemotron Diffusion 8B20.455.547.5$0.030*
  55. 65Qwen3-Reranker-4B16.348.444.7$0.052*
  56. 66decider-2b20.764.944.6$0.015*
  57. 67BAAI bge-reranker-v2-m30.159.344.1$0.023*
  58. 68kev 4B19.565.844.0$0.014*
  59. 69Mixedbread mxbai-rerank-base-v21.160.343.9$0.021*
  60. 70lev-350m4.880.043.3$0.0046*
  61. 71Certo v10.296.943.2$0.0013*
  62. 72smalljev semantic-v98.760.843.0$0.020*
  63. 73jul fast7.569.042.8$0.011*
  64. 74Alibaba GTE Reranker ModernBERT-base0.069.341.7$0.011*
  65. 75watt-flash-0.13.889.041.6$0.0023
  66. 76Decision Fast6.180.140.6$0.0046*
  67. 77openJev Verdict 1.45.086.640.4$0.0028*
  68. 78Quyet-1.0-Small-EN6.189.139.5$0.0023*
  69. 79OpenDecision7.479.138.0$0.0050*
  70. 80verdict-small2.3100.037.6$0.00087*
  71. 81kev 0.6B7.680.136.9$0.0046*
  72. 82WaterSheep5.368.735.6$0.011*
  73. 83Raw Phi-4 mini direct logits13.753.235.3$0.036*
  74. 84Fastino GLiNER-2.5-DecideAPI7.045.834.6$0.064
  75. 85Raw Qwen3 4B Instruct 2507 direct logits34.063.434.6$0.017*
  76. 86kev 0.5B4.680.132.9$0.0046*
  77. 87Quyet-1.0-Small4.079.632.5$0.0048*
  78. 88Quyet-1.0-Tiny3.788.431.1$0.0024*
  79. 89Open Jev JSON Canvas61.249.330.6$0.049*
  80. 90openJev Verdict4.186.626.6$0.0028*
  81. 91Tacet Sonata2.289.024.3$0.0023*
  82. 92CLM-8B0.150.520.1$0.045*
  83. 93Deem 0.8B v16.580.318.1$0.0045*
  84. 94Laya multilingual0.382.217.0$0.0039*
  85. 95ClassOne Gemma 4 E2B7.870.215.2$0.0098*
  86. 96Raw Qwen3 1.7B direct logits5.168.58.3$0.011*
  87. 97Raw Qwen3 0.6B direct logits5.677.68.2$0.0056*

Wide coloured bar = Capability Score (0–100). Thin bars: cost above, median latency below; shared log ratio scale 0.25× … 64× Jev (v1.5 reference), ticks at 1× and 2× (cap). Shorter is cheaper or faster. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ or hover a row or thin bar for details.

Cost and latency correlate here (Spearman ρ = 0.26, n = 135).

  • Jev — reference (TypeSafe, closed) (1)
  • Closed API (weights not public) (2)
  • Open weights · LLM decoder (98)
  • Open weights · diffusion LM (4)
  • Open weights · encoder / classifier (17)
  • Open weights · reranker (5)
  • Base model control (no decision fine-tune, raw logits) (5)
  • System (router / cascade / ensemble) (3)
  • green: ≤ reference
  • amber: ≤ cap (2× reference)
  • red: > cap
Show general-purpose LLMs and other systems outside the limits

Sorted by Capability Score, not numbered. Each row says which limit it misses, measured against Jev ($0.032 per 1,000 decisions, median 0.62 s).

  1. –Perplexity Decider v1.1 27B75.729.982.8$0.22*Outside: cost 6.7× Jev (v1.5 reference)
  2. –torchcast-decision-27b71.830.381.6$0.21*Outside: cost 6.5× Jev (v1.5 reference)
  3. –wity-1API70.358.879.1$0.024Outside: latency 2.6× Jev (v1.5 reference)API price · eligibility checked at the developer's list price; at base-model pricing it would exceed the cost cap (2.70× Jev)
  4. –Kev 27B64.218.378.9$0.53*Outside: cost 16.4× Jev (v1.5 reference)
  5. –SPX-CD ProAPI64.943.178.6$0.079*Outside: cost 2.4× Jev (v1.5 reference)
  6. –Eikos-27B64.428.777.2$0.24*Outside: cost 7.4× Jev (v1.5 reference)
  7. –OpenJev70.328.575.9$0.24*Outside: cost 7.5× Jev (v1.5 reference), latency 3.0× Jev (v1.5 reference)
  8. –Clef59.128.175.3$0.25*Outside: cost 7.7× Jev (v1.5 reference)
  9. –SimpleJev Qwen3.8-27B57.915.174.9$0.68*Outside: cost 20.9× Jev (v1.5 reference), latency 2.3× Jev (v1.5 reference)
  10. –JEV-27B56.831.073.9$0.20*Outside: cost 6.2× Jev (v1.5 reference)
  11. –swanOne56.242.272.4$0.085*Outside: cost 2.6× Jev (v1.5 reference)
  12. –AutoJev-27B55.629.471.6$0.23*Outside: cost 7.0× Jev (v1.5 reference)
  13. –Decision 2.0 Vega 27B60.429.671.3$0.22*Outside: cost 6.9× Jev (v1.5 reference)
  14. –JPT-35B-A3B51.433.670.7$0.16*Outside: cost 5.0× Jev (v1.5 reference)
  15. –Jebadiah 27B57.629.970.3$0.22*Outside: cost 6.7× Jev (v1.5 reference)
  16. –NInfer Qwen3.8-Flash-Next mixed49.042.569.7$0.082*Outside: cost 2.5× Jev (v1.5 reference)
  17. –JevOne46.039.868.9$0.10*Outside: cost 3.1× Jev (v1.5 reference)
  18. –NInfer Qwen3.8-27B NVFP451.829.168.0$0.23*Outside: cost 7.1× Jev (v1.5 reference)
  19. –Open-Jev 27B v1.152.514.468.0$0.72*Outside: cost 22.2× Jev (v1.5 reference), latency 2.8× Jev (v1.5 reference)
  20. –LitJev43.328.466.6$0.24*Outside: cost 7.6× Jev (v1.5 reference), latency 5.0× Jev (v1.5 reference)
  21. –decider-35b-a3b48.734.466.2$0.15*Outside: cost 4.8× Jev (v1.5 reference)
  22. –Bev / Bonsai 27B43.828.266.1$0.25*Outside: cost 7.6× Jev (v1.5 reference), latency 3.4× Jev (v1.5 reference)
  23. –Open-Jev 9B43.933.161.8$0.17*Outside: cost 5.3× Jev (v1.5 reference), latency 2.2× Jev (v1.5 reference)
  24. –Qwen3.5-9B Jev-like data-mix v241.945.760.9$0.065*Outside: cost 2.001× Jev (v1.5 reference)
  25. –reflex 4B30.963.159.0$0.017*Outside: latency 4.6× Jev (v1.5 reference)
  26. –Standard One 8B32.843.358.4$0.078*Outside: cost 2.4× Jev (v1.5 reference)
  27. –Bespoke Nimble 9B43.136.856.5$0.13*Outside: cost 4.0× Jev (v1.5 reference)
  28. –kev 8B29.340.450.0$0.097*Outside: cost 3.0× Jev (v1.5 reference)
  29. –JADE47.933.649.2$0.16*Outside: cost 5.1× Jev (v1.5 reference)
  30. –Open-Jev 2B21.833.147.7$0.17*Outside: cost 5.3× Jev (v1.5 reference)
  31. –Laya typed-decisions5.582.544.2$0.0038*Outside: latency 3.0× Jev (v1.5 reference)
  32. –Qwen3.5-0.8B Decision Model7.379.542.6$0.0048*Outside: latency 18.3× Jev (v1.5 reference)
  33. –open-jev-deberta-v3-large2.877.635.4$0.0056*Outside: latency 5.7× Jev (v1.5 reference)
  34. –Laya1.884.932.5$0.0032*Outside: latency 2.9× Jev (v1.5 reference)
  35. –SimpleJev13.070.331.6$0.0098*Outside: latency 14.3× Jev (v1.5 reference)
  36. –Raw Qwen3 8B direct logits26.545.530.1$0.065*Outside: cost 2.02× Jev (v1.5 reference)
  37. –system-one29.345.029.7$0.068*Outside: cost 2.1× Jev (v1.5 reference)
  38. –Mirror0.089.325.8$0.0023*Outside: latency 2.4× Jev (v1.5 reference)

Jev-class = cost per decision at most 2× Jev's (≤ $0.065 per 1,000 decisions) and median latency at most 2× Jev's (≤ 1.23 s), the adjusted p50 — the same median the speed chart plots, not the four-axis Speed score. 97 of 135 systems qualify; the other 38, including the general-purpose LLMs, are listed below the divider in the ranking. The charts below show speed and cost beside Capability Score; the official JevBench Score weighs all four axes.

Capability against cost and speed

Jev-class systems are shown by default. Bubble size follows the official JevBench Score; official rank stays unchanged. The five most capable Jev-class systems are labelled.

Capability vs cost

Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.

Capability vs cost: 97 systems. Upper right is best: more capable and cheaper.102030405060708090100$0.0010$0.010$0.10$ per 1,000 decisions (log)Capability↑2× Jev (v1.5 reference)← priciercheaper →1. Quyet-1.0-Large2. decisio v0.8.0 on gemma-4-31B-it3. Sage 1.3.04. deck-31B5. Jev 1.13.0
97 systems. Tap a bubble for its values.

Capability vs speed

Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). Self-hosted systems are plotted at the est. incl. network latency (2x measured + 150 ms); their measured loopback p50 (on-GPU, no network; not directly comparable with API latency) is in the tooltip. The dashed line is 2× Jev’s median latency.

Capability vs speed: 97 systems. Upper right is best: more capable and faster.010203040506070809010070≈3.2 s80≈1.0 s90≈316 ms100≈100 msMedian-latency speedCapability↑2× Jev (v1.5 reference)← slowerfaster →1. Quyet-1.0-Large2. decisio v0.8.0 on gemma-4-31B-it3. Sage 1.3.04. deck-31B5. Jev 1.13.0
97 systems. Tap a bubble for its values.
  • Jev — reference (TypeSafe, closed) (1)
  • Closed API (weights not public) (2)
  • Open weights · LLM decoder (98)
  • Open weights · diffusion LM (4)
  • Open weights · encoder / classifier (17)
  • Open weights · reranker (5)
  • Base model control (no decision fine-tune, raw logits) (5)
  • System (router / cascade / ensemble) (3)
  • faint = outside Jev-class
173 of 173 systems

Model kind

Jev-class

Release

Parameters

No system in this release reports an exact parameter count.

Developer/API price $/1k
Base-model price $/1k
Scored cost $/1k
Alternative price $/1k
p50 latency (s)
p95 latency (s)

JevBench v1.6.1

JevBench Composite Score: 135 ranked systems

Official· four axes 0–100, equal-weight harmonic mean · Method notes ↓

Weights:
Adjust weights ↓

View by:Capability ↑

The official order weighs Intelligence, Calibration, Speed and Cost equally. Each button re-sorts the same systems by one axis.

Greener = stronger within its column.

135 of 135 systems, sorted by official rank, #1 first.

  1. 1Sage 1.3.0APInewClosed APIBase model: undisclosed74.0I 66C 92S 94K 58est.$0.025
  2. 2H2O-Lightning-4B v1.1newfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource72.5I 60C 90S 93K 60est.$0.021
  3. 3Mercury DecideAPInewSystemBase model: undisclosedsource72.4I 66C 81S 85K 62est.$0.018
  4. 4decisio v0.8.0 on gemma-4-31B-itnew31B · FP8 on loadBase model: google/gemma-4-31B-itsource71.7I 70C 89S 91K 52est.$0.041
  5. 5Jev 1.13.0APIJev referenceBase model: undisclosed71.5I 64C 91S 91K 55$0.032
  6. 6Quyet-1.0-Largenewadapter · 31BBase model: google/gemma-4-31B-itsource71.4I 73C 90S 87K 51est.$0.045
  7. 7decider-12b v2newfine-tune · 12BBase model: google/gemma-4-12B-itsource70.9I 63C 82S 90K 58est.$0.026
  8. 8wity-1APInewClosed APIBase model: undisclosedBase-model reference price (base undisclosed at the author's request): 44.0 (would be #33)70.8I 70C 88S 72K 59$0.024
  9. 9decider-12b v1, stock Gemma-4-12B-itnew12BBase model: google/gemma-4-12B-itsource70.4I 61C 84S 90K 58est.$0.026
  10. 10torchcast-decision-12bnewadapter · 12BBase model: google/gemma-4-12B-itsource69.9I 61C 83S 92K 56est.$0.028
  11. 11Winnow-12B Q8merge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsource68.9I 60C 83S 87K 57est.$0.028
  12. 12deck-31Bnew31B · FP8Base model: google/gemma-4-31B-itsource68.7I 73C 82S 87K 50est.$0.048
  13. 13Cygnet12BBase model: google/gemma-4-12B-itsource68.6I 55C 87S 92K 56est.$0.028
  14. 14decisio v0.8.0 on gemma-4-12B-itnew12BBase model: google/gemma-4-12B-itsource68.2I 52C 89S 87K 59est.$0.023
  15. 15decisio v0.9.0 on gemma-4-12B-itnew12BBase model: google/gemma-4-12B-itsource67.8I 52C 88S 86K 59est.$0.023
  16. 16Jev-Omnifine-tuneBase model: google/gemma-4-12B-itsource67.7I 56C 87S 85K 56est.$0.029
  17. 17Xor 26B-A4Bnewmerge · 26B-A4BBase model: google/gemma-4-26B-A4B-itsource67.4I 59C 86S 91K 51est.$0.044
  18. 18Bobcat Flash 1.2newfine-tune · 26B / 4B active · FP8 on loadBase model: google/gemma-4-26B-A4B-itsource67.3I 66C 81S 91K 50est.$0.048
  19. 19SPX-CD FlashAPInewSystemBase model: undisclosedsource66.7I 58C 90S 79K 52est.$0.039
  20. 20decider chat on Gemma-4-31B-itnew31BBase model: google/gemma-4-31B-itsource66.1I 59C 83S 86K 51est.$0.044
Show all 135 systems (115 more ranked)
  1. 21Hopper 12B trainednewfine-tune · 12BBase model: google/gemma-4-12B-itsource65.8I 52C 85S 85K 56est.$0.029
  2. 22Surogate Rune 26B-A4B v3new26B-A4BBase model: google/gemma-4-26B-A4B-itsource*165.0I 56C 91S 87K 50est.$0.048
  3. 23GEV-26B-Decidenewdistilled · 26B-A4BBase model: google/gemma-4-26B-A4B-itsource64.2I 50C 90S 90K 51est.$0.043
  4. 24SPX-CD-Omninewfine-tune · 12BBase model: google/gemma-4-12B-itsource63.2I 49C 87S 89K 58est.$0.025
  5. 25Diffusion Jevnew26B / 4B activeBase model: google/diffusiongemma-26B-A4B-itsource59.3I 55C 61S 87K 50est.$0.048
  6. 26René-1 31B FP8newfine-tune · 31B · FP8Base model: google/gemma-4-31B-itsource55.8I 62C 90S 87K 46est.$0.063
  7. 27Aplomb 1newfine-tune · 5.3BBase model: Qwen/Qwen3.5-4Bsource54.6I 46C 89S 89K 58est.$0.025
  8. 28Bespoke Nimble 9B v3newadapter · 9BBase model: Qwen/Qwen3.5-9Bsource53.5I 57C 77S 89K 46est.$0.063
  9. 29APUS-OpenJev-v1-9Bnewfine-tune · 9BBase model: Qwen/Qwen3.5-9Bsource49.2I 47C 67S 82K 49est.$0.051
  10. 30Plumb-4Bfine-tune · 4BBase model: alibiserikbay/JevK5source48.3I 43C 88S 94K 63est.$0.017
  11. 31SPX-CD ProAPInewSystemBase model: undisclosedsource47.7I 65C 92S 78K 43est.$0.079
  12. 32Messier One v0.2newfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource45.3I 42C 74S 93K 65$0.015
  13. 33spark-s1-4b-v6adapter · 4BBase model: Qwen3.5-4Bsource44.9I 43C 65S 87K 60est.$0.021
  14. 34swanOneNVFP4Base model: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source43.8I 56C 89S 83K 42est.$0.085
  15. 35Quyet-1.0-Mediumnewadapter · 4BBase model: Qwen/Qwen3.5-4Bsource43.6I 42C 77S 89K 63est.$0.017
  16. 36levadapter · 4BBase model: Qwen/Qwen3.5-4Bsource42.2I 41C 82S 88K 62est.$0.019
  17. 37NInfer Qwen3.8-Flash-Next mixedLLM decoderBase model: Qwen3.8-Flash-Nextsource42.0I 49C 90S 89K 43est.$0.082
  18. 38jev-local9BBase model: Qwen/Qwen3.5-9Bsource41.3I 42C 73S 76K 59est.$0.024
  19. 39decider-4b v2adapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource41.2I 40C 90S 92K 65est.$0.015
  20. 40ClassOne Qwen 3.5 9Bnew9BBase model: Qwen/Qwen3.5-9Bsource38.5I 42C 59S 90K 49est.$0.049
  21. 41JevK5 v0.3adapterBase model: Qwen/Qwen3.5-4Bsource37.4I 39C 90S 94K 63est.$0.017
  22. 42metask-jev-4b4BBase model: Qwen3.5-4Bsource37.3I 39C 84S 91K 58est.$0.026
  23. 43deck-4B v1.0newadapter · FP8Base model: alibiserikbay/JevK5 (built on Qwen/Qwen3.5-4B)source 1source 237.3I 39C 89S 90K 63est.$0.017
  24. 44Clef-Flashfine-tuneBase model: Qwen/Qwen3.5-9BsourceWorkers AI price scenario (latency unmeasured): 39.2 (would be #40)36.6I 42C 85S 88K 47est.$0.059
  25. 45Decision-4BadapterBase model: Qwen/Qwen3.5-4Bsource35.4I 38C 80S 91K 66est.$0.014
  26. 46janus 4Bnewfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource35.2I 38C 88S 93K 64est.$0.015
  27. 47Qwen3.5-9B Jev-like data-mix v2adapter · 9BBase model: Qwen/Qwen3.5-9Bsource33.6I 42C 80S 85K 46est.$0.065
  28. 48JevK5 v0.2.0adapter · 4BBase model: Qwen3.5-4Bsource32.1I 36C 89S 92K 63est.$0.017
  29. 49JevOneadapterBase model: Qwen/Qwen3.6-35B-A3Bsource31.1I 46C 92S 90K 40est.$0.101
  30. 50Kahn1 4Bnewfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource31.1I 40C 84S 90K 46est.$0.061
  31. 51TypeCastLM 1.4.0new3.8BBase model: Qwen/Qwen3.5-4Bsource29.8I 35C 77S 93K 65est.$0.015
  32. 52Hopperadapter · 4BBase model: Qwen/Qwen3.5-4Bsource28.9I 35C 90S 91K 62est.$0.018
  33. 53Imajev-4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsource28.7I 35C 89S 91K 63est.$0.017
  34. 54Malkuth-4Badapter · 4BBase model: Qwen/Qwen3.5-4B-Basesource26.2I 34C 88S 90K 56est.$0.030
  35. 55jqv32BBase model: Qwen3-32Bsource25.4I 34C 90S 83K 51est.$0.042
  36. 56JPT-35B-A3Bnewmerge · 35B-A3BBase model: Qwen/Qwen3.5-35B-A3Bsource25.4I 51C 90S 91K 34est.$0.163
  37. 57decider-35b-a3bfine-tune · 35B-A3BBase model: Qwen/Qwen3.5-35B-A3B-Basesource24.8I 49C 84S 91K 34est.$0.154
  38. 58CoCo-Decision-4Bnewfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource24.6I 33C 81S 91K 65est.$0.014
  39. 59Manchego v2.1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource24.5I 33C 84S 91K 64est.$0.015
  40. 60Decision 4B v1.2adapter · 4BBase model: Qwen/Qwen3.5-4Bsource24.5I 32C 88S 94K 63est.$0.017
  41. 61Raw Qwen3 4B Instruct 2507 direct logitsoriginal · 4BBase model: Qwen/Qwen3-4B-Instruct-2507 (unmodified)source21.8I 34C 35S 91K 63est.$0.017
  42. 62JEV-27Bnewdistilled · 27BBase model: Qwen/Qwen3.8-27Bsource21.3I 57C 91S 87K 31est.$0.199
  43. 63torchcast-decision-27bnewmerge · 27BBase model: startlux-models/StartLux-Decision-27Bsource 1source 2*221.3I 72C 91S 88K 30est.$0.210
  44. 64Bespoke Nimble 9Badapter · 9BBase model: Qwen3.5-9Bsource21.1I 43C 70S 86K 37est.$0.128
  45. 65reflex 4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsource20.6I 31C 87S 69K 63est.$0.017
  46. 66Perplexity Decider v1.1 27Bnewfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource20.6I 76C 90S 87K 30est.$0.217
  47. 67JADEnewadapter · 27BBase model: Qwen/Qwen3.8-27Bsource20.2I 48C 51S 86K 34est.$0.164
  48. 68typecastlm4BBase model: Qwen3.5-4Bsource19.7I 30C 83S 93K 64est.$0.016
  49. 69Jebadiah 27Bnewmerge · 27BBase model: Qwen/Qwen3.8-27Bsource19.3I 58C 83S 86K 30est.$0.217
  50. 70Decision 2.0 Vega 27Bnewadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 218.9I 60C 82S 84K 30est.$0.222
  51. 71Decision 4B v1.1adapter · 4BBase model: Qwen/Qwen3.5-4Bsource18.7I 29C 85S 94K 63est.$0.017
  52. 72AutoJev-27Bfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource18.5I 56C 88S 89K 29est.$0.226
  53. 73Eikos-27Badapter · 27BBase model: Qwen/Qwen3.8-27Bsource18.1I 64C 90S 88K 29est.$0.238
  54. 74NInfer Qwen3.8-27B NVFP427BBase model: Qwen3.8-27Bsource17.7I 52C 84S 90K 29est.$0.231
  55. 75OpenJev26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource 1source 217.3I 70C 81S 73K 29est.$0.241
  56. 76Open-Jev 9Badapter · 9BBase model: Qwen/Qwen3.5-9Bsource17.0I 44C 80S 74K 33est.$0.170
  57. 77Standard One 8Badapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source16.9I 33C 84S 93K 43est.$0.078
  58. 78Cleffine-tuneBase model: Qwen/Qwen3.8-27BsourceWorkers AI price scenario (latency unmeasured): 29.4 (would be #52)16.8I 59C 92S 83K 28est.$0.249
  59. 79JEV Qwen3.5-9B Base NVFP49B · NVFP4Base model: ig1/Qwen3.5-9B-NVFP4source16.0I 29C 85S 94K 48est.$0.056
  60. 80SemIf4BBase model: Qwen/Qwen3.5-4Bsource15.5I 27C 84S 91K 63est.$0.017
  61. 81Bev / Bonsai 27B27B · GGUF PQ2_0Base model: Qwen/Qwen3.8-27Bsource11.7I 44C 89S 73K 28est.$0.247
  62. 82LitJev27BBase model: Qwen/Qwen3.8-27Bsource11.5I 43C 90S 68K 28est.$0.244
  63. 83local-jev Qwen3.5-4Bfine-tune · 4BBase model: Qwen/Qwen3.5-4Bsource11.4I 24C 86S 86K 59est.$0.023
  64. 84system-one8BBase model: Qwen3-8Bsource11.0I 29C 30S 91K 45est.$0.068
  65. 85kev 8Badapter · 8BBase model: Qwen/Qwen3-8B-Basesource10.6I 29C 71S 89K 40est.$0.097
  66. 86Decision 2Badapter · 2BBase model: openbmb/MiniCPM5-2Bsource10.3I 23C 87S 90K 66est.$0.013
  67. 87OpenSourceJevadapter · 4B · GGUF Q4_K_MBase model: Qwen3.5-4Bsource10.3I 23C 74S 77K 69est.$0.011
  68. 88open-alternative-jev4BBase model: Qwen3.5-4Bsource9.4I 22C 76S 91K 63est.$0.017
  69. 89Raw Qwen3 8B direct logitsoriginal · 8BBase model: Qwen/Qwen3-8B-Basesource9.3I 27C 34S 89K 46est.$0.065
  70. 90decider-2badapter · 2BBase model: Qwen/Qwen3.5-2B-Basesource7.7I 21C 69S 95K 65est.$0.015
  71. 91Nemotron Diffusion 8B8BBase model: nvidia/Nemotron-Labs-Diffusion-8Bsource7.3I 20C 75S 94K 55est.$0.030
  72. 92kev 4Badapter · 4BBase model: Qwen/Qwen3-4B-Basesource*36.6I 20C 68S 90K 66est.$0.014
  73. 93Kev 27Bnewfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource5.8I 64C 94S 84K 18est.$0.529
  74. 94Malkuth-2Badapter · 2BBase model: empero-ai/Qwen3.8-2B-Distillsource4.0I 16C 79S 92K 66est.$0.014
  75. 95Qwen3-Reranker-4BRerankerBase model: Qwen/Qwen3-4B-Basesource3.7I 16C 73S 82K 48est.$0.052
  76. 96SimpleJev Qwen3.8-27Bnew27BBase model: Qwen/Qwen3.8-27Bsource3.4I 58C 92S 76K 15est.$0.676
  77. 97ZeroEntropy zerank-2RerankerBase model: Qwen/Qwen3-4Bsource3.3I 16C 83S 83K 48est.$0.052
  78. 98EXAONE-4.0-1.2B-JEV v0.3new1.2BBase model: LGAI-EXAONE/EXAONE-4.0-1.2Bsource3.3I 15C 81S 94K 58est.$0.025
  79. 99Open-Jev 2Badapter · 2BBase model: Qwen/Qwen3.5-2Bsource3.2I 22C 74S 77K 33est.$0.170
  80. 100Open-Jev 27B v1.1adapter · 27BBase model: Qwen/Qwen3.8-27Bsource2.9I 52C 83S 73K 14est.$0.716
  81. 101Raw Phi-4 mini direct logitsoriginalBase model: microsoft/Phi-4-mini-instruct (unmodified)source2.5I 14C 57S 92K 53est.$0.036
  82. 102SimpleJev0.8BBase model: Qwen/Qwen3.5-0.8Bsource2.1I 13C 50S 59K 70est.$0.0098
  83. 103smalljev semantic-v9adapter · 2BBase model: openbmb/MiniCPM5-2B-Basesource0.8I 9C 77S 90K 61est.$0.020
  84. 104kev 0.6Badapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource0.5I 8C 66S 92K 80est.$0.0046
  85. 105jul fastnew2BBase model: openbmb/MiniCPM5-2Bsource0.5I 7C 78S 91K 69est.$0.011
  86. 106OpenDecisionEncoder / classifierBase model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0source0.5I 7C 69S 87K 79est.$0.0050
  87. 107ClassOne Gemma 4 E2BnewE2BBase model: google/gemma-4-E2B-itsource0.5I 8C 23S 93K 70est.$0.0098
  88. 108Qwen3.5-0.8B Decision Modelfine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource0.5I 7C 78S 56K 80est.$0.0048
  89. 109Fastino GLiNER-2.5-DecideAPInewfine-tune · 340MBase model: fastino/gliner2-large-v1source 1source 2*40.3I 7C 62S 85K 46$0.064
  90. 110Deem 0.8B v1LLM decoderBase model: Qwen/Qwen3.5-0.8Bsource0.3I 7C 30S 85K 80est.$0.0045
  91. 111Decision Fastadapter · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource0.3I 6C 75S 91K 80est.$0.0046
  92. 112Quyet-1.0-Small-ENnewfine-tune · 153MBase model: answerdotai/ModernBERT-basesource0.3I 6C 73S 95K 89est.$0.0023
  93. 113Laya typed-decisionsfine-tune · 421MBase model: ModernBERT-largesource0.2I 5C 83S 71K 83est.$0.0038
  94. 114WaterSheepnewEncoder / classifierBase model: answerdotai/ModernBERT-basesource0.2I 5C 66S 78K 69est.$0.011
  95. 115openJev Verdict 1.4Encoder / classifierBase model: knowledgator/gliclass-modern-base-v2.0source0.2I 5C 76S 81K 87est.$0.0028
  96. 116Raw Qwen3 0.6B direct logitsoriginal · 0.6BBase model: Qwen/Qwen3-0.6B-Basesource0.2I 6C 11S 94K 78est.$0.0056
  97. 117lev-350madapterBase model: LiquidAI/LFM2.5-350Msource0.2I 5C 82S 94K 80est.$0.0046
  98. 118Raw Qwen3 1.7B direct logitsoriginal · 1.7BBase model: Qwen/Qwen3-1.7B-Basesource0.1I 5C 12S 93K 69est.$0.011
  99. 119kev 0.5Badapter · 0.5BBase model: Qwen/Qwen2.5-0.5Bsource0.1I 5C 61S 92K 80est.$0.0046
  100. 120openJev Verdictfine-tuneBase model: knowledgator/gliclass-modern-base-v2.0source 1source 20.1I 4C 49S 79K 87est.$0.0028
  101. 121Quyet-1.0-Smallnewfine-tune · 328MBase model: aisingapore/SEA-LION-ModernBERT-300Msource0.1I 4C 61S 95K 80est.$0.0048
  102. 122watt-flash-0.1newfine-tune · 140MBase model: jhu-clsp/mmBERT-smallsource0.1I 4C 79S 76K 89$0.0023
  103. 123Quyet-1.0-Tinynewdistilled · 183MBase model: jhu-clsp/mmBERT-smallsource0.1I 4C 58S 95K 88est.$0.0024
  104. 124open-jev-deberta-v3-largeadapterBase model: microsoft/deberta-v3-largesource0.0I 3C 68S 68K 78est.$0.0056
  105. 125verdict-smallfine-tune · 118MBase model: intfloat/multilingual-e5-smallsource0.0I 2C 73S 89K 100est.$0.0009
  106. 126Tacet Sonatanewfine-tune · 144MBase model: jhu-clsp/mmBERT-smallsource0.0I 2C 46S 80K 89est.$0.0023
  107. 127Layafine-tune · 421MBase model: ModernBERT-largesource0.0I 2C 63S 73K 85est.$0.0032
  108. 128Mixedbread mxbai-rerank-base-v2RerankerBase model: Qwen2.5 (size not stated)source 1source 20.0I 1C 87S 93K 60est.$0.021
  109. 129Laya multilingualfine-tune · 322MBase model: mmBERT-basesource0.0I 0C 34S 78K 82est.$0.0039
  110. 130Certo v1Encoder / classifierBase model: ModernBERT-largesource0.0I 0C 86S 93K 97est.$0.0013
  111. 131CLM-8Bfine-tune · 8BBase model: Qwen/Qwen3-8Bsource0.0I 0C 40S 93K 51est.$0.045
  112. 132BAAI bge-reranker-v2-m3fine-tuneBase model: BAAI/bge-m3source0.0I 0C 88S 94K 59est.$0.023
  113. 133Alibaba GTE Reranker ModernBERT-basefine-tuneBase model: answerdotai/ModernBERT-basesource0.0I 0C 83S 94K 69est.$0.011
  114. 134MirrorEncoder / classifierBase model: undisclosed0.0I 0C 52S 73K 89est.$0.0023
  115. 135Open Jev JSON Canvas26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource0.0I 61C 0S 86K 49est.$0.049

Base-model notes

  1. *1 Surogate Rune 26B-A4B v3 (base model: google/gemma-4-26B-A4B-it): This model also powers our System1 Models s1-pro service.
  2. *2 torchcast-decision-27b (base model: startlux-models/StartLux-Decision-27B): Direct base is StartLux-Decision-27B (startlux-models, CC-BY-NC-4.0), itself a modified Alibaba Qwen model; its card does not name the Qwen version. Qwen3.8-27B is inferred from the identical text_config.
  3. *3 kev 4B (base model: Qwen/Qwen3-4B-Base): Evaluated Qwen3 variant; later releases use a different base.
  4. *4 Fastino GLiNER-2.5-Decide (base model: fastino/gliner2-large-v1): Measured via Fastino's hosted API. A 340M English classifier (8k context), not built for multi-step reasoning: near chance on Noul and Score items; 23 very long items exceed its context. Our request follows Fastino's documented format. The open checkpoint scores 11.2 on the community Decision Index (rank 53 of 70; Jev 57.9). Fastino's larger GLiDE is a separate model, ranked separately on the API board.
Weights:
Adjust weights ↓

Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; the gates still apply when an axis sits at 0. Only equal weights give the official JevBench Score and rank.

Score = 4 / (1/I + 1/C + 1/S + 1/K) (each 0–100; × (axis / 50)² for Intelligence, Speed or Cost below 50)

  • Jev — reference (TypeSafe, closed) (1)
  • Closed API (weights not public) (2)
  • Open weights · LLM decoder (98)
  • Open weights · diffusion LM (4)
  • Open weights · encoder / classifier (17)
  • Open weights · reranker (5)
  • Base model control (no decision fine-tune, raw logits) (5)
  • System (router / cascade / ensemble) (3)
  • Striped bar = same system under the labelled alternative price assumption
I, C, S, K = Intelligence, Calibration, Speed, Cost; the est. pill = estimated cost; ann. = announced price; API = the operator's endpoint saw held-out benchmark inputs, without answers; $/1k decisions = US dollars per 1,000 decisions (not heat-shaded). Names link to each project.

Compare two systems

Pick any two. Radars for the score axes, capability by subject topic and use cases, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.

  • A: Jev 1.13.0REFERENCE · TypeSafe — Jev — reference (TypeSafe, closed) · Score 71.5 (#5)
  • B: Sage 1.3.0 — Closed API (weights not public) · Score 74.0 (#1)

The four score axes

Radar: the four score axes, two systemsThe four score axes, Jev 1.13.0 vs Sage 1.3.0. Intelligence: 63.6 vs 65.6; Calibration: 90.6 vs 91.6; Speed: 91.5 vs 93.6; Cost: 54.7 vs 58.2.50100Intelligence: Jev 1.13.0: 63.6; Sage 1.3.0: 65.6Intelligence63.6 · 65.6Calibration: Jev 1.13.0: 90.6; Sage 1.3.0: 91.6Calibration90.6 · 91.6Speed: Jev 1.13.0: 91.5; Sage 1.3.0: 93.6Speed91.5 · 93.6Cost: Jev 1.13.0: 54.7; Sage 1.3.0: 58.2Cost54.7 · 58.2
0–100, the values in the table. An axis a system has no published value for is left as a gap (it counts as 0 in the composite).

Capability by subject topic

Radar: capability by subject topic, two systemsCapability by subject topic, Jev 1.13.0 vs Sage 1.3.0. Rules, policy & law: 70.6 vs 68.1; Coding & software: 74.5 vs 80.6; Math & numbers: 30.5 vs 37.9; Support & operations: 34.9 vs 60.9; Finance & commerce: 60.3 vs 66.3; Everyday language: 51.5 vs 47.4; Safety & security: 35.2 vs 65.3. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Rules, policy & law: Jev 1.13.0: 70.6; Sage 1.3.0: 68.1 — Rules, policy & law: applying written rules: company policies, contracts, regulations, eligibility. 763 items (151 open, 612 sealed).Rules, policy& law70.6 · 68.1Coding & software: Jev 1.13.0: 74.5; Sage 1.3.0: 80.6 — Coding & software: code, SQL, repositories, developer tools and IT systems. 302 items (62 open, 240 sealed).Coding &software74.5 · 80.6Math & numbers: Jev 1.13.0: 30.5; Sage 1.3.0: 37.9 — Math & numbers: a calculation decides the answer: arithmetic, word problems, probability, dates, units. 193 items (37 open, 156 sealed).Math &numbers30.5 · 37.9Support & operations: Jev 1.13.0: 34.9; Sage 1.3.0: 60.9 — Support & operations: support tickets, incidents, logistics, scheduling desks and routing work to a team. 93 items (21 open, 72 sealed).Support &operations34.9 · 60.9Finance & commerce: Jev 1.13.0: 60.3; Sage 1.3.0: 66.3 — Finance & commerce: money: payments, refunds, invoices, orders, expenses, insurance payouts. 66 items (12 open, 54 sealed).Finance &commerce60.3 · 66.3Everyday language: Jev 1.13.0: 51.5; Sage 1.3.0: 47.4 — Everyday language: short everyday messages: intents, assistant requests, reading a detail out of a text. 46 items (11 open, 35 sealed).Everydaylanguage51.5 · 47.4Safety & security: Jev 1.13.0: 35.2; Sage 1.3.0: 65.3 — Safety & security: untrusted or injected instructions, fraud, moderation, access and security triage. 37 items (6 open, 31 sealed).Safety &security35.2 · 65.3

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

What the decision is about — math and dates, coding, rules, money, support work, everyday messages, safety. Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Jev 1.13.0: open+sealed; Sage 1.3.0: open+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.
What each category means · items per category
  • Rules, policy & law — applying written rules: company policies, contracts, regulations, eligibility. 763 items (151 open / 612 sealed)
  • Coding & software — code, SQL, repositories, developer tools and IT systems. 302 items (62 open / 240 sealed)
  • Math & numbers — a calculation decides the answer: arithmetic, word problems, probability, dates, units. 193 items (37 open / 156 sealed)
  • Support & operations — support tickets, incidents, logistics, scheduling desks and routing work to a team. 93 items (21 open / 72 sealed)
  • Finance & commerce — money: payments, refunds, invoices, orders, expenses, insurance payouts. 66 items (12 open / 54 sealed)
  • Everyday language — short everyday messages: intents, assistant requests, reading a detail out of a text. 46 items (11 open / 35 sealed)
  • Safety & security — untrusted or injected instructions, fraud, moderation, access and security triage. 37 items (6 open / 31 sealed)

Use cases (TypeSafe categories)

Radar: use cases (typesafe categories), two systemsUse cases (TypeSafe categories), Jev 1.13.0 vs Sage 1.3.0. Model routing: 78.2 vs 83.2; Legal & compliance: 81.3 vs 70.3; Customer support: 47.6 vs 63.3; E-commerce: 44.6 vs 66.5; Risk assessment: 42.1 vs 33.7; Insurance claims: 76.5 vs 65.1; Financial crime: 36.0 vs 35.0; Moderation: 53.4 vs 70.5. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Model routing: Jev 1.13.0: 78.2; Sage 1.3.0: 83.2 — Model routing: deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 442 items (86 open, 356 sealed).Model routing78.2 · 83.2Legal & compliance: Jev 1.13.0: 81.3; Sage 1.3.0: 70.3 — Legal and compliance: contracts, policies, regulations, eligibility rules, checking documents against written requirements. 387 items (74 open, 313 sealed).Legal &compliance81.3 · 70.3Customer support: Jev 1.13.0: 47.6; Sage 1.3.0: 63.3 — Customer support: classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 149 items (37 open, 112 sealed).Customersupport47.6 · 63.3E-commerce: Jev 1.13.0: 44.6; Sage 1.3.0: 66.5 — E-commerce marketplaces: product listings, product attributes, orders and deliveries, seller catalog policy. 48 items (9 open, 39 sealed).E-commerce44.6 · 66.5Risk assessment: Jev 1.13.0: 42.1; Sage 1.3.0: 33.7 — Risk assessment: estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 45 items (8 open, 37 sealed).Riskassessment42.1 · 33.7Insurance claims: Jev 1.13.0: 76.5; Sage 1.3.0: 65.1 — Insurance claims: first-notice-of-loss reports, claim complexity, missing information, claim triage. 41 items (9 open, 32 sealed).Insuranceclaims76.5 · 65.1Financial crime: Jev 1.13.0: 36.0; Sage 1.3.0: 35.0 — Financial crime: suspicious transactions, KYC, fraud alerts, entity matching for investigations. 39 items (9 open, 30 sealed).Financialcrime36.0 · 35.0Moderation: Jev 1.13.0: 53.4; Sage 1.3.0: 70.5 — Moderation and trust and safety: user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 32 items (7 open, 25 sealed).Moderation53.4 · 70.5

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

The real-world application area of each decision, after the TypeSafe use-case map; an item can count in two. Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Jev 1.13.0: open+sealed; Sage 1.3.0: open+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.

Low sample, n < 30 — indicative only

Category (items)A: Jev 1.13.0B: Sage 1.3.0
Feature extraction for predictive modeling (19)22.5 n=1934.4 n=19
Graphs and knowledge graphs (18)49.8 n=1874.1 n=18
LLM guardrails (17)44.1 n=1765.7 n=17
Search and retrieval (17)80.5 n=1783.3 n=17
Lead generation (17)67.5 n=1775.5 n=17
Semantic code linting (16)12.2 n=1622.9 n=16
Demand forecasting (16)0.0 n=160.0 n=16

No published value for either system (under 15 answered items, or no per-category values): Gaming (14), Scientific discovery (14), Recruiting (13), Advertising (11).

Not drawn on the radar: with so few items a single answer moves a category score by several points, so these values are noise-prone. These values become spokes once the item pool reaches 30 items per category.

What each category means · items per category
  • Model routing — deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 442 items (86 open / 356 sealed)
  • Legal and compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 387 items (74 open / 313 sealed)
  • Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 149 items (37 open / 112 sealed)
  • E-commerce marketplaces — product listings, product attributes, orders and deliveries, seller catalog policy. 48 items (9 open / 39 sealed)
  • Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 45 items (8 open / 37 sealed)
  • Insurance claims — first-notice-of-loss reports, claim complexity, missing information, claim triage. 41 items (9 open / 32 sealed)
  • Financial crime — suspicious transactions, KYC, fraud alerts, entity matching for investigations. 39 items (9 open / 30 sealed)
  • Moderation and trust and safety — user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 32 items (7 open / 25 sealed)

Not drawn: Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 145 items (24 open / 121 sealed) — not a use case of its own, so it is counted but not drawn.

Competence per request type, open / sealed

Radar: competence per request type, open / sealed, two systemsCompetence per request type, open / sealed, Jev 1.13.0 vs Sage 1.3.0. Choice · open: 79.0 vs 85.1; Choice · sealed: 75.8 vs 79.7; Noul · open: 54.5 vs 54.3; Noul · sealed: 46.1 vs 56.4; Score · open: 63.3 vs 57.7; Score · sealed: 63.0 vs 60.5. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Choice · open: Jev 1.13.0: 79.0; Sage 1.3.0: 85.1Choice · open79.0 · 85.1Choice · sealed: Jev 1.13.0: 75.8; Sage 1.3.0: 79.7Choice ·sealed75.8 · 79.7Noul · open: Jev 1.13.0: 54.5; Sage 1.3.0: 54.3Noul · open54.5 · 54.3Noul · sealed: Jev 1.13.0: 46.1; Sage 1.3.0: 56.4Noul · sealed46.1 · 56.4Score · open: Jev 1.13.0: 63.3; Sage 1.3.0: 57.7Score · open63.3 · 57.7Score · sealed: Jev 1.13.0: 63.0; Sage 1.3.0: 60.5Score ·sealed63.0 · 60.5

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Chance-corrected competence for Choice, Noul and Score on the 300 open and 1200 sealed decisions.

Competence per tier — open set

Radar: competence per tier — open set, two systemsCompetence per tier — open set, Jev 1.13.0 vs Sage 1.3.0. Easy: 93.9 vs 89.6; Standard: 63.2 vs 62.3; Judge: 67.7 vs 67.5; Hard: 65.6 vs 70.5. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Easy: Jev 1.13.0: 93.9; Sage 1.3.0: 89.6Easy93.9 · 89.6Standard: Jev 1.13.0: 63.2; Sage 1.3.0: 62.3Standard63.2 · 62.3Judge: Jev 1.13.0: 67.7; Sage 1.3.0: 67.5Judge67.7 · 67.5Hard: Jev 1.13.0: 65.6; Sage 1.3.0: 70.5Hard65.6 · 70.5

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence, the three request types pooled by their published decision counts.

Competence per tier — sealed set

Radar: competence per tier — sealed set, two systemsCompetence per tier — sealed set, Jev 1.13.0 vs Sage 1.3.0. Easy: 78.4 vs 77.5; Standard: 68.7 vs 75.8; Judge: 66.9 vs 73.2; Hard: 58.9 vs 60.8. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Easy: Jev 1.13.0: 78.4; Sage 1.3.0: 77.5Easy78.4 · 77.5Standard: Jev 1.13.0: 68.7; Sage 1.3.0: 75.8Standard68.7 · 75.8Judge: Jev 1.13.0: 66.9; Sage 1.3.0: 73.2Judge66.9 · 73.2Hard: Jev 1.13.0: 58.9; Sage 1.3.0: 60.8Hard58.9 · 60.8

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence on the sealed decisions, types pooled the same way; item text stays private.
How the categories were made

Chance-corrected competence (0 = chance, 100 = perfect) over the category's items the system saw, open and sealed pooled, tiers pooled: per request type the method's competence cell, then the mean over the request types present. Not part of the JevBench Score; never equated for API systems; compare systems within a category, not categories with each other. Values can be negative (below chance).

Family and language are authoring metadata of every item in the frozen v1.6 pool. Each of the 1,500 v1.6 items was also labelled with one subject topic (the v1.2 topic list, unchanged) and one TypeSafe use-case category by Winnow-12B Q8 on our own GPU pod (same model, questions and taxonomy as v1.5); where a uc1 item was written for a use case, that authoring use case is used instead. Sealed items stayed on our own infrastructure. 75 public items (5 %) were checked by hand; item-group rules fix the systematic misses (topic agreement before rules 93 %). All non-English uc1 items are machine-authored and not native-reviewed. Self-hosted systems saw S u P (1,500 items); hosted API systems saw only A u P (600 items), so their cells cover fewer items.

  • Cells with fewer than 15 items are omitted; categories under 30 items are marked low-n.
  • Core v1.6 items carry no use case; the use-case view covers the use-case (uc1) items only.
All values as a table
SpokeA: Jev 1.13.0B: Sage 1.3.0
The four score axes
Intelligence63.665.6
Calibration90.691.6
Speed91.593.6
Cost54.758.2
Capability by subject topic
Rules, policy & law70.668.1
Coding & software74.580.6
Math & numbers30.537.9
Support & operations34.960.9
Finance & commerce60.366.3
Everyday language51.547.4
Safety & security35.265.3
Use cases (TypeSafe categories)
Model routing78.283.2
Legal & compliance81.370.3
Customer support47.663.3
E-commerce44.666.5
Risk assessment42.133.7
Insurance claims76.565.1
Financial crime36.035.0
Moderation53.470.5
Competence per request type, open / sealed
Choice · open79.085.1
Choice · sealed75.879.7
Noul · open54.554.3
Noul · sealed46.156.4
Score · open63.357.7
Score · sealed63.060.5
Competence per tier — open set
Easy93.989.6
Standard63.262.3
Judge67.767.5
Hard65.670.5
Competence per tier — sealed set
Easy78.477.5
Standard68.775.8
Judge66.973.2
Hard58.960.8

Self-hosted cells and hosted-API cells pool S 1,200 + P 300 (1,500 items); raw and unequated. Cells under 15 items are omitted. Sealed counts refer to self-hosted S (1,200); API rows use their original measured sealed basis.

Languages

Raw chance-corrected competence per item language (0 = chance, 100 = perfect; can be negative), from each system's own measured items: Self-hosted rows use S 1200 + P 300; API rows use their measured sealed subset plus P. Sage (A3) language cells and A2/A3 topic/use-case cells cover public P300 only. Raw and outside the Composite. Cells under 15 completed supported responses are left empty; recorded input refusals count toward coverage. A dagger (†) marks fewer than 30 completed responses. Competence retains all supported scored observations, including failures and refusals. Per-language coverage grows with the expanded uc1.1 multilingual pool, a candidate for a later release that is not part of v1.6.1. English (1,160 items) is listed first; the other 21 languages and the mixed-language group share 340 items. In 14 further groups no system reaches the 15-item reporting minimum, so they get no column (items in the pool shown): Mixed-language (14), Hindi (13), Chinese (13), Turkish (12), Arabic (12), Ukrainian (12), Korean (11), Dutch (11), Czech (11), Swedish (10), Norwegian (9), Greek (9), Finnish (8), Indonesian (5) — 150 items, scored like every other item.

135 systems

JevBench v1.6.1 competence by item language and system
11603131292722171716
Sage 1.3.0API · S+P68478477†80†35†61†54†69†
H2O-Lightning-4B v1.1self-hosted · S+P61658472†65†35†44†48†74†
Mercury DecideAPI · S+P66607284†81†50†81†26†77†
decisio v0.8.0 on gemma-4-31B-itself-hosted · S+P73668286†92†64†70†83†92†
Jev 1.13.0API · S+P67288076†72†48†48†33†76†
Quyet-1.0-Largeself-hosted · S+P74638784†94†64†66†68†74†
decider-12b v2self-hosted · S+P64778372†66†51†48†23†74†
wity-1API · S+P75557879†85†44†67†30†45†
decider-12b v1, stock Gemma-4-12B-itself-hosted · S+P61598071†76†46†42†52†81†
torchcast-decision-12bself-hosted · S+P62567573†85†43†56†36†79†
Winnow-12B Q8self-hosted · S+P61567876†79†35†45†29†79†
deck-31Bself-hosted · S+P76728389†97†67†72†82†94†
Cygnetself-hosted · S+P52478275†77†34†35†46†77†
decisio v0.8.0 on gemma-4-12B-itself-hosted · S+P52478373†81†44†33†47†76†
decisio v0.9.0 on gemma-4-12B-itself-hosted · S+P52478273†77†44†33†48†76†
Jev-Omniself-hosted · S+P52747877†56†40†40†47†54†
Xor 26B-A4Bself-hosted · S+P60608257†88†35†46†59†91†
Bobcat Flash 1.2self-hosted · S+P70577766†72†35†60†34†70†
SPX-CD FlashAPI · S+P61537852†68†28†32†45†73†
decider chat on Gemma-4-31B-itself-hosted · S+P61467879†75†56†59†48†73†
Hopper 12B trainedself-hosted · S+P51558970†82†45†48†23†62†
Surogate Rune 26B-A4B v3self-hosted · S+P56528074†77†34†37†33†64†
GEV-26B-Decideself-hosted · S+P50467757†80†43†50†14†38†
SPX-CD-Omniself-hosted · S+P49607147†52†32†12†45†55†
Diffusion Jevself-hosted · S+P55767873†95†37†50†52†84†
René-1 31B FP8self-hosted · S+P65538657†77†38†41†31†46†
Aplomb 1self-hosted · S+P44428450†68†13†52†26†72†
Bespoke Nimble 9B v3self-hosted · S+P58557776†68†26†41†27†78†
APUS-OpenJev-v1-9Bself-hosted · S+P45417855†67†35†36†15†70†
Plumb-4Bself-hosted · S+P42437861†73†35†22†5†46†
SPX-CD ProAPI · S+P68437274†78†59†55†61†71†
Messier One v0.2self-hosted · S+P43437053†68†39†38†63†43†
spark-s1-4b-v6self-hosted · S+P42297970†71†18†24†33†70†
swanOneself-hosted · S+P57488668†85†26†45†42†91†
Quyet-1.0-Mediumself-hosted · S+P42217750†72†39†25†18†31†
levself-hosted · S+P40326657†45†4†31†3†36†
NInfer Qwen3.8-Flash-Next mixedself-hosted · S+P49467876†68†27†43†27†85†
jev-localself-hosted · S+P43247134†49†24†49†15†60†
decider-4b v2self-hosted · S+P39197939†61†11†43†25†2†
ClassOne Qwen 3.5 9Bself-hosted · S+P43277770†65†43†0†33†56†
JevK5 v0.3self-hosted · S+P40267942†60†20†13†0†56†
metask-jev-4bself-hosted · S+P39247261†51†22†41†25†40†
deck-4B v1.0self-hosted · S+P40267957†60†18†13†0†57†
Clef-Flashself-hosted · S+P42287366†64†1†48†0†69†
Decision-4Bself-hosted · S+P4296939†63†6†26†20†42†
janus 4Bself-hosted · S+P37257435†64†7†48†23†61†
Qwen3.5-9B Jev-like data-mix v2self-hosted · S+P42276644†57†0†31†0†70†
JevK5 v0.2.0self-hosted · S+P33327749†70†32†18†26†25†
JevOneself-hosted · S+P46357661†72†29†42†16†36†
Kahn1 4Bself-hosted · S+P37476656†69†17†35†45†70†
TypeCastLM 1.4.0self-hosted · S+P37195547†56†25†40†24†54†
Hopperself-hosted · S+P33237249†55†5†24†0†34†
Imajev-4Bself-hosted · S+P35207041†58†1†40†7†12†
Malkuth-4Bself-hosted · S+P35196141†47†18†46†11†31†
jqvself-hosted · S+P37455846†61†0†40†23†35†
JPT-35B-A3Bself-hosted · S+P53368551†66†34†39†42†67†
decider-35b-a3bself-hosted · S+P49387347†64†18†22†0†47†
CoCo-Decision-4Bself-hosted · S+P34326436†54†7†39†13†41†
Manchego v2.1self-hosted · S+P32155443†66†15†49†5†9†
Decision 4B v1.2self-hosted · S+P32167336†67†5†41†0†31†
Raw Qwen3 4B Instruct 2507 direct logitsself-hosted · S+P32375830†55†11†24†9†38†
JEV-27Bself-hosted · S+P58347767†75†39†46†36†78†
torchcast-decision-27bself-hosted · S+P74737567†73†45†67†44†69†
Bespoke Nimble 9Bself-hosted · S+P42527448†53†3†46†9†55†
reflex 4Bself-hosted · S+P31107657†51†1†22†20†17†
Perplexity Decider v1.1 27Bself-hosted · S+P77728387†81†51†65†52†76†
JADEself-hosted · S+P0050†7†0†0†0†8†
typecastlmself-hosted · S+P27235528†64†24†27†23†28†
Jebadiah 27Bself-hosted · S+P60487273†79†25†61†30†64†
Decision 2.0 Vega 27Bself-hosted · S+P61446165†68†35†51†23†52†
Decision 4B v1.1self-hosted · S+P29126846†55†0†33†1†44†
AutoJev-27Bself-hosted · S+P56457454†83†40†50†30†75†
Eikos-27Bself-hosted · S+P66477369†78†45†57†48†65†
NInfer Qwen3.8-27B NVFP4self-hosted · S+P51518485†75†28†52†46†92†
OpenJevself-hosted · S+P76327689†93†29†82†31†66†
Open-Jev 9Bself-hosted · S+P44346756†78†29†59†0†21†
Standard One 8Bself-hosted · S+P35457540†53†12†46†17†50†
Clefself-hosted · S+P62367780†84†34†70†46†76†
JEV Qwen3.5-9B Base NVFP4self-hosted · S+P27116639†62†3†14†0†31†
SemIf, formerly OpenJevself-hosted · S+P2754939†57†15†31†18†35†
Bev / Bonsai 27Bself-hosted · S+P44305561†72†24†19†43†20†
LitJevself-hosted · S+P45437569†78†43†35†33†52†
local-jev Qwen3.5-4Bself-hosted · S+P1204227†57†0†16†0†34†
system-oneself-hosted · S+P30144438†48†25†9†8†27†
kev 8Bself-hosted · S+P34276946†38†0†9†0†23†
Decision 2Bself-hosted · S+P1394225†49†4†0†0†0†
OpenSourceJevself-hosted · S+P19236032†54†14†30†0†3†
open-alternative-jevself-hosted · S+P21236331†47†13†36†17†14†
Raw Qwen3 8B direct logitsself-hosted · S+P27195325†49†0†19†20†36†
decider-2bself-hosted · S+P23264419†53†10†41†0†4†
Nemotron Diffusion 8Bself-hosted · S+P14354517†56†0†38†9†44†
kev 4Bself-hosted · S+P23126327†37†0†24†0†4†
Kev 27Bself-hosted · S+P66578569†80†37†61†42†77†
Malkuth-2Bself-hosted · S+P13224019†41†0†8†0†0†
Qwen3-Reranker-4Bself-hosted · S+P00120†3†0†0†0†0†
SimpleJev Qwen3.8-27Bself-hosted · S+P58488482†75†50†49†31†77†
ZeroEntropy zerank-2self-hosted · S+P00250†5†0†0†0†0†
EXAONE-4.0-1.2B-JEV v0.3self-hosted · S+P60250†19†11†0†0†0†
Open-Jev 2Bself-hosted · S+P18224919†65†6†15†0†0†
Open-Jev 27B v1.1self-hosted · S+P53336964†73†25†34†11†46†
Raw Phi-4 mini direct logitsself-hosted · S+P00181†4†0†23†0†14†
SimpleJevself-hosted · S+P9143617†51†6†5†0†0†
smalljev semantic-v9self-hosted · S+P0330†1†0†11†0†0†
kev 0.6Bself-hosted · S+P0800†13†0†0†0†6†
jul fastself-hosted · S+P0105†22†0†0†0†0†
OpenDecisionself-hosted · S+P0060†21†10†0†0†6†
ClassOne Gemma 4 E2Bself-hosted · S+P015017†21†0†0†0†2†
Qwen3.5-0.8B Decision Modelself-hosted · S+P0100†16†0†0†0†0†
Fastino GLiNER-2.5-DecideAPI · S+P00200†7†0†0†0†0†
Deem 0.8B v1self-hosted · S+P01200†37†0†0†1†0†
Decision Fastself-hosted · S+P0000†0†0†0†0†0†
Quyet-1.0-Small-ENself-hosted · S+P0000†0†0†7†0†0†
Laya typed-decisionsself-hosted · S+P0000†0†0†0†0†0†
WaterSheepself-hosted · S+P0000†6†0†0†0†0†
openJev Verdict 1.4self-hosted · S+P0000†0†0†0†0†0†
Raw Qwen3 0.6B direct logitsself-hosted · S+P0000†0†9†0†0†23†
lev-350mself-hosted · S+P0000†0†0†0†0†0†
Raw Qwen3 1.7B direct logitsself-hosted · S+P00026†0†0†0†7†16†
kev 0.5Bself-hosted · S+P0000†0†0†0†0†0†
openJev Verdictself-hosted · S+P0000†0†0†0†2†0†
Quyet-1.0-Smallself-hosted · S+P0000†5†0†0†2†0†
watt-flash-0.1self-hosted · S+P0000†0†0†0†0†0†
Quyet-1.0-Tinyself-hosted · S+P0000†7†0†0†9†0†
open-jev-deberta-v3-largeself-hosted · S+P0000†0†0†0†0†0†
verdict-smallself-hosted · S+P0000†0†0†2†0†0†
Tacet Sonataself-hosted · S+P0000†0†0†0†0†0†
Layaself-hosted · S+P0000†0†0†0†0†0†
Mixedbread mxbai-rerank-base-v2self-hosted · S+P0000†0†0†0†0†0†
Laya multilingualself-hosted · S+P0000†0†0†0†0†0†
Certo v1self-hosted · S+P0000†0†0†0†0†0†
CLM-8Bself-hosted · S+P0000†0†0†0†3†0†
BAAI bge-reranker-v2-m3self-hosted · S+P0000†0†0†0†0†0†
Alibaba GTE Reranker ModernBERT-baseself-hosted · S+P0000†0†0†0†0†0†
Mirrorself-hosted · S+P0000†0†0†0†0†0†
Open Jev JSON Canvasself-hosted · S+P62647274†78†25†45†62†94†

Intelligence gate and Noul decisiveness

The Composite applies a soft gate below Intelligence 50 (× (I/50)²). Rows under it show their value with a "below gate" label. The decisive rate is the share of valid Noul answers with P(yes) ≥ 0.80 or ≤ 0.20 (for systems that return only a label, every valid label counts as decisive); only decisive answers count as answers. Accuracy among decisive answers is a diagnostic and does not enter any score.

SystemIntelligenceChoiceNoulScoreNoul decisive rateAccuracy among decisive
Sage 1.3.065.6nativenativenative83 %95 %
H2O-Lightning-4B v1.160.0nativenativenative100 %85 %
Mercury Decide65.9nativenativenative90 %90 %
decisio v0.8.0 on gemma-4-31B-it70.4nativenativenative94 %96 %
Jev 1.13.063.6nativenativenative79 %98 %
Quyet-1.0-Large73.4nativenativenative91 %93 %
decider-12b v263.0nativenativenative99 %83 %
wity-170.3nativenativenative91 %94 %
decider-12b v1, stock Gemma-4-12B-it60.6nativenativenative96 %87 %
torchcast-decision-12b60.5nativenativenative96 %86 %
Winnow-12B Q859.5nativenativenative86 %92 %
deck-31B73.0nativenativenative96 %94 %
Cygnet54.8nativenativenative81 %92 %
decisio v0.8.0 on gemma-4-12B-it52.3nativenativenative82 %91 %
decisio v0.9.0 on gemma-4-12B-it52.3nativenativenative81 %91 %
Jev-Omni55.5nativenativenative74 %94 %
Xor 26B-A4B58.7nativenativenative80 %94 %
Bobcat Flash 1.265.8nativenativenative89 %92 %
SPX-CD Flash58.4nativenativenative81 %94 %
decider chat on Gemma-4-31B-it58.5nativenativenative80 %98 %
Hopper 12B trained51.7nativenativenative79 %93 %
Surogate Rune 26B-A4B v356.1nativenativenative73 %97 %
GEV-26B-Decide49.8below gatenativenativenative70 %96 %
SPX-CD-Omni48.8below gatenativenativenative68 %93 %
Diffusion Jev54.9nativenativenative93 %85 %
René-1 31B FP861.7nativenativenative78 %98 %
Aplomb 145.8below gatenativenativenative68 %93 %
Bespoke Nimble 9B v356.5nativenativenative87 %89 %
APUS-OpenJev-v1-9B47.2below gatenativenativenative88 %85 %
Plumb-4B43.0below gatenativenativenative71 %91 %
SPX-CD Pro64.9nativenativenative84 %95 %
Messier One v0.242.4below gatenativenativenative85 %85 %
spark-s1-4b-v643.2below gatenativenativenative84 %83 %
swanOne56.2nativenativenative81 %93 %
Quyet-1.0-Medium41.7below gatenativenativenative76 %84 %
lev41.1below gatenativenativenative84 %83 %
NInfer Qwen3.8-Flash-Next mixed49.0below gatenativenativenative75 %93 %
jev-local41.9below gatenativenativenative85 %82 %
decider-4b v240.1below gatenativenativenative64 %92 %
ClassOne Qwen 3.5 9B42.2below gatenativenativenative97 %79 %
JevK5 v0.338.5below gatenativenativenative66 %95 %
metask-jev-4b39.2below gatenativenativenative75 %88 %
deck-4B v1.038.6below gatenativenativenative68 %93 %
Clef-Flash42.2below gatenativenativenative68 %93 %
Decision-4B37.9below gatenativenativenative72 %88 %
janus 4B37.6below gatenativenativenative63 %94 %
Qwen3.5-9B Jev-like data-mix v241.9below gatenativenativenative71 %91 %
JevK5 v0.2.036.3below gatenativenativenative62 %93 %
JevOne46.0below gatenativenativenative62 %96 %
Kahn1 4B39.7below gatenativenativenative70 %87 %
TypeCastLM 1.4.035.4below gatenativenativenative77 %81 %
Hopper34.7below gatenativenativenative56 %94 %
Imajev-4B34.6below gatenativenativenative52 %96 %
Malkuth-4B33.8below gatenativenativenative65 %92 %
jqv33.9below gatenativenativenative66 %94 %
JPT-35B-A3B51.4nativenativenative74 %94 %
decider-35b-a3b48.7below gatenativenativenative74 %91 %
CoCo-Decision-4B32.7below gatenativenativenative56 %96 %
Manchego v2.132.5below gatenativenativenative62 %90 %
Decision 4B v1.232.4below gatenativenativenative59 %94 %
Raw Qwen3 4B Instruct 2507 direct logits34.0below gatenativenativenative92 %73 %
JEV-27B56.8nativenativenative79 %95 %
torchcast-decision-27b71.8nativenativenative83 %97 %
Bespoke Nimble 9B43.1below gatenativenativenative77 %87 %
reflex 4B30.9below gatenativenativenative58 %92 %
Perplexity Decider v1.1 27B75.7nativenativenative91 %95 %
JADE47.9below gatenativenativenative94 %91 %
typecastlm29.8below gatenativenativenative57 %90 %
Jebadiah 27B57.6nativenativenative74 %98 %
Decision 2.0 Vega 27B60.4nativenativenative73 %99 %
Decision 4B v1.129.1below gatenativenativenative57 %94 %
AutoJev-27B55.6nativenativenative70 %99 %
Eikos-27B64.4nativenativenative81 %98 %
NInfer Qwen3.8-27B NVFP451.8nativenativenative82 %89 %
OpenJev70.3nativenativenative96 %93 %
Open-Jev 9B43.9below gatenativenativenative81 %86 %
Standard One 8B32.8below gatenativenativenative68 %90 %
Clef59.1nativenativenative82 %94 %
JEV Qwen3.5-9B Base NVFP429.3below gatenativenativenative53 %89 %
SemIf, formerly OpenJev27.1below gatenativenativenative62 %90 %
Bev / Bonsai 27B43.8below gatenativenativenative70 %91 %
LitJev43.3below gatenativenativenative66 %98 %
local-jev Qwen3.5-4B24.1below gatenativenativenative35 %99 %
system-one29.3below gatenativenativenative92 %73 %
kev 8B29.3below gatenativenativenative77 %81 %
Decision 2B22.9below gatenativenativenative39 %94 %
OpenSourceJev23.2below gatenativenativenative49 %95 %
open-alternative-jev22.3below gatenativenativenative60 %87 %
Raw Qwen3 8B direct logits26.5below gatenativenativenative92 %71 %
decider-2b20.7below gatenativenativenative68 %78 %
Nemotron Diffusion 8B20.4below gatenativenativenative61 %77 %
kev 4B19.5below gatenativenativenative67 %82 %
Kev 27B64.2nativenativenative80 %97 %
Malkuth-2B15.9below gatenativenativenative53 %84 %
Qwen3-Reranker-4B16.3below gatenativenativenative25 %75 %
SimpleJev Qwen3.8-27B57.9nativenativenative78 %96 %
ZeroEntropy zerank-215.6below gatenativenativenative14 %87 %
EXAONE-4.0-1.2B-JEV v0.314.8below gatenativenativenative40 %85 %
Open-Jev 2B21.8below gatenativenativenative67 %75 %
Open-Jev 27B v1.152.5nativenativenative67 %94 %
Raw Phi-4 mini direct logits13.7below gatenativenativenative50 %61 %
SimpleJev13.0below gatenativenativenative100 %58 %
smalljev semantic-v98.7below gatenativenativenative23 %73 %
kev 0.6B7.6below gatenativenativenative43 %69 %
jul fast7.5below gatenativenativenative30 %89 %
OpenDecision7.4below gatenativenativenative51 %68 %
ClassOne Gemma 4 E2B7.8below gatenativenativenative83 %59 %
Qwen3.5-0.8B Decision Model7.3below gatenativenativenative22 %88 %
Fastino GLiNER-2.5-Decide7.0below gateconfidenceconfidenceconfidence100 %54 %
Deem 0.8B v16.5below gatenativenativenative77 %59 %
Decision Fast6.1below gatenativenativenative14 %90 %
Quyet-1.0-Small-EN6.1below gatenativenativenative28 %64 %
Laya typed-decisions5.5below gatenativenativenative1 %100 %
WaterSheep5.3below gatenativenativenative37 %69 %
openJev Verdict 1.45.0below gatenativenativenative0 %—
Raw Qwen3 0.6B direct logits5.6below gatenativenativenative100 %42 %
lev-350m4.8below gatenativenativenative8 %69 %
Raw Qwen3 1.7B direct logits5.1below gatenativenativenative100 %42 %
kev 0.5B4.6below gatenativenativenative26 %56 %
openJev Verdict4.1below gatenativenativenative33 %60 %
Quyet-1.0-Small4.0below gatenativenativenative29 %71 %
watt-flash-0.13.8below gatenativenativenative1 %60 %
Quyet-1.0-Tiny3.7below gatenativenativenative20 %64 %
open-jev-deberta-v3-large2.8below gatenativenativenative15 %58 %
verdict-small2.3below gatenativenativenative45 %62 %
Tacet Sonata2.2below gatenativenativenative33 %56 %
Laya1.8below gatenativenativenative19 %61 %
Mixedbread mxbai-rerank-base-v21.1below gatenativenativenative0 %—
Laya multilingual0.3below gatenativenativenative72 %50 %
Certo v10.2below gatenativenativenative1 %25 %
CLM-8B0.1below gatenativenativenative51 %69 %
BAAI bge-reranker-v2-m30.1below gatenativenativenative0 %—
Alibaba GTE Reranker ModernBERT-base0.0below gatenativenativenative0 %—
Mirror0.0below gatenativenativenative92 %60 %
Open Jev JSON Canvas61.2labellabellabel100 %85 %
All data (173 systems)

Not yet measured on v1.6 · 28 systems with a dated carried score

Every ranked system of the live v1.5.7 board that is not measured on the v1.6.0 pool keeps its last published score, marked with the release that first published that measurement and that release's publication day. Carried rows are listed separately and never ranked together with v1.6-measured rows. v1.5 protocol (1,624 decisions: 904 open + 720 sealed); scores are on the v1.5 scale and are not comparable with v1.6-measured rows.

Dates: v1.5.0 published 2026-09-28 (25) · v1.5.3 published 2026-09-29 (2) · v1.5.6 published 2026-10-03 (1). The date is the publication day of the release that first published the measurement, not a per-model measurement timestamp.

Carried JevBench v1.5.x results, not ranked with v1.6 measurements
SystemMeasured onCapability (v1.5 scale)v1.5 CompositeCost / 1,000Median latency
Qwen3.8 27B (Chutes TEE)measured on v1.5.0 (2026-09-28)96.80.0 · was #108 on v1.5.7$2.1784estimate6.49 s
GPT-6 Luna (default medium reasoning effort)measured on v1.5.0 (2026-09-28)95.938.8 · was #38 on v1.5.7$0.1138estimate1.56 s
DeepSeek V4.1 Flash (thinking default)measured on v1.5.0 (2026-09-28)95.36.6 · was #72 on v1.5.7$0.4976estimate1.78 s
GPT-6 Luna (low reasoning effort)measured on v1.5.0 (2026-09-28)95.140.5 · was #37 on v1.5.7$0.1075estimate1.58 s
GPT-5.6 Luna (low reasoning effort)measured on v1.5.0 (2026-09-28)94.522.4 · was #51 on v1.5.7$0.2047tariff1.32 s
Autoloops – Gemma 4 31B ITmeasured on v1.5.0 (2026-09-28)81.340.5 · was #36 on v1.5.7$0.1032tariff0.61 s
SimpleJev Qwen3.8-27Bmeasured on v1.5.0 (2026-09-28)80.03.4 · was #74 on v1.5.7$0.6868estimate1.68 s
AutoJev-27B (RTX PRO 6000)measured on v1.5.0 (2026-09-28)79.719.5 · was #56 on v1.5.7$0.2262estimate0.33 s
Surogate Rune 26B-A4B v3 (RTX PRO 6000)Not yet measured on the v1.6 pool.measured on v1.5.0 (2026-09-28)79.066.5 · was #19 on v1.5.7$0.0502estimate0.35 s
Gemini 3.1 Flash-Litemeasured on v1.5.0 (2026-09-28)76.119.6 · was #54 on v1.5.7$0.2194tariff0.86 s
reflex-27b (Qwen3.8-27B)measured on v1.5.0 (2026-09-28)74.413.2 · was #68 on v1.5.7$0.2973estimate2.75 s
Instinct (ZooWork, Qwen3.8-27B)measured on v1.5.0 (2026-09-28)73.918.3 · was #60 on v1.5.7$0.2298estimate0.52 s
NInfer Qwen3.8-27B NVFP4 (T=1.5)measured on v1.5.0 (2026-09-28)73.818.5 · was #59 on v1.5.7$0.2305estimate0.25 s
Vansa-3.4 (Vansa, hosted System One API)Hosted API retained at its v1.5.6 score under the three-refresh exposure cadence.measured on v1.5.6 (2026-10-03)72.871.6 · was #5 on v1.5.7$0.0193estimate0.14 s
openjev-sglang (Qwen3.6-35B-A3B on SGLang)measured on v1.5.0 (2026-09-28)70.829.0 · was #45 on v1.5.7$0.1398estimate1.20 s
Instinct Dual 4Bmeasured on v1.5.3 (2026-09-29)65.747.0 · was #30 on v1.5.7$0.0212estimate0.51 s
system-one-open (Gemma 4 E2B LoRA on an L4)measured on v1.5.0 (2026-09-28)56.642.4 · was #35 on v1.5.7$0.0114estimate1.15 s
decision-machine-1 (milliseconds.ai)measured on v1.5.0 (2026-09-28)48.03.2 · was #75 on v1.5.7$0.0286estimate0.18 s
jeff (Logan Markewich, GLiFormer 400M)measured on v1.5.0 (2026-09-28)42.40.1 · was #89 on v1.5.7$0.0043estimate7.14 s
Von (wfzyx, Option-Marker 395M)measured on v1.5.0 (2026-09-28)41.70.0 · was #110 on v1.5.7$0.0038estimate0.92 s
Bosun v3.1 0.6Bmeasured on v1.5.3 (2026-09-29)39.42.5 · was #78 on v1.5.7$0.0056estimate4.04 s
JevAct (einptein, jev1-2b-v2)measured on v1.5.0 (2026-09-28)36.91.5 · was #81 on v1.5.7$0.0112estimate0.80 s
GLiNER2.5 multi (Fastino, 287M)measured on v1.5.0 (2026-09-28)36.02.7 · was #77 on v1.5.7$0.0028estimate1.37 s
GLiNER2.5 small (Fastino, 74M)measured on v1.5.0 (2026-09-28)32.50.9 · was #85 on v1.5.7$0.0028estimate0.47 s
GLiNER2 large (Fastino)measured on v1.5.0 (2026-09-28)32.48.4 · was #70 on v1.5.7$0.0056estimate1.88 s
GLiNER2 (Fastino, gliner2.5-base)measured on v1.5.0 (2026-09-28)24.52.3 · was #79 on v1.5.7$0.0028estimate1.00 s
Needle 3 (Cactus, 2-bit, local CPU)measured on v1.5.0 (2026-09-28)0.00.0 · was #102 on v1.5.7$0.0191estimate135.51 s
Needle 3, options as tools (post-hoc adapter mode)measured on v1.5.0 (2026-09-28)0.00.0 · was #103 on v1.5.7$0.0191estimate58.16 s

* Self-hosted rows: the first value is the p50 we measured on our own hardware over loopback (on-GPU, or on-CPU where marked, no network). It is not directly comparable with API latency, which we time end to end from our measuring host, network included. “est.” is the est. incl. network (2x + 150 ms) that the ranking and the latency cap use for every self-hosted row. Model authors can ask us to time a hosted endpoint instead (on request).

Method · v1.6.1

System types (colours)

  • Jev — reference (TypeSafe, closed) — The closed system JevBench is named after, shown as the reference.
  • Closed API (weights not public) — Available through a hosted API; the weights cannot be downloaded.
  • Open weights · LLM decoder — An autoregressive language model with public weights, including fine-tunes, merges and Jev rebuilds.
  • Open weights · diffusion LM — A language model that generates by iterative denoising instead of token by token.
  • Open weights · encoder / classifier — BERT-style encoders, NLI zero-shot classifiers and GLiNER-type models.
  • Open weights · reranker — A cross-encoder or LLM reranker that scores options against the input.
  • Base model control (no decision fine-tune, raw logits) — An official open checkpoint without any decision fine-tune, read out from raw logits, used as a floor.
  • System (router / cascade / ensemble) — Several models combined at inference time.

Colours show architecture only; they never change a score or rank. Each class is assigned from cited evidence (config.json, model card, provider docs, or our own run record for hosted APIs).

Method amendments

  • 5 Oct 2026, v1.6.1 (Florian's decision): hosted/API systems now answer the same full item set as self-hosted systems (1,200 sealed + 300 public). Their earlier answers on the API subsets were reused and only the missing sealed items were sent; every item was logged in the exposure ledger before it was sent. API rows are no longer equated. Consequence: every v1.6.0 sealed item has now been seen by at least two providers, so the whole v1.6.0 sealed draw is retired for future releases; v1.6.2 onward draws fresh items from the reserve.
  • 5 Oct 2026, v1.6.1 cost rule (Florian's decision): cost per 1,000 decisions is now measured on one common item set for every system: all items except the 23 very long items that lie outside the Jev reference's accepted input range. Before, a system that processed those items paid for their tokens while a system that refused them did not. Refusals still count against Intelligence exactly as before. Hosted APIs with a per-token tariff are priced from their measured tokens on this common set; carried per-decision prices of self-hosted systems are unchanged. Effect: Sage 1.3.0 USD 0.0247 per 1,000 decisions on the common set (0.0650 on all 1,500 items); Jev 1.13.0 0.0323 (unchanged).

Rotating item sets

Each release draws fresh sealed decisions from a larger reserve. Self-hosted open-weights models (run offline on our own GPU pods or Sandy) answer S and P; externally hosted models answer the same full set (S and P) as the self-hosted systems since v1.6.1, so hosted APIs are no longer equated.

SetItemsChoiceNoulScoreAnswered by
S · Sealed release draw1,200600300300self-hosted systems (run offline on our own GPU pods or Sandy) and, from v1.6.1, externally hosted APIs
A · API subset (part of S; used by the v1.6.0 hosted-API measurements)3001507575every system (inside S since v1.6.1); retired for future draws
P · Public set3001507575every system
  • Self-hosted systems: 1,500 items (S 1,200 + P 300). Hosted APIs: 1,500 items (S 1,200 + P 300), the same as self-hosted systems; A (300) is the part of S that earlier API measurements used.
  • A sealed item is scored in at most three releases, then retired. An item used in one release is not drawn again in the next. If coverage minimums cannot be met, the release waits for newly reviewed items.
  • The selection seed is committed (SHA-256) before any inference, and the draw is a deterministic function of policy, seed and item id.

API-exposure rule

  • An external exposure is any sealed item sent to an endpoint we do not control: closed APIs, and open-weights models reached through third-party hosts, routers or a submitter's endpoint. Timeouts count as exposure.
  • From v1.6.1 hosted models receive the full item set (S plus P); items they had already answered were reused and only the missing sealed items were sent. Hosted models are re-measured at most once every three refresh releases unless a verified new model version ships. Between measurements they keep their last score with its measurement date.
  • Every externally sent sealed item is logged before dispatch. An item exposed to a provider is never scored again for that provider; once two different providers have received it, it retires for everyone. In v1.6.0 Jev 1.13.0 and Fastino GLiNER-2.5-Decide both received the same A, so A retires globally. With the v1.6.1 amendment every v1.6.0 sealed item has been seen by at least two providers, so the whole v1.6.0 sealed draw is retired for future releases.

Public-versus-sealed gap penalty

Intelligence is half public, half sealed. A system whose public score exceeds its sealed score by more than the field-median gap (G_med = 2.6 points) plus 8 points loses one Intelligence point per excess point. Hosted APIs are compared on P versus A against the same self-hosted systems' P-versus-A gap (-1.2 points).

Hosted APIs (not equated since v1.6.1)

Hosted APIs that answered the full set are scored exactly like self-hosted systems; no equating offset is applied to them. Category and language values stay raw.

Headline and Composite

Capability = mean(Intelligence, Calibration) for systems within twice the Jev 1.13.0 cost and median latency (the official caps; the sliders change only your view). The Composite (option A) is the equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost with the v1.5 low-axis gates; it remains secondary. Request types Choice, Noul and Score weigh equally; tiers weigh easy 0.1, standard 0.2, hard 0.4, judge 0.3.

Latency. API models are timed end to end from our measuring host, network included. Self-hosted models are timed on our own hardware over loopback (on-GPU, no network), so their measured value is not directly comparable with API latency. The ranking and the latency cap use the same estimate for every self-hosted row: 2 × measured p50 + 150 ms (est. incl. network (2x + 150 ms)). Tables show the measured value first, tagged “on-GPU, no network”, and the estimate second. A model author can provide a hosted endpoint; we then time it exactly like an API, network included, with no assumption (on request: info@productivity-boost.com, or note it in your submission).

Jev-class caps use a fixed reference: Jev 1.13.0 as measured in v1.5 (p50 0.62 s, USD 0.0323 per 1,000 decisions); caps = 2x (1.23 s, USD 0.0646). Jev's own v1.6 p50 is 0.24 s.

Costs of v1.6-measured systems carry each system's published v1.5.4 cost per 1,000 decisions (pricing rules unchanged; v1.6 item lengths differ) unless the row says otherwise; Fastino's is an estimate from its published tariff and measured tokens.

Noul decisiveness and Score baseline (addendum B)

Scored with method option B (scorer setting O1S), selected on 3 Oct 2026 after the v1.6 scores were known and disclosed as a post-results change. Each split × type competence is clipped at 0 before the type weighting, so a type answered no better than chance counts as chance instead of negative. The Score chance baseline is the error of always predicting the mid-scale level, so a flat know-nothing distribution earns about 0. A Score cell whose golds all sit at mid-scale keeps the v1.5 random-level baseline; this only occurs in small breakdown and bootstrap cells. Calibration is unchanged. This run: G_med = 2.57 points.

Failed requests and very long items

23 items of this draw are very long (about 77,000 to 81,000 input tokens). Systems with a shorter context window refuse them; Jev-Omni runs out of GPU memory on them on its listed RTX 6000 (48 GB) recipe; decider-4b v2's server truncates them to about 32,800 tokens and answers; Plumb-4B reads them in full. Each outcome is the system's own recipe and is scored as such. A failed, refused or unparseable answer counts as wrong for Intelligence and stays in the denominator; it does not enter Calibration (the v1.5 rule, applied to every system). Failed answers per system: Sage 1.3.0 0/1500 · H2O-Lightning-4B v1.1 0/1500 · Mercury Decide 23/1500 · decisio v0.8.0 on gemma-4-31B-it 23/1500 · Jev 1.13.0 23/1500 · Quyet-1.0-Large 0/1500 · decider-12b v2 0/1500 · wity-1 (Wity, reasoning auto) 0/1500 · decider-12b v1, stock Gemma-4-12B-it 0/1500 · torchcast-decision-12b 23/1500 · Winnow-12B Q8 23/1500 · deck-31B 23/1500 · Cygnet 23/1500 · decisio v0.8.0 on gemma-4-12B-it 23/1500 · decisio v0.9.0 on gemma-4-12B-it 23/1500 · Jev-Omni 23/1500 · Xor 26B-A4B 0/1500 · Bobcat Flash 1.2 0/1500 · SPX-CD Flash 23/1500 · decider chat on Gemma-4-31B-it 22/1500 · Hopper 12B trained 23/1500 · Surogate Rune 26B-A4B v3 23/1500 · GEV-26B-Decide 23/1500 · SPX-CD-Omni 23/1500 · Diffusion Jev 26/1500 · René-1 31B FP8 23/1500 · Aplomb 1 0/1500 · Bespoke Nimble 9B v3 23/1500 · APUS-OpenJev-v1-9B 23/1500 · Plumb-4B 0/1500 · SPX-CD Pro 23/1500 · Messier One v0.2 0/1500 · spark-s1-4b-v6 0/1500 · swanOne 23/1500 · Quyet-1.0-Medium 0/1500 · lev 25/1500 · NInfer Qwen3.8-Flash-Next mixed 23/1500 · jev-local 23/1500 · decider-4b v2 0/1500 · ClassOne Qwen 3.5 9B 23/1500 · JevK5 v0.3 23/1500 · metask-jev-4b 23/1500 · deck-4B v1.0 23/1500 · Clef-Flash 0/1500 · Decision-4B 23/1500 · janus 4B 23/1500 · Qwen3.5-9B Jev-like data-mix v2 23/1500 · JevK5 v0.2.0 23/1500 · JevOne 0/1500 · Kahn1 4B 23/1500 · TypeCastLM 1.4.0 0/1500 · Hopper 0/1500 · Imajev-4B 23/1500 · Malkuth-4B 23/1500 · jqv 23/1500 · JPT-35B-A3B 23/1500 · decider-35b-a3b 0/1500 · CoCo-Decision-4B 23/1500 · Manchego v2.1 23/1500 · Decision 4B v1.2 23/1500 · Raw Qwen3 4B Instruct 2507 direct logits 23/1500 · JEV-27B 23/1500 · torchcast-decision-27b 0/1500 · Bespoke Nimble 9B 23/1500 · reflex 4B 0/1500 · Perplexity Decider v1.1 27B 23/1500 · JADE 392/1500 · typecastlm 0/1500 · Jebadiah 27B 23/1500 · Decision 2.0 Vega 27B 23/1500 · Decision 4B v1.1 23/1500 · AutoJev-27B 23/1500 · Eikos-27B 23/1500 · NInfer Qwen3.8-27B NVFP4 23/1500 · OpenJev 23/1500 · Open-Jev 9B 23/1500 · Standard One 8B 23/1500 · Clef 0/1500 · JEV Qwen3.5-9B Base NVFP4 23/1500 · SemIf, formerly OpenJev 23/1500 · Bev / Bonsai 27B 23/1500 · LitJev 23/1500 · local-jev Qwen3.5-4B 0/1500 · system-one 23/1500 · kev 8B 25/1500 · Decision 2B 23/1500 · OpenSourceJev 23/1500 · open-alternative-jev 0/1500 · Raw Qwen3 8B direct logits 23/1500 · decider-2b 0/1500 · Nemotron Diffusion 8B 23/1500 · kev 4B 29/1500 · Kev 27B 23/1500 · Malkuth-2B 23/1500 · Qwen3-Reranker-4B 0/1500 · SimpleJev Qwen3.8-27B 23/1500 · ZeroEntropy zerank-2 0/1500 · EXAONE-4.0-1.2B-JEV v0.3 23/1500 · Open-Jev 2B 23/1500 · Open-Jev 27B v1.1 23/1500 · Raw Phi-4 mini direct logits 23/1500 · SimpleJev 23/1500 · smalljev semantic-v9 0/1500 · kev 0.6B 30/1500 · jul fast 2/1500 · OpenDecision 0/1500 · ClassOne Gemma 4 E2B 23/1500 · Qwen3.5-0.8B Decision Model 0/1500 · Fastino GLiNER-2.5-Decide 31/1500 · Deem 0.8B v1 23/1500 · Decision Fast 23/1500 · Quyet-1.0-Small-EN 0/1500 · Laya typed-decisions 0/1500 · WaterSheep 0/1500 · openJev Verdict 1.4 0/1500 · Raw Qwen3 0.6B direct logits 23/1500 · lev-350m 23/1500 · Raw Qwen3 1.7B direct logits 23/1500 · kev 0.5B 32/1500 · openJev Verdict 0/1500 · Quyet-1.0-Small 0/1500 · watt-flash-0.1 23/1500 · Quyet-1.0-Tiny 0/1500 · open-jev-deberta-v3-large 73/1500 · verdict-small 0/1500 · Tacet Sonata 0/1500 · Laya 0/1500 · Mixedbread mxbai-rerank-base-v2 0/1500 · Laya multilingual 0/1500 · Certo v1 0/1500 · CLM-8B 0/1500 · BAAI bge-reranker-v2-m3 0/1500 · Alibaba GTE Reranker ModernBERT-base 0/1500 · Mirror 592/1500 · Open Jev JSON Canvas 23/1500 · APUS-OpenJev-v1-35B-A3B 23/1500 · BB-Qwen3.5-4B-LoRA 0/1500 · Seb-9B 23/1500 · wity-1 (Wity, reasoning always) 0/1500 · wity-1 (Wity, reasoning off) 0/1500.

Calibration basis

Systems that return a full probability distribution are calibrated on all components (top-label error, plus distribution distance for Choice and ranked-probability error for Score). Fastino GLiNER-2.5-Decide returns a single confidence value, so its Calibration is the top-label error only and is not like-for-like with full-distribution systems.

Overnight full re-measure (4–5 Oct 2026)

Every system with a reproducible recipe was re-run on the v1.6.0 pool overnight with the same pinned inputs and scorer (method option B / O1S). This page uses scoring round score-v161-3 (v1.6.1) (2026-10-06 00:39:36 UTC). Only complete runs (1,500 items self-hosted, the full API input for hosted APIs) are ranked; partial runs are never ranked, and systems not yet re-measured keep their dated v1.5.x score in the separate table.

  • Scores are the official v1.6.0 scorer (score_v16.py, method option B / O1S, bootstrap B = 1,000) over complete outputs only; each scoring round is kept separately.
  • Hosted APIs are scored on the same full item set as self-hosted systems (S 1,200 + P 300, v1.6.1) and are not equated.
  • GPU-class deviations: reproducible recipes used the hardware listed per row, including H100 for large fast-lane decoders and RTX 6000 for the baseline; hardware differences remain in measured latency. The standard x2 + 0.15 s self-hosted adjustment is an assumption, not a hardware normalization.
  • Wity auto remains outside the latency cap and stays ranked in Composite A. OFF and ALWAYS are unranked variants of the AUTO main row.
  • 6 further measured candidates await a separate publication decision.
  • Sage 1.3.0 (Levanto Labs) was measured on 5 Oct from Sandy (Helsinki), text only. Cost uses the Levanto list tariff (USD 0.05/M input, USD 10/M output; levanto.ai/pricing, read 5 Oct) over all 600 answered rows: USD 0.0766/1,000 decisions. Superseded by the v1.6.1 cost rule (common item set): USD 0.0247/1,000 decisions, inside the Jev-class cost cap.
  • A fresh Monday Jev 1.13.0 re-check on A3 measured p50 0.239 s and equated Capability 77.5 versus the official 76.5, within the confidence interval; the official v1.6.0 Jev row is retained.

Supplementary API draws A2 and A3

History: hosted APIs were first measured on API subsets (A, then A2 and A3) and equated to the self-hosted scale in v1.6.0. From v1.6.1 they answer the same full set as self-hosted systems, so no equating is applied; the earlier subsets A, A2 and A3 are retired.

Per-model exposure counts (hosted and author-hosted endpoints)

System (provider)v1.6 sealed items sentScored sealed setStatus
wity-1-auto (wity via railway)1200S (1,200 sealed) + P (300 public)full set since v1.6.1 (5 Oct 2026); earlier API-subset answers reused, only missing sealed items sent; whole v1.6.0 sealed draw retired
wity-1-off (wity via railway)1200S (1,200 sealed) + P (300 public)full set since v1.6.1 (5 Oct 2026); earlier API-subset answers reused, only missing sealed items sent; whole v1.6.0 sealed draw retired
wity-1-always (wity via railway)1200S (1,200 sealed) + P (300 public)full set since v1.6.1 (5 Oct 2026); earlier API-subset answers reused, only missing sealed items sent; whole v1.6.0 sealed draw retired
jev-1.13.0 (typesafe)1200S (1,200 sealed) + P (300 public)full set since v1.6.1 (5 Oct 2026); earlier API-subset answers reused, only missing sealed items sent; whole v1.6.0 sealed draw retired
fastino-gliner-2-5-decide (fastino)1200S (1,200 sealed) + P (300 public)full set since v1.6.1 (5 Oct 2026); earlier API-subset answers reused, only missing sealed items sent; whole v1.6.0 sealed draw retired
gpt-6-luna (openai)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
gpt-6-luna-low (openai)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
gpt-5.6-luna (openai)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
gemini-3.1-flash-lite (google)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
deepseek-flash (deepseek)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
qwen3.8-27b (chutes)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
decision-machine-1 (milliseconds.ai)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
classifier-dev-fast (classifier.dev)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
vansa-3.4 (vansa)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
kushal-gemma4-31b-it-autoloops (autoloops)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
system-one-open (modal via modal)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
openjev-sglang (modal via modal)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
instinct (zoowork)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
instinct-dual-4b (zoowork)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
simplejev-qwen3.8-27b (featherless)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
simplejev-qwen3.6-35b-a3b (featherless)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
jevact (jevact)0v1.5 sealed-720 (retired for scoring)carried: last measured on v1.5 (API-exposure rule; not due / not in tonight's scope)
Sage 1.3.0 (Levanto Labs)1200S (1,200 sealed) + P (300 public)full set since v1.6.1 (5 Oct 2026); earlier API-subset answers reused, only missing sealed items sent; whole v1.6.0 sealed draw retired

Revision history

  • v1.6.1 · 2026-10-05 · Hosted APIs (fastino-gliner-2-5-decide, jev-1.13.0, sage-1.3.0, wity-1, wity-1-always, wity-1-off) answer the full 1,500-item set instead of the 600-item API subset and are no longer equated; self-hosted rows unchanged; whole v1.6.0 sealed draw retired. Cost per 1,000 decisions is measured on one common item set (all items except the 23 outside the Jev reference input range) for every token-priced row.
  • v1.6.0 · 2026-10-05 · Rotating item draw (1,200 sealed + 300 public); hosted APIs on API subsets equated to the self-hosted scale; /jev-models/v1.6.0 stays available unchanged.

Provenance

Measured on the v1.6 pool (run completion day, UTC): 2026-10-02: Winnow-12B Q8, Cygnet, Jev-Omni, Plumb-4B, swanOne, NInfer Qwen3.8-Flash-Next mixed, jev-local, decider-4b v2, JevK5 v0.3, metask-jev-4b, Qwen3.5-9B Jev-like data-mix v2, JevK5 v0.2.0, JevOne, Hopper, Imajev-4B, Malkuth-4B, jqv, decider-35b-a3b, Decision 4B v1.2, Raw Qwen3 4B Instruct 2507 direct logits, Bespoke Nimble 9B, reflex 4B, typecastlm, Decision 4B v1.1, AutoJev-27B, Eikos-27B, NInfer Qwen3.8-27B NVFP4, OpenJev, Open-Jev 9B, Standard One 8B, JEV Qwen3.5-9B Base NVFP4, SemIf, formerly OpenJev, LitJev, local-jev Qwen3.5-4B, system-one, kev 8B, OpenSourceJev, open-alternative-jev, Raw Qwen3 8B direct logits, decider-2b, kev 4B, Malkuth-2B, Qwen3-Reranker-4B, ZeroEntropy zerank-2, Open-Jev 2B, Raw Phi-4 mini direct logits, SimpleJev, smalljev semantic-v9, kev 0.6B, OpenDecision, Qwen3.5-0.8B Decision Model, openJev Verdict 1.4, Raw Qwen3 0.6B direct logits, Raw Qwen3 1.7B direct logits, kev 0.5B, openJev Verdict, open-jev-deberta-v3-large, verdict-small, Laya, Mixedbread mxbai-rerank-base-v2, Certo v1, CLM-8B, BAAI bge-reranker-v2-m3, Alibaba GTE Reranker ModernBERT-base, Mirror, Open Jev JSON Canvas · 2026-10-04: H2O-Lightning-4B v1.1, Quyet-1.0-Large, decider-12b v2, decider-12b v1, stock Gemma-4-12B-it, torchcast-decision-12b, Quyet-1.0-Medium, lev, Clef-Flash, Decision-4B, janus 4B, Manchego v2.1, Clef, Bev / Bonsai 27B, Nemotron Diffusion 8B, Open-Jev 27B v1.1, Deem 0.8B v1, Quyet-1.0-Small-EN, Laya typed-decisions, Quyet-1.0-Small, Quyet-1.0-Tiny, Laya multilingual · 2026-10-05: Sage 1.3.0, Jev 1.13.0, deck-31B, spark-s1-4b-v6, deck-4B v1.0, Decision 2B, Fastino GLiNER-2.5-Decide, Decision Fast, lev-350m · 2026-10-06: decisio v0.8.0 on gemma-4-31B-it, wity-1 (Wity, reasoning auto), decisio v0.8.0 on gemma-4-12B-it, Xor 26B-A4B, SPX-CD Flash, decider chat on Gemma-4-31B-it, Hopper 12B trained, Surogate Rune 26B-A4B v3, GEV-26B-Decide, SPX-CD-Omni, Diffusion Jev, Aplomb 1, Bespoke Nimble 9B v3, SPX-CD Pro, JPT-35B-A3B, JEV-27B, torchcast-decision-27b, Perplexity Decider v1.1 27B, JADE, Jebadiah 27B, Decision 2.0 Vega 27B, Kev 27B, SimpleJev Qwen3.8-27B, wity-1 (Wity, reasoning always), wity-1 (Wity, reasoning off) · 2026-10-07: Mercury Decide, decisio v0.9.0 on gemma-4-12B-it, Bobcat Flash 1.2, René-1 31B FP8, APUS-OpenJev-v1-9B, Messier One v0.2, ClassOne Qwen 3.5 9B, Kahn1 4B, TypeCastLM 1.4.0, CoCo-Decision-4B, EXAONE-4.0-1.2B-JEV v0.3, jul fast, ClassOne Gemma 4 E2B, WaterSheep, watt-flash-0.1, Tacet Sonata, APUS-OpenJev-v1-35B-A3B, BB-Qwen3.5-4B-LoRA, Seb-9B.

Aggregate files: results sha256 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1 · categories sha256 0159c4828b3e8620148af06848ef0a5ccfc7d7f0c11aec02460d844dc0c62689 · dated carry sha256 334e851944e92383ab3071e52b3012040c7023d0fe236395953169ebee8f0459. Scoring source sha256 e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911. The method, release data and carry artifact are independently hashable.

Historical v1.3.0 board: weightings, per-task grid, topic radars and held-out diagnostics

Open this section to load the earlier public-only board and diagnostics.