Benchmarks
More
OverviewBenchmaxxingCompareChartsCost vs CapabilityProviders per ModelProvider explorerGatewaysEU & SovereignJev-class modelsImage JevBenchAudioJevBenchSubmit a modelAbout

Image benchmark · v0.3.0

ImageJevBench — image decision model benchmark

ImageJevBench by Benchmark Heaven compares image decision models on accuracy, probability calibration, latency and cost. JevImageBench Capability Score leads within the frozen serving budget.

57 measured systems · 55 accepted for ranking · 2 dated carries. Built 2026-10-12.

New item pool: 2,441 items on 2,160 images, fresh 300 public / 1,200 sealed draw. Measured rows with unresolved acceptance or equating holds remain unrankable. Carried scores belong to v0.1.5 and are not ranked on the v0.3 scale.

JevImageBench v0.3.0 · headline

JevImageBench Capability Score

Capability ranking of Jev-class systems

Capability Score averages Intelligence and Calibration. Imajev-4B leads the Jev-class systems with 68.0.

Jev-class means at most 2× the cost and median latency of Jev 1.13.0 (JevBench). How we choose ↘

Adjust cost / latency caps · 2× official
Official
#SystemICCap.$/1k
  1. 1Imajev-4B49.951.868.0$0.038
  2. 2Surogate Rune 26B-A4B v344.847.164.8$0.058
  3. 3AutoJev-27B37.647.962.1$0.054
  4. 4Jevify Gemma 4 26B-A4B38.547.358.7$0.057
  5. 5NeoHorse Jev 4B36.850.457.6$0.042
  6. 6vjev-vision21.759.055.7$0.023
  7. 7JPT-4B38.345.755.6$0.065
  8. 8Jev-Omni37.560.054.3$0.022
  9. 9Reflex 4B27.448.453.3$0.052
  10. 10shisa-de-134.947.251.5$0.057
Show all 30 Jev-class systems (20 more)
  1. 11Gevva E4B17.447.049.6$0.059
  2. 12Liquid d1-3B16.354.147.8$0.034
  3. 13Gevva E2B multimodal12.848.847.3$0.051
  4. 14OmniJev 4B28.049.347.2$0.049
  5. 15imajev 2B19.548.046.4$0.048
  6. 16JPT-0.8B20.453.746.1$0.035
  7. 17Jevify Qwen3-VL-2B T215.153.543.9$0.036
  8. 18Jevify Gemma 4 E4B20.458.743.7$0.024
  9. 19Mapika decider-2b-vision BF1623.154.042.8$0.034
  10. 20OmniJev 0.8B11.454.138.6$0.034
  11. 21OmniJev 2B15.052.337.9$0.039
  12. 22Jev-Vision 8B39.047.437.8$0.056
  13. 23Visual-Jev 4B Answer-SFT24.047.435.9$0.057
  14. 24OmniJev-Qwen3.5-9B-v429.057.633.1$0.026
  15. 25Glance21.352.828.4$0.035
  16. 26jev-spatial13.958.827.4$0.024
  17. 27Liquid d1-omni-600M5.763.626.5$0.016
  18. 28PlayJev 0.8B7.653.925.6$0.034
  19. 29djev-spark NVFP41.536.624.5$0.064
  20. 30Qevi-2B2.752.311.1$0.039

Wide coloured bar = Capability Score (0–100). Thin bars: cost above, median latency below; shared log ratio scale 0.25× … 64× Jev 1.13.0 (JevBench), ticks at 1× and 2× (cap). Shorter is cheaper or faster. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ or hover a row or thin bar for details.

Cost and latency correlate here (Spearman ρ = 0.69, n = 55).

  • Closed API (weights not public) (9)
  • Open weights · LLM decoder (44)
  • Open weights · diffusion LM (4)
  • green: ≤ reference
  • amber: ≤ cap (2× reference)
  • red: > cap
Show general-purpose LLMs and other systems outside the limits

Sorted by Capability Score, not numbered. Each row says which limit it misses, measured against Jev 1.13.0 ($0.032 per 1,000 decisions, median 0.62 s).

  1. –GPT-5.6 LunaAPI76.020.976.9$0.43Outside: cost 13.4× Jev 1.13.0 (JevBench), latency 5.4× Jev 1.13.0 (JevBench)
  2. –GPT-6 Luna (low)API72.525.074.5$0.32Outside: cost 9.8× Jev 1.13.0 (JevBench), latency 6.8× Jev 1.13.0 (JevBench)
  3. –Wity-1API68.771.874.1$0.0087Outside: latency 4.4× Jev 1.13.0 (JevBench)
  4. –Autoloops – Gemma 4 31B ITAPI49.742.468.4$0.083Outside: cost 2.6× Jev 1.13.0 (JevBench)
  5. –imajev 9B48.340.167.1$0.089Outside: cost 2.7× Jev 1.13.0 (JevBench)
  6. –Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj43.228.963.1$0.23Outside: cost 7.2× Jev 1.13.0 (JevBench), latency 2.4× Jev 1.13.0 (JevBench)
  7. –GPT-6 Luna via OpenAI DecisionsAPI45.132.262.9$0.18Outside: cost 5.6× Jev 1.13.0 (JevBench)
  8. –H2O-Lightning-4B44.943.260.4$0.078Outside: cost 2.4× Jev 1.13.0 (JevBench)
  9. –JPT-9B44.342.558.7$0.083Outside: cost 2.6× Jev 1.13.0 (JevBench)
  10. –System1 s1-visionAPI42.139.558.3$0.10Outside: cost 3.2× Jev 1.13.0 (JevBench)
  11. –Clef47.432.258.1$0.18Outside: cost 5.6× Jev 1.13.0 (JevBench)
  12. –Visual-Jev generic29.740.456.7$0.091Outside: cost 2.8× Jev 1.13.0 (JevBench)
  13. –Wity-1API37.571.856.2$0.0087Outside: latency 2.7× Jev 1.13.0 (JevBench)
  14. –Clef-Flash37.342.252.2$0.084Outside: cost 2.6× Jev 1.13.0 (JevBench)
  15. –Image Hopper32.542.751.7$0.077Outside: cost 2.4× Jev 1.13.0 (JevBench)
  16. –Standard One 8B27.7—51.5—Outside: no cost reported
  17. –Winnow-12B38.638.050.0$0.087Outside: cost 2.7× Jev 1.13.0 (JevBench)
  18. –Standard One 3B13.5—49.6—Outside: no cost reported
  19. –CUA-S1 4B30.042.949.4$0.080Outside: cost 2.5× Jev 1.13.0 (JevBench)
  20. –Gemini 3.1 Flash LiteAPI45.822.146.9$0.39Outside: cost 12.2× Jev 1.13.0 (JevBench), latency 2.6× Jev 1.13.0 (JevBench)
  21. –djev-distill-v418.841.140.2$0.072Outside: cost 2.2× Jev 1.13.0 (JevBench)
  22. –diffusiongemma-26b djev v10 step16023.241.236.7$0.071Outside: cost 2.2× Jev 1.13.0 (JevBench)
  23. –djev-dev BF163.935.425.4$0.072Outside: cost 2.2× Jev 1.13.0 (JevBench)
  24. –Gemini 3.8 FlashAPI7.70.019.4$2.17Outside: cost 67.2× Jev 1.13.0 (JevBench), latency 8.0× Jev 1.13.0 (JevBench)
  25. –OpenJev 4B NLI v532.834.016.4$0.15Outside: cost 4.8× Jev 1.13.0 (JevBench), latency 2.0× Jev 1.13.0 (JevBench)
  26. –JEVision15.045.016.2$0.066Outside: cost 2.0× Jev 1.13.0 (JevBench)
  27. –OpenJev 4B NLI v216.433.98.2$0.16Outside: cost 4.8× Jev 1.13.0 (JevBench), latency 2.0× Jev 1.13.0 (JevBench)

Core is the headline. Capability is mean Intelligence and Calibration (0–100). The frozen v0.1.5 envelope remains 2× Jev 1.13.0 on JevBench v1.5.4: USD 0.06459 / 1,000 and adjusted p50 ≤ 1.233 s. Cost and latency traffic lights use this anchor. Only accepted rows inside both caps receive a Capability rank. The charts below show speed and cost beside Capability Score; the official JevImageBench Score weighs all four axes.

Capability against cost and speed

Jev-class systems are shown by default. These charts use the official 2× caps. Bubble size follows the official Image JevBench Score; official rank stays unchanged. The five most capable Jev-class systems are labelled.

Capability vs cost

Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.

Capability vs cost: 30 systems. Upper right is best: more capable and cheaper.102030405060708090100$0.010$0.10$ per 1,000 decisions (log)Capability↑2× Jev 1.13.0 (JevBench)← priciercheaper →1. Imajev-4B2. Surogate Rune 26B-A4B v33. AutoJev-27B4. Jevify Gemma 4 26B-A4B5. NeoHorse Jev 4B
30 systems. Tap a bubble for its values.

Capability vs speed

Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). Self-hosted systems are plotted at the est. incl. network latency (2x measured + 150 ms); their measured loopback p50 (on-GPU, no network; not directly comparable with API latency) is in the tooltip. The dashed line is 2× Jev’s median latency.

Capability vs speed: 30 systems. Upper right is best: more capable and faster.10203040506070809010070≈3.2 s80≈1.0 s90≈316 ms100≈100 msMedian-latency speedCapability↑2× Jev 1.13.0 (JevBench)← slowerfaster →1. Imajev-4B2. Surogate Rune 26B-A4B v33. AutoJev-27B4. Jevify Gemma 4 26B-A4B5. NeoHorse Jev 4B
30 systems. Tap a bubble for its values.
Explore capability, cost and speed in 3D

Image JevBench v0.3.0

Image JevBench composite: 55 ranked systems

Official· four axes 0–100, equal-weight harmonic mean · Method notes ↓

Weights:
Adjust weights ↓

View by:Capability ↑

The official order weighs Intelligence, Calibration, Speed and Cost equally. Each button re-sorts the same systems by one axis.

Greener = stronger within its column.

57 of 57 systems, sorted by official rank, #1 first.

  1. 1Wity-1 (AlphaNimble), reasoning autoAPIdetailsClosed APIBase model: undisclosed*171.2I 69C 79S 66K 72$0.0087
  2. 2Imajev-4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource63.7I 50C 86S 86K 52$0.038
  3. 3Surogate Rune 26B-A4B v3details26B-A4BBase model: google/gemma-4-26B-A4B-itsource42.5I 45C 85S 85K 47$0.058
  4. 4Autoloops – Gemma 4 31B ITAPIdetailsClosed APIBase model: undisclosed41.5I 50C 87S 75K 42$0.083
  5. 5imajev 9Bdetailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource34.5I 48C 86S 81K 40$0.089
  6. 6H2O-Lightning-4BdetailsLLM decoderBase model: undisclosed34.1I 45C 76S 84K 43$0.078
  7. 7Wity-1 (AlphaNimble), reasoning offAPIdetailsClosed APIBase model: undisclosed33.2I 37C 75S 72K 72$0.0087
  8. 8Jev-Omnidetailsfine-tune · 12BBase model: Gemma 4 12B ITsource32.8I 38C 71S 89K 60$0.022
  9. 9JPT-9Bdetailsadapter · 9BBase model: Qwen3.5-9Bsource31.4I 44C 73S 82K 42$0.083
  10. 10NeoHorse Jev 4Bdetails4BBase model: NeoHorse-1-4Bsource30.5I 37C 78S 87K 50$0.042
  11. 11Jevify Gemma 4 26B-A4Bdetailsadapter · 26B-A4BBase model: google/gemma-4-26B-A4B-itsource29.7I 38C 79S 86K 47$0.057
  12. 12AutoJev-27Bdetailsfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource29.3I 38C 87S 86K 48$0.054
  13. 13JPT-4Bdetailsadapter · 4BBase model: Qwen3.5-4Bsource26.5I 38C 73S 83K 46$0.065
  14. 14Jev-Vision 8Bdetailsadapter · 8BBase model: Qwen/Qwen3-VL-8B-Instructsource25.5I 39C 37S 85K 47$0.056
  15. 15System1 s1-visionAPIClosed APIBase model: undisclosed24.0I 42C 75S 91K 39$0.104
  16. 16shisa-de-1details26B-A4BBase model: Gemma 4 26B-A4Bsource22.8I 35C 68S 85K 47$0.057
  17. 17Clef-FlashLLM decoderBase model: undisclosed20.4I 37C 67S 81K 42$0.084
  18. 18ClefLLM decoderBase model: undisclosed18.7I 47C 69S 76K 32$0.182
  19. 19GPT-6 Luna via OpenAI DecisionsAPIdetailsClosed APIBase model: undisclosed17.4I 45C 81S 82K 32$0.181
  20. 20Winnow-12Bdetailsmerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsource17.1I 39C 61S 83K 38$0.087
Show all 57 systems (35 more ranked, 2 more not ranked)
  1. 21Image HopperLLM decoderBase model: undisclosed15.3I 32C 71S 83K 43$0.077
  2. 22OmniJev-Qwen3.5-9B-v4detailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource15.0I 29C 37S 89K 58$0.026
  3. 23OmniJev 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource14.8I 28C 66S 86K 49$0.049
  4. 24Reflex 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource13.8I 27C 79S 86K 48$0.052
  5. 25CUA-S1 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource12.7I 30C 69S 82K 43$0.080
  6. 26GPT-6 LunaAPIdetailsClosed APIBase model: undisclosed12.2I 72C 77S 65K 25$0.315
  7. 27Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjdetailsLLM decoderBase model: undisclosedsource12.0I 43C 83S 73K 29$0.234
  8. 28Visual-Jev genericdetails4BBase model: undisclosed11.1I 30C 84S 81K 40$0.091
  9. 29Mapika decider-2b-vision BF16detailsfine-tune · 2BBase model: Qwen/Qwen3.5-2B-Basesource9.5I 23C 63S 86K 54$0.034
  10. 30vjev-visiondetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource8.9I 22C 90S 90K 59$0.023
  11. 31Visual-Jev 4B Answer-SFTdetailsadapter · 4BBase model: Qwen/Qwen3-VL-4B-Instructsource8.7I 24C 48S 84K 47$0.057
  12. 32GPT-5.6 LunaAPIdetailsClosed APIBase model: undisclosed7.9I 76C 78S 67K 21$0.433
  13. 33Jevify Gemma 4 E4Bdetailsadapter · 4BBase model: google/gemma-4-E4B-itsource7.3I 20C 67S 90K 59$0.024
  14. 34JPT-0.8Bdetailsadapter · 0.8BBase model: Qwen3.5-0.8Bsource7.2I 20C 72S 87K 54$0.035
  15. 35Glancedetails4BBase model: Qwen3-VL-4Bsource6.9I 21C 36S 88K 53$0.035
  16. 36Gemini 3.1 Flash LiteAPIdetailsClosed APIBase model: undisclosed6.5I 46C 48S 74K 22$0.394
  17. 37diffusiongemma-26b djev v10 step160detailsfine-tune · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource5.9I 23C 50S 83K 41$0.071
  18. 38imajev 2Bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2Bsource5.8I 20C 73S 85K 48$0.048
  19. 39Gevva E4Bdetails4BBase model: Gemma 4 E4Bsource4.1I 17C 82S 84K 47$0.059
  20. 40Liquid d1-3BdetailsLLM decoderBase model: undisclosed4.1I 16C 79S 88K 54$0.034
  21. 41djev-distill-v4detailsdistilled · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource3.6I 19C 62S 83K 41$0.072
  22. 42Jevify Qwen3-VL-2B T2detailsadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsource3.3I 15C 73S 87K 53$0.036
  23. 43OmniJev 2Bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2Bsource3.2I 15C 61S 88K 52$0.039
  24. 44jev-spatialdetailsadapterBase model: allenai/Molmo2-ERsource2.5I 14C 41S 90K 59$0.024
  25. 45Gevva E2B multimodaldetails2BBase model: Gemma 4 E2Bsource2.0I 13C 82S 85K 49$0.051
  26. 46JEVisiondetails0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource1.8I 15C 17S 84K 45$0.066
  27. 47OmniJev 0.8Bdetailsadapter · 0.8BBase model: Qwen/Qwen3.5-0.8Bsource1.6I 11C 66S 88K 54$0.034
  28. 48PlayJev 0.8Bdetailsfine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource0.5I 8C 44S 87K 54$0.034
  29. 49Liquid d1-omni-600MdetailsLLM decoderBase model: undisclosed0.2I 6C 47S 91K 64$0.016
  30. 50djev-dev BF16details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource0.0I 4C 47S 83K 35$0.072
  31. 51Qevi-2Bdetailsadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsource0.0I 3C 20S 86K 52$0.039
  32. 52djev-spark NVFP4details26B-A4B · NVFP4Base model: google/diffusiongemma-26B-A4B-itsource 1source 20.0I 2C 48S 84K 37$0.064
  33. 53OpenJev 4B NLI v2details4BBase model: Qwen/Qwen3.5-4Bsource0.0I 16C 0S 75K 34$0.155
  34. 54OpenJev 4B NLI v5details4BBase model: Qwen/Qwen3.5-4Bsource0.0I 33C 0S 75K 34$0.155
  35. 55Gemini 3.8 FlashAPIdetailsClosed APIBase model: undisclosed0.0I 8C 31S 64K 0$2.170
  36. Standard One 3B (unranked)detailsadapter · 3BBase model: mistralai/Ministral-3-3B-Instruct-2512-BF16source—I 13C 86S 85K –—
  37. Standard One 8B (unranked)detailsadapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source—I 28C 75S 83K –—

Base-model notes

  1. *1 Wity-1 (base model: undisclosed): Base model: undisclosed (author request).
Weights:
Adjust weights ↓

Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; the gates still apply when an axis sits at 0. Only equal weights give the official Image JevBench Score and rank.

Score = 4 / (1/I + 1/C + 1/S + 1/K) (each 0–100; × (axis / 50)² for Intelligence, Speed or Cost below 50)

I, C, S, K = Intelligence, Calibration, Speed, Cost; the est. pill = estimated cost; ann. = announced price; API = the operator's endpoint saw held-out benchmark inputs, without answers; $/1k decisions = US dollars per 1,000 decisions (not heat-shaded). Names link to each project.

Compare two systems

Pick any two. Compare the four score axes, capability by image type and use cases and use cases; request-type and tier aggregates are not published for this benchmark. Further out is better on every spoke; the link keeps the pair.

  • A: Wity-1 — Closed API (weights not public) · Score 71.2 (#1)
  • B: Imajev-4B — Open weights · LLM decoder · Score 63.7 (#2)

The four score axes

Radar: the four score axes, two systemsThe four score axes, Wity-1 vs Imajev-4B. Intelligence: 68.7 vs 49.9; Calibration: 79.4 vs 86.1; Speed: 66.3 vs 86.3; Cost: 71.8 vs 51.8.50100Intelligence: Wity-1: 68.7; Imajev-4B: 49.9Intelligence68.7 · 49.9Calibration: Wity-1: 79.4; Imajev-4B: 86.1Calibration79.4 · 86.1Speed: Wity-1: 66.3; Imajev-4B: 86.3Speed66.3 · 86.3Cost: Wity-1: 71.8; Imajev-4B: 51.8Cost71.8 · 51.8
0–100, the values in the table. An axis a system has no published value for is left as a gap (it counts as 0 in the composite).

Capability by image type

Radar: capability by image type, two systemsCapability by image type, Wity-1 vs Imajev-4B. UI screenshots: 100.0 vs 99.0; Small text: 58.5 vs 10.3; Spatial: 24.1 vs 40.8; Diagrams: 76.2 vs 77.8; Receipts: 90.9 vs 80.3; Document photos: 57.8 vs 24.8; Misleading charts: 75.7 vs 58.0; Multi-panel comparison: 81.9 vs 55.1; Counting: 58.1 vs 11.6; Dense UIs: n=23 vs 67.2; Agent safety: n=26 vs 81.4; Chart reasoning: 100.0 vs 86.7; Multilingual: n=14 vs 100.0. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100UI screenshots: Wity-1: 100.0; Imajev-4B: 99.0 — UI screenshots: decisions on app and web screenshots. 199 items (39 public, 160 sealed).UIscreenshots100.0 · 99.0Small text: Wity-1: 58.5; Imajev-4B: 10.3 — Small text: reading small print, labels and fine text. 175 items (36 public, 139 sealed).Small text58.5 · 10.3Spatial: Wity-1: 24.1; Imajev-4B: 40.8 — Spatial: positions, directions, layouts and maps. 160 items (30 public, 130 sealed).Spatial24.1 · 40.8Diagrams: Wity-1: 76.2; Imajev-4B: 77.8 — Diagrams: flow charts, schematics and process diagrams. 141 items (32 public, 109 sealed).Diagrams76.2 · 77.8Receipts: Wity-1: 90.9; Imajev-4B: 80.3 — Receipts: checking receipts, totals and line items. 132 items (22 public, 110 sealed).Receipts90.9 · 80.3Document photos: Wity-1: 57.8; Imajev-4B: 24.8 — Document photos: photographed or scanned documents and forms. 118 items (30 public, 88 sealed).Documentphotos57.8 · 24.8Misleading charts: Wity-1: 75.7; Imajev-4B: 58.0 — Misleading charts: charts with truncated axes, dual scales and other traps. 115 items (21 public, 94 sealed).Misleadingcharts75.7 · 58.0Multi-panel comparison: Wity-1: 81.9; Imajev-4B: 55.1 — Multi-panel comparison: comparing several panels or images side by side. 109 items (17 public, 92 sealed).Multi-panelcomparison81.9 · 55.1Counting: Wity-1: 58.1; Imajev-4B: 11.6 — Counting: counting objects and instances. 105 items (22 public, 83 sealed).Counting58.1 · 11.6Dense UIs: Wity-1: n=23 — too few completed items to plot; Imajev-4B: 67.2 — Dense UIs: crowded dashboards and dense interfaces. 66 items (12 public, 54 sealed).Dense UIsn=23 · 67.2Agent safety: Wity-1: n=26 — too few completed items to plot; Imajev-4B: 81.4 — Agent safety: spotting risky actions and safety signals on screens. 61 items (14 public, 47 sealed).Agent safetyn=26 · 81.4Chart reasoning: Wity-1: 100.0; Imajev-4B: 86.7 — Chart reasoning: reading values and trends from charts. 60 items (16 public, 44 sealed).Chartreasoning100.0 · 86.7Multilingual: Wity-1: n=14 — too few completed items to plot; Imajev-4B: 100.0 — Multilingual: images with non-English text. 40 items (3 public, 37 sealed).Multilingualn=14 · 100.0

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

v0.3.0 item families and use cases; values are measured on the v0.3 draw (self-hosted: S+P, APIs: A+P). Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Wity-1: public+sealed; Imajev-4B: public+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.A (Wity-1) has values on 10 of 13 spokes; lines join only adjacent measured spokes.Gaps, not zeros: Open spokes (n/a or n=…) have fewer than 30 answered items for that system or no published value; hosted APIs answer a smaller item set, so more of their category cells stay under that.

Low sample, n < 30 — indicative only

Category (items)A: Wity-1B: Imajev-4B
Dense UIs (66; a system answered fewer than 30)65.6 n=2367.2 n=66
Agent safety (61; a system answered fewer than 30)92.3 n=2681.4 n=61
Surface inspection (19)—82.5 n=19

Not drawn on the radar: with so few items a single answer moves a category score by several points, so these values are noise-prone. These values become spokes once the item pool reaches 30 items per category.

What each category means · items per category
  • UI screenshots — decisions on app and web screenshots. 199 items (39 public / 160 sealed)
  • Small text — reading small print, labels and fine text. 175 items (36 public / 139 sealed)
  • Spatial — positions, directions, layouts and maps. 160 items (30 public / 130 sealed)
  • Diagrams — flow charts, schematics and process diagrams. 141 items (32 public / 109 sealed)
  • Receipts — checking receipts, totals and line items. 132 items (22 public / 110 sealed)
  • Document photos — photographed or scanned documents and forms. 118 items (30 public / 88 sealed)
  • Misleading charts — charts with truncated axes, dual scales and other traps. 115 items (21 public / 94 sealed)
  • Multi-panel comparison — comparing several panels or images side by side. 109 items (17 public / 92 sealed)
  • Counting — counting objects and instances. 105 items (22 public / 83 sealed)
  • Dense UIs — crowded dashboards and dense interfaces. 66 items (12 public / 54 sealed)
  • Agent safety — spotting risky actions and safety signals on screens. 61 items (14 public / 47 sealed)
  • Chart reasoning — reading values and trends from charts. 60 items (16 public / 44 sealed)
  • Multilingual — images with non-English text. 40 items (3 public / 37 sealed)

Use cases (ImageJev v0.3 decision use cases)

Radar: use cases (imagejev v0.3 decision use cases), two systemsUse cases (ImageJev v0.3 decision use cases), Wity-1 vs Imajev-4B. Document checks: 69.3 vs 41.6; Diagrams and planning: 49.7 vs 58.1; Chart decisions: 84.9 vs 62.9; UI agent actions: 92.4 vs 89.3; Counting and inspection: 64.5 vs 22.4. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Document checks: Wity-1: 69.3; Imajev-4B: 41.6 — Document checks: verifying fields, totals and facts in documents and receipts. 465 items (91 public, 374 sealed).Documentchecks69.3 · 41.6Diagrams and planning: Wity-1: 49.7; Imajev-4B: 58.1 — Diagrams and planning: reasoning over diagrams, maps, layouts and plans. 301 items (62 public, 239 sealed).Diagrams andplanning49.7 · 58.1Chart decisions: Wity-1: 84.9; Imajev-4B: 62.9 — Chart decisions: decisions based on charts and tables. 284 items (54 public, 230 sealed).Chartdecisions84.9 · 62.9UI agent actions: Wity-1: 92.4; Imajev-4B: 89.3 — UI agent actions: choosing the right or safe action on a screen (computer and browser use). 326 items (65 public, 261 sealed).UI agentactions92.4 · 89.3Counting and inspection: Wity-1: 64.5; Imajev-4B: 22.4 — Counting and inspection: counting and visual inspection of photos. 124 items (28 public, 96 sealed).Counting andinspection64.5 · 22.4

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

v0.3.0 item families and use cases; values are measured on the v0.3 draw (self-hosted: S+P, APIs: A+P). Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Wity-1: public+sealed; Imajev-4B: public+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.
What each category means · items per category
  • Document checks — verifying fields, totals and facts in documents and receipts. 465 items (91 public / 374 sealed)
  • Diagrams and planning — reasoning over diagrams, maps, layouts and plans. 301 items (62 public / 239 sealed)
  • Chart decisions — decisions based on charts and tables. 284 items (54 public / 230 sealed)
  • UI agent actions — choosing the right or safe action on a screen (computer and browser use). 326 items (65 public / 261 sealed)
  • Counting and inspection — counting and visual inspection of photos. 124 items (28 public / 96 sealed)

Use cases (TypeSafe categories)

Radar: use cases (typesafe categories), two systemsUse cases (TypeSafe categories), Wity-1 vs Imajev-4B. Model routing: 86.7 vs 73.3; Legal & compliance: 86.7 vs 64.4; Customer support: 86.7 vs 77.8; Other: 86.7 vs 77.8; E-commerce: 82.2 vs 77.8; Risk assessment: 100.0 vs 77.8; Insurance claims: 91.1 vs 77.8; Financial crime: 91.1 vs 68.9; Moderation: 82.2 vs 77.8; Feature extraction: 82.2 vs 95.6; Knowledge graphs: 95.6 vs 100.0; LLM guardrails: 95.6 vs 82.2; Search & retrieval: 91.1 vs 86.7; Lead generation: 95.6 vs 86.7; Code linting: 95.6 vs 77.8; Demand forecasting: 95.6 vs 77.8; Gaming: 86.7 vs 73.3; Science: 100.0 vs 91.1; Recruiting: 100.0 vs 77.8; Advertising: 95.6 vs 64.4. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Model routing: Wity-1: 86.7; Imajev-4B: 73.3 — Model routing: deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 30 items (0 public, 30 sealed).Modelrouting86.7 · 73.3Legal & compliance: Wity-1: 86.7; Imajev-4B: 64.4 — Legal & compliance: contracts, policies, regulations, eligibility rules, checking documents against written requirements. 30 items (0 public, 30 sealed).Legal &compliance86.7 · 64.4Customer support: Wity-1: 86.7; Imajev-4B: 77.8 — Customer support: classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 30 items (0 public, 30 sealed).Customersupport86.7 · 77.8Other: Wity-1: 86.7; Imajev-4B: 77.8 — Other: none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 30 items (0 public, 30 sealed).Other86.7 · 77.8E-commerce: Wity-1: 82.2; Imajev-4B: 77.8 — E-commerce: product listings, product attributes, orders and deliveries, seller catalog policy. 30 items (0 public, 30 sealed).E-commerce82.2 · 77.8Risk assessment: Wity-1: 100.0; Imajev-4B: 77.8 — Risk assessment: estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 30 items (0 public, 30 sealed).Riskassessment100.0 · 77.8Insurance claims: Wity-1: 91.1; Imajev-4B: 77.8 — Insurance claims: first-notice-of-loss reports, claim complexity, missing information, claim triage. 30 items (0 public, 30 sealed).Insuranceclaims91.1 · 77.8Financial crime: Wity-1: 91.1; Imajev-4B: 68.9 — Financial crime: suspicious transactions, KYC, fraud alerts, entity matching for investigations. 30 items (0 public, 30 sealed).Financialcrime91.1 · 68.9Moderation: Wity-1: 82.2; Imajev-4B: 77.8 — Moderation: user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 30 items (0 public, 30 sealed).Moderation82.2 · 77.8Feature extraction: Wity-1: 82.2; Imajev-4B: 95.6 — Feature extraction: turning natural-language data into probabilistic features or estimates for a downstream prediction. 30 items (0 public, 30 sealed).Featureextraction82.2 · 95.6Knowledge graphs: Wity-1: 95.6; Imajev-4B: 100.0 — Knowledge graphs: entity types, relationships between records, contradictions between facts, multi-step lookups across linked facts. 30 items (0 public, 30 sealed).Knowledgegraphs95.6 · 100.0LLM guardrails: Wity-1: 95.6; Imajev-4B: 82.2 — LLM guardrails: checking LLM inputs, outputs and tool calls: prompt injection, jailbreaks, policy violations, tool-call errors, answer quality. 30 items (0 public, 30 sealed).LLMguardrails95.6 · 82.2Search & retrieval: Wity-1: 91.1; Imajev-4B: 86.7 — Search & retrieval: scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 30 items (0 public, 30 sealed).Search &retrieval91.1 · 86.7Lead generation: Wity-1: 95.6; Imajev-4B: 86.7 — Lead generation: matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 30 items (0 public, 30 sealed).Leadgeneration95.6 · 86.7Code linting: Wity-1: 95.6; Imajev-4B: 77.8 — Code linting: checking code or writing against conventions and guidelines, as in CI review of code. 30 items (0 public, 30 sealed).Code linting95.6 · 77.8Demand forecasting: Wity-1: 95.6; Imajev-4B: 77.8 — Demand forecasting: purchase intent, product interest, demand and supply signals for forecasting. 30 items (0 public, 30 sealed).Demandforecasting95.6 · 77.8Gaming: Wity-1: 86.7; Imajev-4B: 73.3 — Gaming: player reports, in-game chat, game support. 30 items (0 public, 30 sealed).Gaming86.7 · 73.3Science: Wity-1: 100.0; Imajev-4B: 91.1 — Science: screening papers, labelling research passages or survey answers, checking that citations support claims, methodology checks. 30 items (0 public, 30 sealed).Science100.0 · 91.1Recruiting: Wity-1: 100.0; Imajev-4B: 77.8 — Recruiting: resumes, applications, interview feedback, matching candidates to roles. 30 items (0 public, 30 sealed).Recruiting100.0 · 77.8Advertising: Wity-1: 95.6; Imajev-4B: 64.4 — Advertising: ad creatives, campaign copy, brand safety, prohibited claims in ads. 30 items (0 public, 30 sealed).Advertising95.6 · 64.4

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Measured on a separate sealed 600-item TypeSafe-20 image bank (30 items per use case, frozen 10 Oct 2026). Not part of the v0.3 score or rank. Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), both systems: the same 600 sealed TypeSafe-20 items; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.
What each category means · items per category
  • Model routing — deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 30 items (0 public / 30 sealed)
  • Legal & compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 30 items (0 public / 30 sealed)
  • Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 30 items (0 public / 30 sealed)
  • Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 30 items (0 public / 30 sealed)
  • E-commerce — product listings, product attributes, orders and deliveries, seller catalog policy. 30 items (0 public / 30 sealed)
  • Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 30 items (0 public / 30 sealed)
  • Insurance claims — first-notice-of-loss reports, claim complexity, missing information, claim triage. 30 items (0 public / 30 sealed)
  • Financial crime — suspicious transactions, KYC, fraud alerts, entity matching for investigations. 30 items (0 public / 30 sealed)
  • Moderation — user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 30 items (0 public / 30 sealed)
  • Feature extraction — turning natural-language data into probabilistic features or estimates for a downstream prediction. 30 items (0 public / 30 sealed)
  • Knowledge graphs — entity types, relationships between records, contradictions between facts, multi-step lookups across linked facts. 30 items (0 public / 30 sealed)
  • LLM guardrails — checking LLM inputs, outputs and tool calls: prompt injection, jailbreaks, policy violations, tool-call errors, answer quality. 30 items (0 public / 30 sealed)
  • Search & retrieval — scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 30 items (0 public / 30 sealed)
  • Lead generation — matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 30 items (0 public / 30 sealed)
  • Code linting — checking code or writing against conventions and guidelines, as in CI review of code. 30 items (0 public / 30 sealed)
  • Demand forecasting — purchase intent, product interest, demand and supply signals for forecasting. 30 items (0 public / 30 sealed)
  • Gaming — player reports, in-game chat, game support. 30 items (0 public / 30 sealed)
  • Science — screening papers, labelling research passages or survey answers, checking that citations support claims, methodology checks. 30 items (0 public / 30 sealed)
  • Recruiting — resumes, applications, interview feedback, matching candidates to roles. 30 items (0 public / 30 sealed)
  • Advertising — ad creatives, campaign copy, brand safety, prohibited claims in ads. 30 items (0 public / 30 sealed)
How the categories were made

Chance-corrected accuracy (0 = chance, 100 = perfect) over the category's items, public and sealed pooled. Not part of the score; compare systems within a category.

Image types are the item source families (fixed at generation, no model). Use cases are assigned per item family at generation time.

  • Errors count as wrong.
  • Hosted APIs answer the A+P set (600 items), self-hosted systems S+P (1,500).
  • Each category cell records competence, scored items and completed supported responses. At least 30 completed responses are required to plot; errors remain counted wrong in the scored denominator.
All values as a table
SpokeA: Wity-1B: Imajev-4B
The four score axes
Intelligence68.749.9
Calibration79.486.1
Speed66.386.3
Cost71.851.8
Capability by image type
UI screenshots100.099.0
Small text58.510.3
Spatial24.140.8
Diagrams76.277.8
Receipts90.980.3
Document photos57.824.8
Misleading charts75.758.0
Multi-panel comparison81.955.1
Counting58.111.6
Dense UIsn=2367.2
Agent safetyn=2681.4
Chart reasoning100.086.7
Multilingualn=14100.0
Use cases (ImageJev v0.3 decision use cases)
Document checks69.341.6
Diagrams and planning49.758.1
Chart decisions84.962.9
UI agent actions92.489.3
Counting and inspection64.522.4
Use cases (TypeSafe categories)
Model routing86.773.3
Legal & compliance86.764.4
Customer support86.777.8
Other86.777.8
E-commerce82.277.8
Risk assessment100.077.8
Insurance claims91.177.8
Financial crime91.168.9
Moderation82.277.8
Feature extraction82.295.6
Knowledge graphs95.6100.0
LLM guardrails95.682.2
Search & retrieval91.186.7
Lead generation95.686.7
Code linting95.677.8
Demand forecasting95.677.8
Gaming86.773.3
Science100.091.1
Recruiting100.077.8
Advertising95.664.4

Jev REFERENCE · TypeSafe

Jev 1.13.0 defines the Jev-class cost and latency reference. Its native API accepts text only, so it has no native image score and is not ranked here. On the TypeSafe-20 use-case radar it appears as an image-to-text reference: a fixed offline Qwen3-VL-4B caption of each image (fallible) followed by Jev 1.13.0 on the question and options — not a native vision result. Use “Compare with Jev” in the comparison below. Of its 600 items, 9 count as wrong: 7 captions hit the 2,048-token cap and 2 Jev responses whose probabilities did not sum to 1 were rejected by the pre-registered strict parser. Compare its measured text results.

Measured on v0.3.0 — core track

SystemRankCapabilityJev-class (within caps)CompositeStatusScore artifact date
Wity-1 (AlphaNimble), reasoning autoAlphaNimble · Licence not reported#174.09No71.23Ranked2026-10-07
Imajev-4BAuthor not reported · Licence not reported#268.01Yes63.71Ranked2026-10-06
Surogate Rune 26B-A4B v3surogate (model repository owner) · Licence not reported#364.81Yes42.46Ranked2026-10-06
Autoloops – Gemma 4 31B ITHosted API provider · Licence not reported#468.40No41.48Ranked2026-10-07
imajev 9Bmohit67890 (model repository owner) · Licence not reported#567.07No34.52Ranked2026-10-06
H2O-Lightning-4B (image)H2O.ai · Apache-2.0 · 4B#660.43No34.14Ranked2026-10-12
Wity-1 (AlphaNimble), reasoning offAlphaNimble · Licence not reported#756.21No33.21Ranked2026-10-07
Jev-OmniAuthor not reported · Licence not reported#854.25Yes32.84Ranked2026-10-06
JPT-9B (kirp / llm2jev)kirp (model repository owner) · Licence not reported#958.72No31.43Ranked2026-10-06
NeoHorse Jev 4BTokenRhythm (model repository owner) · Licence not reported#1057.62Yes30.45Ranked2026-10-07
Jevify Gemma 4 26B-A4Bkushalpatil (model repository owner) · Licence not reported#1158.73Yes29.68Ranked2026-10-06
AutoJev-27Bdenis-pplx (model repository owner) · Licence not reported#1262.15Yes29.34Ranked2026-10-06
JPT-4B (kirp / llm2jev)kirp (model repository owner) · Licence not reported#1355.62Yes26.55Ranked2026-10-06
Jev-Vision 8B (SeanLiu)SeanLiu (model repository owner) · Licence not reported#1437.76Yes25.51Ranked2026-10-06
System1 s1-vision (System1 Models)Hosted API provider · Licence not reported#1558.32No24.02Ranked2026-10-06
shisa-de-1shisa-ai (model repository owner) · Licence not reported#1651.46Yes22.83Ranked2026-10-06
Clef-Flash (Cloudflare)Author not reported · Licence not reported#1752.24No20.41Ranked2026-10-06
Clef (Cloudflare)Author not reported · Licence not reported#1858.08No18.65Ranked2026-10-06
GPT-6 Luna via OpenAI Decisions (OpenAI)OpenAI · Licence not reported#1962.92No17.36Ranked2026-10-06
Winnow-12B (Q8_0 + F16 vision projector)Author not reported · Licence not reported#2049.99No17.08Ranked2026-10-06
Image Hopper (HopitAI)Author not reported · Licence not reported#2151.68No15.30Ranked2026-10-06
OmniJev-Qwen3.5-9B-v4 (tzcfly)tzcfly (model repository owner) · Licence not reported#2233.14Yes14.99Ranked2026-10-06
OmniJev 4B (tinnel123)tinnel123 (model repository owner) · Licence not reported#2347.21Yes14.83Ranked2026-10-06
Reflex 4B (released stable configuration)Author not reported · Licence not reported#2453.34Yes13.80Ranked2026-10-06
CUA-S1 4Bcua-ai (model repository owner) · Licence not reported#2549.35No12.71Ranked2026-10-07
GPT-6 Luna (low reasoning effort)Hosted API provider · Licence not reported#2674.50No12.19Ranked2026-10-07
Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjAuthor not reported · Licence not reported#2763.07No11.98Ranked2026-10-06
Visual-Jev generic (zero-shot Qwen3.5-4B baseline)Author not reported · Licence not reported#2856.73No11.14Ranked2026-10-06
Mapika decider-2b-vision BF16Author not reported · Licence not reported#2942.84Yes9.52Ranked2026-10-06
vjev-vision (yah01)yah01 (model repository owner) · Licence not reported#3055.66Yes8.86Ranked2026-10-06
Visual-Jev 4B Answer-SFTguanxuyu (model repository owner) · Licence not reported#3135.89Yes8.71Ranked2026-10-06
GPT-5.6 LunaHosted API provider · Licence not reported#3276.88No7.87Ranked2026-10-07
Jevify Gemma 4 E4Bkushalpatil (model repository owner) · Licence not reported#3343.71Yes7.26Ranked2026-10-06
JPT-0.8B (kirp / llm2jev)kirp (model repository owner) · Licence not reported#3446.07Yes7.17Ranked2026-10-06
Glance (frozen Qwen3-VL-4B)Author not reported · Licence not reported#3528.43Yes6.85Ranked2026-10-06
Gemini 3.1 Flash LiteHosted API provider · Licence not reported#3646.89No6.47Ranked2026-10-07
diffusiongemma-26b djev v10 step160 (snowicarus)snowicarus (model repository owner) · Licence not reported#3736.74No5.90Ranked2026-10-06
imajev 2Bmohit67890 (model repository owner) · Licence not reported#3846.42Yes5.76Ranked2026-10-06
Gevva E4B (text checkpoint, image input)davidburhans (model repository owner) · Licence not reported#3949.58Yes4.14Ranked2026-10-06
Liquid d1-3B (Liquid AI)Liquid AI · LFM Open License v1.0 · 3.1B#4047.78Yes4.10Ranked2026-10-10
djev-distill-v4 (tarsur385)tarsur385 (model repository owner) · Licence not reported#4140.25No3.62Ranked2026-10-06
Jevify Qwen3-VL-2B T2Praveenrajus (model repository owner) · Licence not reported#4243.93Yes3.29Ranked2026-10-06
OmniJev 2B (tinnel123)tinnel123 (model repository owner) · Licence not reported#4337.93Yes3.17Ranked2026-10-06
jev-spatial (Fr0zencr4nE)Fr0zencr4nE (model repository owner) · Licence not reported#4427.38Yes2.48Ranked2026-10-07
Gevva E2B multimodaldavidburhans (model repository owner) · Licence not reported#4547.34Yes2.02Ranked2026-10-06
JEVision (divyanshx11, visual route)Author not reported · Licence not reported#4616.20No1.84Ranked2026-10-06
OmniJev 0.8B (tinnel123)tinnel123 (model repository owner) · Licence not reported#4738.58Yes1.57Ranked2026-10-06
PlayJev 0.8BOmniJev (model repository owner) · Licence not reported#4825.60Yes0.50Ranked2026-10-06
Liquid d1-omni-600M (Liquid AI)Liquid AI · LFM Open License v1.0 · 587M#4926.54Yes0.24Ranked2026-10-10
djev-dev BF16Author not reported · Licence not reported#5025.41No0.04Ranked2026-10-06
Qevi-2B (MeerDevelopment)MeerDevelopment (model repository owner) · Licence not reported#5111.15Yes0.03Ranked2026-10-06
djev-spark NVFP4Author not reported · Licence not reported#5224.52Yes0.00Ranked2026-10-06
OpenJev 4B NLI v2 (official image-premise path)Author not reported · Licence not reported#538.20No0.00Ranked2026-10-06
OpenJev 4B NLI v5AlexWortega (model repository owner) · Licence not reported#5416.41No0.00Ranked2026-10-06
Gemini 3.8 FlashHosted API provider · Licence not reported#5519.40No0.00Ranked2026-10-07
Standard One 3BStandardThinking (model repository owner) · Licence not reported—49.55No—Measured but unrankablemeasured-unrankable: CORE cost unavailable (runtime returns no GPU-seconds), so no composite2026-10-06
Standard One 8BStandardThinking (model repository owner) · Licence not reported—51.55No—Measured but unrankablemeasured-unrankable: CORE cost unavailable (runtime returns no GPU-seconds), so no composite2026-10-06

Capability here is the raw mean of Intelligence and Calibration. Only ranked rows inside the cost and latency caps (Jev-class) receive a Capability rank in the chart above; Rank is the official composite rank.

Per-type, P/S/A and hard-slice aggregates
Wity-1 (AlphaNimble), reasoning auto
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 181,
        "accuracy": 0.7702127659574468,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 66.05196982397318,
        "ece": 0.10010248398124749,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 73.01573651386184
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 191,
        "accuracy": 0.8162393162393162,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 72.87636669470143,
        "ece": 0.09101040359860688,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 77.33714298749003
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 190,
        "accuracy": 0.8878504672897196,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 84.19492920645374,
        "ece": 0.08762237265331527,
        "probability_coverage": 1,
        "valid_answer_count": 214
      },
      "score": 83.33522733789533
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 83,
        "accuracy": 0.7685185185185185,
        "chance": 0.5,
        "chance_corrected_intelligence": 53.703703703703695,
        "ece": 0.19078099761612172,
        "probability_coverage": 1,
        "valid_answer_count": 108
      },
      "score": 57.773752090239675
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 99,
        "accuracy": 0.673469387755102,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 57.04697986577181,
        "ece": 0.11040303422404674,
        "probability_coverage": 1,
        "valid_answer_count": 147
      },
      "score": 67.48318651048123
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 159,
        "accuracy": 0.7718446601941747,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 65.9461417703176,
        "ece": 0.12256330386049791,
        "probability_coverage": 1,
        "valid_answer_count": 206
      },
      "score": 70.71674049910901
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 213,
        "accuracy": 0.8098859315589354,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 72.15777262180976,
        "ece": 0.10466502917850407,
        "probability_coverage": 1,
        "valid_answer_count": 263
      },
      "score": 75.61238339305447
    }
  }
}
Imajev-4B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 157,
        "accuracy": 0.6680851063829787,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 50.9639564124057,
        "ece": 0.044296675137570955,
        "probability_coverage": 0.9787234042553191,
        "valid_answer_count": 230
      },
      "score": 70.08272915018527
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 619,
        "accuracy": 0.6592119275825347,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 49.345715491768686,
        "ece": 0.07003359848566827,
        "probability_coverage": 0.9818956336528222,
        "valid_answer_count": 922
      },
      "score": 66.8910709723182
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 165,
        "accuracy": 0.7051282051282052,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 56.47603027754417,
        "ece": 0.08475386241537297,
        "probability_coverage": 0.9786324786324786,
        "valid_answer_count": 229
      },
      "score": 68.87535082547276
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 365,
        "accuracy": 0.6709558823529411,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 52.98135014447071,
        "ece": 0.10378565703495901,
        "probability_coverage": 0.9761029411764706,
        "valid_answer_count": 531
      },
      "score": 65.16527362268329
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 196,
        "accuracy": 0.7396226415094339,
        "chance": 0.5,
        "chance_corrected_intelligence": 47.92452830188678,
        "ece": 0.0316432694855534,
        "probability_coverage": 0.9735849056603774,
        "valid_answer_count": 258
      },
      "score": 69.56076848027442
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 215,
        "accuracy": 0.589041095890411,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 45.965417867435164,
        "ece": 0.0664569542691658,
        "probability_coverage": 0.9945205479452055,
        "valid_answer_count": 363
      },
      "score": 66.09945567352383
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 274,
        "accuracy": 0.5299806576402321,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 29.32279800281157,
        "ece": 0.13117475327152242,
        "probability_coverage": 0.9806576402321083,
        "valid_answer_count": 507
      },
      "score": 50.83052861288318
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 502,
        "accuracy": 0.7640791476407914,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 65.32309183787612,
        "ece": 0.027749444943461647,
        "probability_coverage": 0.9817351598173516,
        "valid_answer_count": 645
      },
      "score": 79.02404333316443
    }
  }
}
Surogate Rune 26B-A4B v3
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 145,
        "accuracy": 0.6170212765957447,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 43.41994970662196,
        "ece": 0.09283480841119661,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 62.42649401219132
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 595,
        "accuracy": 0.6336528221512248,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 45.546644153651336,
        "ece": 0.06683982261469067,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 66.0893398153566
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 155,
        "accuracy": 0.6623931623931624,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 50.16820857863752,
        "ece": 0.06087556406395757,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 68.99654788292301
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 386,
        "accuracy": 0.7095588235294118,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 58.49750459679539,
        "ece": 0.03829496576585914,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 75.41925572181178
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 205,
        "accuracy": 0.7735849056603774,
        "chance": 0.5,
        "chance_corrected_intelligence": 54.71698113207548,
        "ece": 0.04181989887393105,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 73.17650067864463
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 149,
        "accuracy": 0.40821917808219177,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 22.190201729106626,
        "ece": 0.16336562614003902,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 44.75853825054941
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 288,
        "accuracy": 0.5570599613152805,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 33.3947355664356,
        "ece": 0.060345944874397664,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 60.66277329577804
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 452,
        "accuracy": 0.6879756468797564,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 54.13699243073938,
        "ece": 0.08961941812489219,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 68.10655440288048
    }
  }
}
Autoloops – Gemma 4 31B IT
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 158,
        "accuracy": 0.6723404255319149,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 51.592623637887684,
        "ece": 0.05819260909112196,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 69.97705090983165
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 157,
        "accuracy": 0.6709401709401709,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 51.42977291841884,
        "ece": 0.05484318415892469,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 70.23056804331695
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 163,
        "accuracy": 0.7616822429906542,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 66.4142245637142,
        "ece": 0.07264279817488485,
        "probability_coverage": 1,
        "valid_answer_count": 214
      },
      "score": 75.94283246436862
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 85,
        "accuracy": 0.7870370370370371,
        "chance": 0.5,
        "chance_corrected_intelligence": 57.40740740740742,
        "ece": 0.05920992050321712,
        "probability_coverage": 1,
        "valid_answer_count": 108
      },
      "score": 72.782711653382
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 67,
        "accuracy": 0.4557823129251701,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 28.41163310961969,
        "ece": 0.11579413121942898,
        "probability_coverage": 1,
        "valid_answer_count": 147
      },
      "score": 52.626403432866944
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 123,
        "accuracy": 0.5970873786407767,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 39.86233546673107,
        "ece": 0.07306402692456503,
        "probability_coverage": 1,
        "valid_answer_count": 206
      },
      "score": 62.62476504090903
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 192,
        "accuracy": 0.7300380228136882,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 60.46403712296984,
        "ece": 0.08528870574049323,
        "probability_coverage": 1,
        "valid_answer_count": 263
      },
      "score": 71.7031479874356
    }
  }
}
imajev 9B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 140,
        "accuracy": 0.5957446808510638,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 40.276613579212075,
        "ece": 0.10136766591375182,
        "probability_coverage": 0.9531914893617022,
        "valid_answer_count": 224
      },
      "score": 58.13560161314629
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 640,
        "accuracy": 0.6815761448349308,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 52.66990291262135,
        "ece": 0.02949635316566651,
        "probability_coverage": 0.9680511182108626,
        "valid_answer_count": 909
      },
      "score": 71.8821096003372
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 167,
        "accuracy": 0.7136752136752137,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 57.73759461732549,
        "ece": 0.054456238728709416,
        "probability_coverage": 0.9743589743589743,
        "valid_answer_count": 228
      },
      "score": 72.2807535350962
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 370,
        "accuracy": 0.6801470588235294,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 54.29472025216706,
        "ece": 0.05839794167263383,
        "probability_coverage": 0.9503676470588235,
        "valid_answer_count": 517
      },
      "score": 69.11579103697476
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 191,
        "accuracy": 0.720754716981132,
        "chance": 0.5,
        "chance_corrected_intelligence": 44.1509433962264,
        "ece": 0.03794290781641962,
        "probability_coverage": 0.9547169811320755,
        "valid_answer_count": 253
      },
      "score": 66.1888469141305
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 219,
        "accuracy": 0.6,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 47.40634005763688,
        "ece": 0.08500762270355994,
        "probability_coverage": 0.9945205479452055,
        "valid_answer_count": 363
      },
      "score": 64.97501467501235
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 291,
        "accuracy": 0.562862669245648,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 34.26729361578361,
        "ece": 0.06102816237338186,
        "probability_coverage": 0.97678916827853,
        "valid_answer_count": 505
      },
      "score": 60.01194042519202
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 489,
        "accuracy": 0.7442922374429224,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 62.41470599202058,
        "ece": 0.04651388002728381,
        "probability_coverage": 0.9558599695585996,
        "valid_answer_count": 628
      },
      "score": 74.55427587924709
    }
  }
}
H2O-Lightning-4B (image)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 145,
        "accuracy": 0.6170212765957447,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 43.41994970662196,
        "ece": 0.12287550108276513,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 59.42242474503447
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 596,
        "accuracy": 0.6347177848775293,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 45.70493879273955,
        "ece": 0.118783989947551,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 60.97407040161467
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 155,
        "accuracy": 0.6623931623931624,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 50.16820857863752,
        "ece": 0.11516890534077222,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 63.56721375524154
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 372,
        "accuracy": 0.6838235294117647,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 54.820068295245605,
        "ece": 0.12905843268055248,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 64.50419087956755
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 210,
        "accuracy": 0.7924528301886793,
        "chance": 0.5,
        "chance_corrected_intelligence": 58.49056603773586,
        "ece": 0.021797356097571163,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 77.06554740911082
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 159,
        "accuracy": 0.43561643835616437,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 25.79250720461095,
        "ece": 0.1921764391501565,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 43.67860968728983
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 252,
        "accuracy": 0.4874274661508704,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 22.924038974259535,
        "ece": 0.19849712151272136,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 41.61230733585763
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 489,
        "accuracy": 0.7442922374429224,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 62.41470599202058,
        "ece": 0.0539456817250682,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 75.81278482350346
    }
  }
}
Wity-1 (AlphaNimble), reasoning off
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 134,
        "accuracy": 0.5702127659574469,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 36.50461022632021,
        "ece": 0.12313833000875128,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 55.93847211228498
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 140,
        "accuracy": 0.5982905982905983,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 40.70647603027755,
        "ece": 0.11314777109886787,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 59.038460905251995
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 156,
        "accuracy": 0.7289719626168224,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 61.804412248929864,
        "ece": 0.07561309967062195,
        "probability_coverage": 1,
        "valid_answer_count": 214
      },
      "score": 73.34089615740274
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 76,
        "accuracy": 0.7037037037037037,
        "chance": 0.5,
        "chance_corrected_intelligence": 40.74074074074075,
        "ece": 0.1999922802610235,
        "probability_coverage": 1,
        "valid_answer_count": 108
      },
      "score": 50.37114234426802
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 42,
        "accuracy": 0.2857142857142857,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 6.040268456375839,
        "ece": 0.19454272187863678,
        "probability_coverage": 1,
        "valid_answer_count": 147
      },
      "score": 33.56586204032424
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 111,
        "accuracy": 0.5388349514563107,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 31.167733365535565,
        "ece": 0.13259544459733252,
        "probability_coverage": 1,
        "valid_answer_count": 206
      },
      "score": 52.324322223034535
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 163,
        "accuracy": 0.6197718631178707,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 44.31554524361949,
        "ece": 0.11175354424574022,
        "probability_coverage": 1,
        "valid_answer_count": 263
      },
      "score": 60.98241819723572
    }
  }
}
Jev-Omni
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 141,
        "accuracy": 0.6,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 40.905280804694044,
        "ece": 0.1349376966953142,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 56.958870732815605
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 533,
        "accuracy": 0.5676251331203408,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 35.73237653018151,
        "ece": 0.15074642645133382,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 52.79154561995737
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 135,
        "accuracy": 0.5769230769230769,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 37.55256518082422,
        "ece": 0.15954830015626642,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 52.82145257478547
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 380,
        "accuracy": 0.6985294117647058,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 56.92146046755976,
        "ece": 0.08980997458550195,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 69.47973277522969
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 192,
        "accuracy": 0.7245283018867924,
        "chance": 0.5,
        "chance_corrected_intelligence": 44.90566037735848,
        "ece": 0.12433795753404436,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 60.019034435274804
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 102,
        "accuracy": 0.27945205479452057,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 5.259365994236313,
        "ece": 0.2514512859607847,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 27.484554401039684
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 235,
        "accuracy": 0.45454545454545453,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 17.979543361287508,
        "ece": 0.2035371757086256,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 38.63605410978119
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 439,
        "accuracy": 0.6681887366818874,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 51.22860658488385,
        "ece": 0.10355027902563445,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 65.25927538987847
    }
  }
}
JPT-9B (kirp / llm2jev)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 145,
        "accuracy": 0.6170212765957447,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 43.41994970662196,
        "ece": 0.14138300295956607,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 57.57167455735438
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 590,
        "accuracy": 0.6283280085197018,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 44.755170958210215,
        "ece": 0.13033071377052638,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 59.344514102052464
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 154,
        "accuracy": 0.6581196581196581,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 49.53742640874685,
        "ece": 0.1266157091924081,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 62.10714228513262
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 394,
        "accuracy": 0.7242647058823529,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 60.59889676910953,
        "ece": 0.09713303151470204,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 70.58614523308455
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 203,
        "accuracy": 0.7660377358490567,
        "chance": 0.5,
        "chance_corrected_intelligence": 53.207547169811335,
        "ece": 0.10840379580820927,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 65.76339400408474
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 138,
        "accuracy": 0.3780821917808219,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 18.227665706051873,
        "ece": 0.20730931208417888,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 38.38290164460804
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 255,
        "accuracy": 0.4932301740812379,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 23.796597023607546,
        "ece": 0.19675774399048412,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 42.222524112755366
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 480,
        "accuracy": 0.730593607305936,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 60.401208098735964,
        "ece": 0.0820118604002161,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 71.99941800934637
    }
  }
}
NeoHorse Jev 4B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 134,
        "accuracy": 0.5702127659574469,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 36.50461022632021,
        "ece": 0.11434122084171122,
        "probability_coverage": 0.9787234042553191,
        "valid_answer_count": 230
      },
      "score": 55.99763243503518
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 541,
        "accuracy": 0.5761448349307774,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 36.99873364288729,
        "ece": 0.09354291351118535,
        "probability_coverage": 0.9840255591054313,
        "valid_answer_count": 924
      },
      "score": 58.49578299989569
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 145,
        "accuracy": 0.6196581196581197,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 43.860386879730875,
        "ece": 0.10343584275636183,
        "probability_coverage": 0.9829059829059829,
        "valid_answer_count": 230
      },
      "score": 60.908721715949525
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 347,
        "accuracy": 0.6378676470588235,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 48.253217756763846,
        "ece": 0.08270591720273295,
        "probability_coverage": 0.9779411764705882,
        "valid_answer_count": 532
      },
      "score": 64.93551550487936
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 183,
        "accuracy": 0.690566037735849,
        "chance": 0.5,
        "chance_corrected_intelligence": 38.1132075471698,
        "ece": 0.09073752173737282,
        "probability_coverage": 0.9735849056603774,
        "valid_answer_count": 258
      },
      "score": 58.901780902550115
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 145,
        "accuracy": 0.3972602739726027,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 20.749279538904897,
        "ece": 0.13516536770307022,
        "probability_coverage": 0.9972602739726028,
        "valid_answer_count": 364
      },
      "score": 46.75814830536545
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 221,
        "accuracy": 0.4274661508704062,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 13.90760579766348,
        "ece": 0.17411708958247613,
        "probability_coverage": 0.9825918762088974,
        "valid_answer_count": 508
      },
      "score": 38.974792935988816
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 454,
        "accuracy": 0.6910197869101978,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 54.58443640702486,
        "ece": 0.05247732904233688,
        "probability_coverage": 0.9832572298325722,
        "valid_answer_count": 646
      },
      "score": 71.29520837682298
    }
  }
}
Jevify Gemma 4 26B-A4B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 143,
        "accuracy": 0.6085106382978723,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 42.162615255658004,
        "ece": 0.10141101023680356,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 60.94020660414864
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 538,
        "accuracy": 0.5729499467518637,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 36.52384972562263,
        "ece": 0.10715990372950135,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 57.54593448986118
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 141,
        "accuracy": 0.6025641025641025,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 41.33725820016821,
        "ece": 0.09636927020695724,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 61.03170207938838
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 362,
        "accuracy": 0.6654411764705882,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 52.1933280798529,
        "ece": 0.04147387517423727,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 71.94927652250271
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 193,
        "accuracy": 0.7283018867924528,
        "chance": 0.5,
        "chance_corrected_intelligence": 45.66037735849056,
        "ece": 0.045196899223018575,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 68.31049875694342
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 126,
        "accuracy": 0.3452054794520548,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 13.904899135446689,
        "ece": 0.25074233164607873,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 31.878216403115474
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 250,
        "accuracy": 0.4835589941972921,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 22.342333608027538,
        "ece": 0.09921742731197511,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 51.249424072816254
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 431,
        "accuracy": 0.6560121765601218,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 49.43883067974197,
        "ece": 0.10628051461759516,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 64.09136387811148
    }
  }
}
AutoJev-27B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 142,
        "accuracy": 0.6042553191489362,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 41.53394803017603,
        "ece": 0.07635499333586909,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 63.1314746815011
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 531,
        "accuracy": 0.5654952076677316,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 35.41578725200507,
        "ece": 0.060893449057417864,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 61.61854872026075
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 141,
        "accuracy": 0.6025641025641025,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 41.33725820016821,
        "ece": 0.0864691813939783,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 62.02171096068627
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 337,
        "accuracy": 0.6194852941176471,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 45.62647754137116,
        "ece": 0.0688283694934306,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 65.93040182134251
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 190,
        "accuracy": 0.7169811320754716,
        "chance": 0.5,
        "chance_corrected_intelligence": 43.39622641509433,
        "ece": 0.046644015025554236,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 67.03371170499173
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 146,
        "accuracy": 0.4,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 21.109510086455334,
        "ece": 0.07506604795635971,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 53.04815024759169
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 213,
        "accuracy": 0.4119922630560928,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 11.580784332735465,
        "ece": 0.06846912225994096,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 48.943479940373635
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 460,
        "accuracy": 0.700152207001522,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 55.92676833588126,
        "ece": 0.06405252050730165,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 71.55813211721048
    }
  }
}
JPT-4B (kirp / llm2jev)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 134,
        "accuracy": 0.5702127659574469,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 36.50461022632021,
        "ece": 0.148162713850679,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 53.436033728092205
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 555,
        "accuracy": 0.5910543130990416,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 39.21485859012241,
        "ece": 0.1280903155078337,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 56.798397744277835
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 143,
        "accuracy": 0.6111111111111112,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 42.59882253994955,
        "ece": 0.130099440603731,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 58.289467209601675
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 363,
        "accuracy": 0.6672794117647058,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 52.45600210139217,
        "ece": 0.11438553761395456,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 64.78944728930063
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 192,
        "accuracy": 0.7245283018867924,
        "chance": 0.5,
        "chance_corrected_intelligence": 44.90566037735848,
        "ece": 0.09918254967928476,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 62.53457522075077
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 134,
        "accuracy": 0.36712328767123287,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 16.78674351585014,
        "ece": 0.18385819736196407,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 40.007552021728664
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 230,
        "accuracy": 0.4448742746615087,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 16.5252799457075,
        "ece": 0.20736585796262144,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 37.52605417659161
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 459,
        "accuracy": 0.6986301369863014,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 55.703046347738535,
        "ece": 0.07288720768659082,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 70.56280240521019
    }
  }
}
Jev-Vision 8B (SeanLiu)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 142,
        "accuracy": 0.6042553191489362,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 41.53394803017603,
        "ece": 0.3024513798430616,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 40.521836030781856
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 545,
        "accuracy": 0.5804046858359957,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 37.631912199240176,
        "ece": 0.3253724786478993,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 36.278708234830155
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 140,
        "accuracy": 0.5982905982905983,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 40.70647603027755,
        "ece": 0.31285156767232924,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 39.06808124790585
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 351,
        "accuracy": 0.6452205882352942,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 49.303913842920934,
        "ece": 0.29245438043473215,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 45.40651887798725
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 193,
        "accuracy": 0.7283018867924528,
        "chance": 0.5,
        "chance_corrected_intelligence": 45.66037735849056,
        "ece": 0.21739788845250194,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 51.09039983399508
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 143,
        "accuracy": 0.3917808219178082,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 20.028818443804035,
        "ece": 0.43793108886453846,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 16.22130033544817
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 240,
        "accuracy": 0.46421663442940037,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 19.433806776867513,
        "ece": 0.4311080960942475,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 16.606093779009008
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 447,
        "accuracy": 0.680365296803653,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 53.01838249002573,
        "ece": 0.23549300887504238,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 52.95989035750863
    }
  }
}
System1 s1-vision (System1 Models)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 138,
        "accuracy": 0.5872340425531914,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 39.019279128248115,
        "ece": 0.12915208855210783,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 56.59443070891328
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 149,
        "accuracy": 0.6367521367521367,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 46.38351555929352,
        "ece": 0.11264474072126997,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 61.92728370751976
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 146,
        "accuracy": 0.6822429906542056,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 55.21896608495225,
        "ece": 0.08370627606714216,
        "probability_coverage": 1,
        "valid_answer_count": 214
      },
      "score": 69.2388554357619
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 81,
        "accuracy": 0.75,
        "chance": 0.5,
        "chance_corrected_intelligence": 50,
        "ece": 0.09593533393545843,
        "probability_coverage": 1,
        "valid_answer_count": 108
      },
      "score": 65.40646660645416
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 60,
        "accuracy": 0.40816326530612246,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 22.14765100671141,
        "ece": 0.18865834502487094,
        "probability_coverage": 1,
        "valid_answer_count": 147
      },
      "score": 42.20799100086861
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 102,
        "accuracy": 0.49514563106796117,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 24.64678178963894,
        "ece": 0.1465063415935082,
        "probability_coverage": 1,
        "valid_answer_count": 206
      },
      "score": 47.67275673546865
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 185,
        "accuracy": 0.7034220532319392,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 56.5661252900232,
        "ece": 0.09099690753836427,
        "probability_coverage": 1,
        "valid_answer_count": 263
      },
      "score": 69.18337189117517
    }
  }
}
shisa-de-1
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 136,
        "accuracy": 0.5787234042553191,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 37.761944677284156,
        "ece": 0.15245747936857007,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 53.63522440178507
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 518,
        "accuracy": 0.5516506922257721,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 33.35795694385817,
        "ece": 0.16388559454120838,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 50.29041901780825
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 138,
        "accuracy": 0.5897435897435898,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 39.44491169049622,
        "ece": 0.14817272470417756,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 54.90518337483036
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 364,
        "accuracy": 0.6691176470588235,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 52.718676122931434,
        "ece": 0.10552071925505253,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 65.80726613596046
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 191,
        "accuracy": 0.720754716981132,
        "chance": 0.5,
        "chance_corrected_intelligence": 44.1509433962264,
        "ece": 0.059287557610070166,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 66.14671593710618
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 99,
        "accuracy": 0.27123287671232876,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 4.178674351585013,
        "ece": 0.32161894499052285,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 19.92744267674022
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 226,
        "accuracy": 0.437137330754352,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 15.361869213243489,
        "ece": 0.20785467389894935,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 36.89546721672681
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 428,
        "accuracy": 0.6514459665144596,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 48.76766471531376,
        "ece": 0.12812220306371214,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 61.57161205128566
    }
  }
}
Clef-Flash (Cloudflare)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 130,
        "accuracy": 0.5531914893617021,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 33.9899413243923,
        "ece": 0.18935043364259727,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 48.05992729793642
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 554,
        "accuracy": 0.5899893503727369,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 39.056563951034185,
        "ece": 0.15034294734273634,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 54.49398724124346
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 144,
        "accuracy": 0.6153846153846154,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 43.22960470984021,
        "ece": 0.1464618151511018,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 56.96862083980993
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 359,
        "accuracy": 0.6599264705882353,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 51.40530601523509,
        "ece": 0.14588411100066467,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 61.114241907551076
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 191,
        "accuracy": 0.720754716981132,
        "chance": 0.5,
        "chance_corrected_intelligence": 44.1509433962264,
        "ece": 0.08454489554323708,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 63.62098214378949
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 134,
        "accuracy": 0.36712328767123287,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 16.78674351585014,
        "ece": 0.23366180208724513,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 35.02719154920056
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 219,
        "accuracy": 0.42359767891682787,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 13.325900431431482,
        "ece": 0.2708918746778205,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 29.573762747933692
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 465,
        "accuracy": 0.7077625570776256,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 57.04537827659495,
        "ece": 0.0825864413162787,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 70.2640450066696
    }
  }
}
Clef (Cloudflare)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 147,
        "accuracy": 0.625531914893617,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 44.67728415758592,
        "ece": 0.18264565867953875,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 54.07407621083909
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 616,
        "accuracy": 0.6560170394036209,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 48.87083157450402,
        "ece": 0.14201346225641281,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 60.234069561610724
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 165,
        "accuracy": 0.7051282051282052,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 56.47603027754417,
        "ece": 0.1159138986804045,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 66.64662527073165
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 391,
        "accuracy": 0.71875,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 59.81087470449173,
        "ece": 0.15697348321070764,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 64.2080890311751
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 221,
        "accuracy": 0.8339622641509434,
        "chance": 0.5,
        "chance_corrected_intelligence": 66.7924528301887,
        "ece": 0.03207280289054784,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 80.18894612603957
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 151,
        "accuracy": 0.4136986301369863,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 22.910662824207495,
        "ece": 0.2458469663695455,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 36.8706347751492
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 274,
        "accuracy": 0.5299806576402321,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 29.32279800281157,
        "ece": 0.2201338842876369,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 42.64801057264209
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 489,
        "accuracy": 0.7442922374429224,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 62.41470599202058,
        "ece": 0.09507329173783088,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 71.70002382222721
    }
  }
}
GPT-6 Luna via OpenAI Decisions (OpenAI)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 146,
        "accuracy": 0.6212765957446809,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 44.04861693210395,
        "ece": 0.072991452991453,
        "probability_coverage": 0.9957446808510638,
        "valid_answer_count": 234
      },
      "score": 64.54345740222219
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 152,
        "accuracy": 0.6495726495726496,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 48.27586206896553,
        "ece": 0.08704347826086958,
        "probability_coverage": 0.9829059829059829,
        "valid_answer_count": 230
      },
      "score": 64.72767462422635
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 158,
        "accuracy": 0.7383177570093458,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 63.12150148172538,
        "ece": 0.05591549295774648,
        "probability_coverage": 0.9953271028037384,
        "valid_answer_count": 213
      },
      "score": 75.76168532030195
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 76,
        "accuracy": 0.7037037037037037,
        "chance": 0.5,
        "chance_corrected_intelligence": 40.74074074074075,
        "ece": 0.12583333333333338,
        "probability_coverage": 1,
        "valid_answer_count": 108
      },
      "score": 57.78703703703704
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 64,
        "accuracy": 0.43537414965986393,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 25.727069351230426,
        "ece": 0.1083916083916084,
        "probability_coverage": 0.9727891156462585,
        "valid_answer_count": 143
      },
      "score": 50.95877277085331
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 111,
        "accuracy": 0.5388349514563107,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 31.167733365535565,
        "ece": 0.10791262135922329,
        "probability_coverage": 1,
        "valid_answer_count": 206
      },
      "score": 54.79260454684545
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 187,
        "accuracy": 0.7110266159695817,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 57.67981438515081,
        "ece": 0.06709302325581394,
        "probability_coverage": 0.9809885931558935,
        "valid_answer_count": 258
      },
      "score": 71.30758780094042
    }
  }
}
Winnow-12B (Q8_0 + F16 vision projector)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 138,
        "accuracy": 0.5872340425531914,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 39.019279128248115,
        "ece": 0.1608181673812327,
        "probability_coverage": 0.8936170212765957,
        "valid_answer_count": 210
      },
      "score": 49.81950545771603
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 550,
        "accuracy": 0.5857294994675186,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 38.4233853946813,
        "ece": 0.1609150339328168,
        "probability_coverage": 0.9105431309904153,
        "valid_answer_count": 855
      },
      "score": 50.08684136479982
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 150,
        "accuracy": 0.6410256410256411,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 47.0142977291842,
        "ece": 0.11513661672642596,
        "probability_coverage": 0.905982905982906,
        "valid_answer_count": 212
      },
      "score": 58.37511350305266
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 378,
        "accuracy": 0.6948529411764706,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 56.39611242448122,
        "ece": 0.06948987749377529,
        "probability_coverage": 0.8897058823529411,
        "valid_answer_count": 484
      },
      "score": 66.50079505286794
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 192,
        "accuracy": 0.7245283018867924,
        "chance": 0.5,
        "chance_corrected_intelligence": 44.90566037735848,
        "ece": 0.07152559519769215,
        "probability_coverage": 0.8679245283018868,
        "valid_answer_count": 230
      },
      "score": 59.64117475642672
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 118,
        "accuracy": 0.3232876712328767,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 11.023054755043228,
        "ece": 0.35233632032800566,
        "probability_coverage": 0.9616438356164384,
        "valid_answer_count": 351
      },
      "score": 19.71151410762299
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 264,
        "accuracy": 0.5106382978723404,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 26.414271171651553,
        "ece": 0.15383200440811862,
        "probability_coverage": 0.8781431334622823,
        "valid_answer_count": 454
      },
      "score": 43.605640421167
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 424,
        "accuracy": 0.6453576864535768,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 47.87277676274282,
        "ece": 0.15943456406139353,
        "probability_coverage": 0.9299847792998478,
        "valid_answer_count": 611
      },
      "score": 55.60845555922354
    }
  }
}
Image Hopper (HopitAI)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 126,
        "accuracy": 0.5361702127659574,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 31.475272422464375,
        "ece": 0.14891922649266187,
        "probability_coverage": 0.9744680851063829,
        "valid_answer_count": 229
      },
      "score": 49.94933711896854
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 516,
        "accuracy": 0.549520766773163,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 33.041367665681726,
        "ece": 0.13398757959683916,
        "probability_coverage": 0.9861554845580405,
        "valid_answer_count": 926
      },
      "score": 52.61519941253489
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 136,
        "accuracy": 0.5811965811965812,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 38.1833473507149,
        "ece": 0.11982733480209704,
        "probability_coverage": 0.9957264957264957,
        "valid_answer_count": 233
      },
      "score": 56.94647324420847
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 353,
        "accuracy": 0.6488970588235294,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 49.82926188599948,
        "ece": 0.12845499810061323,
        "probability_coverage": 0.9742647058823529,
        "valid_answer_count": 530
      },
      "score": 61.11294914275618
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 191,
        "accuracy": 0.720754716981132,
        "chance": 0.5,
        "chance_corrected_intelligence": 44.1509433962264,
        "ece": 0.09539527461988673,
        "probability_coverage": 0.9811320754716981,
        "valid_answer_count": 260
      },
      "score": 61.772539093897905
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 98,
        "accuracy": 0.2684931506849315,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 3.818443804034582,
        "ece": 0.1938212455016596,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 32.52709735185133
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 240,
        "accuracy": 0.46421663442940037,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 19.433806776867513,
        "ece": 0.16988621092702358,
        "probability_coverage": 0.9961315280464217,
        "valid_answer_count": 515
      },
      "score": 42.60057870227957
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 402,
        "accuracy": 0.6118721461187214,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 42.95089302360266,
        "ece": 0.1042374961620795,
        "probability_coverage": 0.974124809741248,
        "valid_answer_count": 640
      },
      "score": 60.027653887184755
    }
  }
}
OmniJev-Qwen3.5-9B-v4 (tzcfly)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 127,
        "accuracy": 0.5404255319148936,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 32.103939647946355,
        "ece": 0.3048248791242336,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 35.569481911549815
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 480,
        "accuracy": 0.5111821086261981,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 27.342760658505696,
        "ece": 0.318347524563629,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 31.836627872889945
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 128,
        "accuracy": 0.5470085470085471,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 33.13708999158958,
        "ece": 0.2753376552463964,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 39.03477947115515
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 333,
        "accuracy": 0.6121323529411765,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 44.575781455214084,
        "ece": 0.2280463524119684,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 49.4832554864102
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 178,
        "accuracy": 0.6716981132075471,
        "chance": 0.5,
        "chance_corrected_intelligence": 34.33962264150943,
        "ece": 0.24655324827082714,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 42.514486493671996
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 96,
        "accuracy": 0.26301369863013696,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 3.097982708933713,
        "ece": 0.506307964089081,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 1.5489913544668565
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 197,
        "accuracy": 0.38104448742746616,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 6.927141402879443,
        "ece": 0.3975200652756423,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 13.71156417387549
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 410,
        "accuracy": 0.624048706240487,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 44.74066892874454,
        "ece": 0.25059055154381726,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 47.31127930999054
    }
  }
}
OmniJev 4B (tinnel123)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 124,
        "accuracy": 0.5276595744680851,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 30.217937971500422,
        "ece": 0.1490178273022555,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 50.20718625552466
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 477,
        "accuracy": 0.5079872204472844,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 26.867876741241037,
        "ece": 0.17835798416035692,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 45.59813995458482
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 129,
        "accuracy": 0.5512820512820513,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 33.76787216148025,
        "ece": 0.15485658657189144,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 51.39827742355098
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 327,
        "accuracy": 0.6011029411764706,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 42.99973732597846,
        "ece": 0.11477260298873732,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 60.0226083641155
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 167,
        "accuracy": 0.630188679245283,
        "chance": 0.5,
        "chance_corrected_intelligence": 26.0377358490566,
        "ece": 0.1367816763236196,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 49.34070029216634
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 107,
        "accuracy": 0.29315068493150687,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 7.060518731988474,
        "ece": 0.2840972869670721,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 25.120530669287028
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 203,
        "accuracy": 0.39264990328820115,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 8.672257501575451,
        "ece": 0.1968071929460114,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 34.65540945618659
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 398,
        "accuracy": 0.6057838660578386,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 42.05600507103172,
        "ece": 0.15295237023598476,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 55.732765511917385
    }
  }
}
Reflex 4B (released stable configuration)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 120,
        "accuracy": 0.5106382978723404,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 27.703269069572503,
        "ece": 0.10368627715632132,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 53.48300681915411
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 479,
        "accuracy": 0.5101171458998935,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 27.184466019417485,
        "ece": 0.10323502084592033,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 53.26873092511671
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 127,
        "accuracy": 0.5427350427350427,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 32.506307821698904,
        "ece": 0.09035315524000045,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 57.21783838684941
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 341,
        "accuracy": 0.6268382352941176,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 46.677173627528234,
        "ece": 0.0509539248429666,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 68.24319432946746
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 170,
        "accuracy": 0.6415094339622641,
        "chance": 0.5,
        "chance_corrected_intelligence": 28.301886792452823,
        "ece": 0.09939124150943399,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 54.21181924528301
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 88,
        "accuracy": 0.2410958904109589,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0.21613832853025922,
        "ece": 0.2127198605639564,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 28.836083107869488
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 225,
        "accuracy": 0.43520309477756286,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 15.071016530127489,
        "ece": 0.14620054328682164,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 42.91545393638158
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 374,
        "accuracy": 0.5692541856925418,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 36.68667735560609,
        "ece": 0.0852568902160156,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 59.81764965620149
    }
  }
}
CUA-S1 4B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 138,
        "accuracy": 0.5872340425531914,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 39.019279128248115,
        "ece": 0.1135755893362358,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 58.15208063050048
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 490,
        "accuracy": 0.5218317358892439,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 28.925707049387935,
        "ece": 0.1611280623519106,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 48.35004728950291
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 130,
        "accuracy": 0.5555555555555556,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 34.398654331370906,
        "ece": 0.14511780766383187,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 52.68754639930226
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 331,
        "accuracy": 0.6084558823529411,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 44.05043341213553,
        "ece": 0.12131415743251421,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 59.89380096281634
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 171,
        "accuracy": 0.6452830188679245,
        "chance": 0.5,
        "chance_corrected_intelligence": 29.0566037735849,
        "ece": 0.14811105827776846,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 49.7171960590156
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 126,
        "accuracy": 0.3452054794520548,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 13.904899135446689,
        "ece": 0.2153889760625243,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 35.413551961470915
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 222,
        "accuracy": 0.42940038684719534,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 14.198458480779482,
        "ece": 0.20750524882981847,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 36.3487043574079
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 406,
        "accuracy": 0.6179604261796042,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 43.8457809761736,
        "ece": 0.10860012441474448,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 61.062878046612354
    }
  }
}
GPT-6 Luna (low reasoning effort)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 196,
        "accuracy": 0.8340425531914893,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 75.48197820620287,
        "ece": 0.04280288461538462,
        "probability_coverage": 0.8851063829787233,
        "valid_answer_count": 208
      },
      "score": 78.20779761373973
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 192,
        "accuracy": 0.8205128205128205,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 73.5071488645921,
        "ece": 0.0698564593301435,
        "probability_coverage": 0.8931623931623932,
        "valid_answer_count": 209
      },
      "score": 75.17237785109947
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 187,
        "accuracy": 0.8738317757009346,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 82.21929535726046,
        "ece": 0.019840206185566922,
        "probability_coverage": 0.9065420560747663,
        "valid_answer_count": 194
      },
      "score": 84.63815235152744
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 96,
        "accuracy": 0.8888888888888888,
        "chance": 0.5,
        "chance_corrected_intelligence": 77.77777777777777,
        "ece": 0.03292000000000006,
        "probability_coverage": 0.9259259259259259,
        "valid_answer_count": 100
      },
      "score": 82.13703703703703
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 105,
        "accuracy": 0.7142857142857143,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 62.416107382550344,
        "ece": 0.13302439024390245,
        "probability_coverage": 0.8367346938775511,
        "valid_answer_count": 123
      },
      "score": 61.91417614025477
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 157,
        "accuracy": 0.7621359223300971,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 64.49704142011835,
        "ece": 0.0818381502890172,
        "probability_coverage": 0.8398058252427184,
        "valid_answer_count": 173
      },
      "score": 67.36599643821452
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 231,
        "accuracy": 0.8783269961977186,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 82.18097447795824,
        "ece": 0.03829918032786883,
        "probability_coverage": 0.9277566539923955,
        "valid_answer_count": 244
      },
      "score": 83.92508799943539
    }
  }
}
Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 150,
        "accuracy": 0.6382978723404256,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 46.56328583403186,
        "ece": 0.11441866480229579,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 61.83977643678635
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 569,
        "accuracy": 0.6059637912673056,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 41.43098353735753,
        "ece": 0.06983212677853859,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 63.7322790908249
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 151,
        "accuracy": 0.6452991452991453,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 47.645079899074865,
        "ece": 0.07596513771490963,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 66.22602617804647
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 361,
        "accuracy": 0.6636029411764706,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 51.930654058313635,
        "ece": 0.06616810948166797,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 69.34851608099002
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 181,
        "accuracy": 0.6830188679245283,
        "chance": 0.5,
        "chance_corrected_intelligence": 36.60377358490565,
        "ece": 0.13090612858185052,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 55.21127393426777
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 177,
        "accuracy": 0.4849315068493151,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 32.27665706051873,
        "ece": 0.0940207356498986,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 56.7362549652695
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 272,
        "accuracy": 0.5261121856866537,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 28.74109263657957,
        "ece": 0.14927932519816106,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 49.44261379847368
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 447,
        "accuracy": 0.680365296803653,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 53.01838249002573,
        "ece": 0.03511082043376302,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 72.99810920163657
    }
  }
}
Visual-Jev generic (zero-shot Qwen3.5-4B baseline)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 125,
        "accuracy": 0.5319148936170213,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 30.846605196982402,
        "ece": 0.06544607516660969,
        "probability_coverage": 0.9787234042553191,
        "valid_answer_count": 230
      },
      "score": 57.95411226303578
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 491,
        "accuracy": 0.5228966986155484,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 29.08400168847615,
        "ece": 0.0770472667222728,
        "probability_coverage": 0.9818956336528222,
        "valid_answer_count": 922
      },
      "score": 56.07154504893078
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 123,
        "accuracy": 0.5256410256410257,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 29.983179142136258,
        "ece": 0.135889497732809,
        "probability_coverage": 0.9786324786324786,
        "valid_answer_count": 229
      },
      "score": 50.624625904053914
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 331,
        "accuracy": 0.6084558823529411,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 44.05043341213553,
        "ece": 0.05236717708996926,
        "probability_coverage": 0.9761029411764706,
        "valid_answer_count": 531
      },
      "score": 65.71878820702848
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 175,
        "accuracy": 0.660377358490566,
        "chance": 0.5,
        "chance_corrected_intelligence": 32.0754716981132,
        "ece": 0.08373812092119856,
        "probability_coverage": 0.9735849056603774,
        "valid_answer_count": 258
      },
      "score": 56.564364076351225
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 110,
        "accuracy": 0.3013698630136986,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 8.141210374639765,
        "ece": 0.1250766785687342,
        "probability_coverage": 0.9945205479452055,
        "valid_answer_count": 363
      },
      "score": 41.35749989404577
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 224,
        "accuracy": 0.4332688588007737,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 14.78016384701149,
        "ece": 0.11143085047983342,
        "probability_coverage": 0.9806576402321083,
        "valid_answer_count": 507
      },
      "score": 45.49541244705013
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 392,
        "accuracy": 0.5966514459665144,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 40.71367314217532,
        "ece": 0.06485094917580016,
        "probability_coverage": 0.9817351598173516,
        "valid_answer_count": 645
      },
      "score": 63.07694886661413
    }
  }
}
Mapika decider-2b-vision BF16
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 115,
        "accuracy": 0.48936170212765956,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 24.559932942162614,
        "ece": 0.18612303206257308,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 43.667663264824
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 448,
        "accuracy": 0.47710330138445156,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 22.277332207682573,
        "ece": 0.18746185158798748,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 42.39248094504254
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 121,
        "accuracy": 0.5170940170940171,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 28.72161480235493,
        "ece": 0.17077569955756305,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 47.283237445421165
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 298,
        "accuracy": 0.5477941176470589,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 35.382190701339645,
        "ece": 0.18672977436146576,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 49.018117914523245
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 181,
        "accuracy": 0.6830188679245283,
        "chance": 0.5,
        "chance_corrected_intelligence": 36.60377358490565,
        "ece": 0.09077873902102784,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 59.22401289035004
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 84,
        "accuracy": 0.23013698630136986,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.2515004877334404,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 24.849951226655957
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 174,
        "accuracy": 0.3365570599613153,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0.23752969121140366,
        "ece": 0.28566393785331196,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 21.552371060274506
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 389,
        "accuracy": 0.5920852359208524,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 40.04250717774712,
        "ece": 0.10917114039508588,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 59.10413954936497
    }
  }
}
vjev-vision (yah01)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 107,
        "accuracy": 0.4553191489361702,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 19.53059513830679,
        "ece": 0.08119589623296827,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 51.645707945856564
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 452,
        "accuracy": 0.48136315228966986,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 22.910510764035457,
        "ece": 0.036279503139124925,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 57.82730506810523
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 121,
        "accuracy": 0.5170940170940171,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 28.72161480235493,
        "ece": 0.06781327713290602,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 57.57947968788687
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 304,
        "accuracy": 0.5588235294117647,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 36.95823483057526,
        "ece": 0.07345068417828138,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 61.13404899745949
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 153,
        "accuracy": 0.5773584905660377,
        "chance": 0.5,
        "chance_corrected_intelligence": 15.471698113207545,
        "ece": 0.0812839675402656,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 49.607452302577215
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 102,
        "accuracy": 0.27945205479452057,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 5.259365994236313,
        "ece": 0.13061746741540342,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 39.56793625557782
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 188,
        "accuracy": 0.36363636363636365,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 4.309467254835427,
        "ece": 0.08495738389218212,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 43.6589952381995
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 371,
        "accuracy": 0.5646879756468798,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 36.015511391177895,
        "ece": 0.04223881376275927,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 63.78387431931302
    }
  }
}
Visual-Jev 4B Answer-SFT
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 117,
        "accuracy": 0.4978723404255319,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 25.817267393126574,
        "ece": 0.26015413770290274,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 36.89321992627301
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 453,
        "accuracy": 0.48242811501597443,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 23.06880540312368,
        "ece": 0.26189514951119647,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 35.34488775044219
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 119,
        "accuracy": 0.5085470085470085,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 27.460050462573594,
        "ece": 0.2520822429255926,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 38.52180093872754
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 325,
        "accuracy": 0.5974264705882353,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 42.47438928289992,
        "ece": 0.195264905184342,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 51.71070412301576
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 165,
        "accuracy": 0.6226415094339622,
        "chance": 0.5,
        "chance_corrected_intelligence": 24.52830188679245,
        "ece": 0.23218990353024038,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 39.045160590372184
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 80,
        "accuracy": 0.2191780821917808,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.37087152322229344,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 12.912847677770657
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 200,
        "accuracy": 0.38684719535783363,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 7.799699452227442,
        "ece": 0.3015235702579428,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 23.74749270031944
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 370,
        "accuracy": 0.563165905631659,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 35.79178940303515,
        "ece": 0.22452856444895194,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 45.44303825662238
    }
  }
}
GPT-5.6 Luna
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 203,
        "accuracy": 0.8638297872340426,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 79.8826487845767,
        "ece": 0.05904090909090902,
        "probability_coverage": 0.9361702127659575,
        "valid_answer_count": 220
      },
      "score": 81.22260098803304
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 197,
        "accuracy": 0.8418803418803419,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 76.66105971404544,
        "ece": 0.09012162162162163,
        "probability_coverage": 0.9487179487179487,
        "valid_answer_count": 222
      },
      "score": 77.21642729292014
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 192,
        "accuracy": 0.897196261682243,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 85.51201843924926,
        "ece": 0.0323497536945814,
        "probability_coverage": 0.9485981308411215,
        "valid_answer_count": 203
      },
      "score": 87.11722417289565
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 102,
        "accuracy": 0.9444444444444444,
        "chance": 0.5,
        "chance_corrected_intelligence": 88.88888888888889,
        "ece": 0.03355140186915895,
        "probability_coverage": 0.9907407407407407,
        "valid_answer_count": 107
      },
      "score": 90.65740740740739
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 106,
        "accuracy": 0.7210884353741497,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 63.3109619686801,
        "ece": 0.17302272727272716,
        "probability_coverage": 0.8979591836734694,
        "valid_answer_count": 132
      },
      "score": 61.016705474135975
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 175,
        "accuracy": 0.8495145631067961,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 77.53894457191161,
        "ece": 0.056216931216931165,
        "probability_coverage": 0.9174757281553398,
        "valid_answer_count": 189
      },
      "score": 79.48549170343153
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 225,
        "accuracy": 0.8555133079847909,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 78.83990719257542,
        "ece": 0.08842292490118572,
        "probability_coverage": 0.9619771863117871,
        "valid_answer_count": 253
      },
      "score": 79.01272926168696
    }
  }
}
Jevify Gemma 4 E4B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 103,
        "accuracy": 0.43829787234042555,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 17.01592623637888,
        "ece": 0.19627238127225918,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 38.880724990963515
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 448,
        "accuracy": 0.47710330138445156,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 22.277332207682573,
        "ece": 0.1482267098004272,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 46.31599512379857
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 121,
        "accuracy": 0.5170940170940171,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 28.72161480235493,
        "ece": 0.10701840040318401,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 53.658967360859066
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 290,
        "accuracy": 0.5330882352941176,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 33.280798529025475,
        "ece": 0.08721363806837135,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 57.919035457675605
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 174,
        "accuracy": 0.6566037735849056,
        "chance": 0.5,
        "chance_corrected_intelligence": 31.320754716981124,
        "ece": 0.11867879450033744,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 53.792497908456816
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 87,
        "accuracy": 0.23835616438356164,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.29815260916295067,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 20.184739083704933
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 185,
        "accuracy": 0.3578336557059961,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 3.436909205487419,
        "ece": 0.21217823114960688,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 30.50063148778302
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 366,
        "accuracy": 0.5570776255707762,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 34.89690145046421,
        "ece": 0.11873777839640845,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 55.57467288559126
    }
  }
}
JPT-0.8B (kirp / llm2jev)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 109,
        "accuracy": 0.46382978723404256,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 20.78792958927075,
        "ece": 0.1511490219662798,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 45.27906259800739
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 435,
        "accuracy": 0.46325878594249204,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 20.219501899535675,
        "ece": 0.13616293199677554,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 46.49345775009028
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 113,
        "accuracy": 0.4829059829059829,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 23.675357443229608,
        "ece": 0.12880136775268347,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 48.95754194634645
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 278,
        "accuracy": 0.5110294117647058,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 30.128710270554233,
        "ece": 0.1283495232179103,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 52.22940281348609
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 175,
        "accuracy": 0.660377358490566,
        "chance": 0.5,
        "chance_corrected_intelligence": 32.0754716981132,
        "ece": 0.0382727517938426,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 62.210460669672344
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 91,
        "accuracy": 0.2493150684931507,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 1.2968299711815554,
        "ece": 0.23280748361989181,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 27.3676666236016
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 175,
        "accuracy": 0.33849129593810445,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0.5283823743274035,
        "ece": 0.2199508437973448,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 28.26910680742922
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 369,
        "accuracy": 0.5616438356164384,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 35.568067414892425,
        "ece": 0.07696853015942565,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 60.08718069150365
    }
  }
}
Glance (frozen Qwen3-VL-4B)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 110,
        "accuracy": 0.46808510638297873,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 21.41659681475273,
        "ece": 0.32060122618331927,
        "probability_coverage": 0.9744680851063829,
        "valid_answer_count": 229
      },
      "score": 28.19013636653376
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 441,
        "accuracy": 0.4696485623003195,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 21.169269734065008,
        "ece": 0.3177977716598246,
        "probability_coverage": 0.9861554845580405,
        "valid_answer_count": 926
      },
      "score": 28.552607544668547
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 111,
        "accuracy": 0.47435897435897434,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 22.413793103448278,
        "ece": 0.33456913538871835,
        "probability_coverage": 0.9957264957264957,
        "valid_answer_count": 233
      },
      "score": 27.679286062163722
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 293,
        "accuracy": 0.5386029411764706,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 34.06882059364329,
        "ece": 0.2523391105554481,
        "probability_coverage": 0.9742647058823529,
        "valid_answer_count": 530
      },
      "score": 41.163136658147465
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 164,
        "accuracy": 0.6188679245283019,
        "chance": 0.5,
        "chance_corrected_intelligence": 23.77358490566037,
        "ece": 0.26607613256718887,
        "probability_coverage": 0.9811320754716981,
        "valid_answer_count": 260
      },
      "score": 34.83781340850222
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 94,
        "accuracy": 0.25753424657534246,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 2.3775216138328514,
        "ece": 0.4446077882955125,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 6.727981977365175
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 204,
        "accuracy": 0.3945841392649903,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 8.96311018469145,
        "ece": 0.34151383236794364,
        "probability_coverage": 0.9961315280464217,
        "valid_answer_count": 515
      },
      "score": 20.26886192609989
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 347,
        "accuracy": 0.528158295281583,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 30.64618367575227,
        "ece": 0.2958152220628559,
        "probability_coverage": 0.974124809741248,
        "valid_answer_count": 640
      },
      "score": 35.213237633884084
    }
  }
}
Gemini 3.1 Flash Lite
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 138,
        "accuracy": 0.5872340425531914,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 39.019279128248115,
        "ece": 0.2958723404255319,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 39.922405521570866
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 158,
        "accuracy": 0.6752136752136753,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 52.06055508830951,
        "ece": 0.2259656652360516,
        "probability_coverage": 0.9957264957264957,
        "valid_answer_count": 233
      },
      "score": 53.31660233047953
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 161,
        "accuracy": 0.7523364485981309,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 65.09713533091868,
        "ece": 0.11492957746478871,
        "probability_coverage": 0.9953271028037384,
        "valid_answer_count": 213
      },
      "score": 70.87567046919766
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 81,
        "accuracy": 0.75,
        "chance": 0.5,
        "chance_corrected_intelligence": 50,
        "ece": 0.1893518518518518,
        "probability_coverage": 1,
        "valid_answer_count": 108
      },
      "score": 56.064814814814824
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 54,
        "accuracy": 0.3673469387755102,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 16.77852348993289,
        "ece": 0.5255102040816326,
        "probability_coverage": 1,
        "valid_answer_count": 147
      },
      "score": 8.389261744966445
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 111,
        "accuracy": 0.5388349514563107,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 31.167733365535565,
        "ece": 0.3373170731707317,
        "probability_coverage": 0.9951456310679612,
        "valid_answer_count": 205
      },
      "score": 31.7731870711173
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 185,
        "accuracy": 0.7034220532319392,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 56.5661252900232,
        "ece": 0.201634980988593,
        "probability_coverage": 1,
        "valid_answer_count": 263
      },
      "score": 58.119564546152304
    }
  }
}
diffusiongemma-26b djev v10 step160 (snowicarus)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 111,
        "accuracy": 0.4723404255319149,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 22.04526404023471,
        "ece": 0.24356072391289557,
        "probability_coverage": 0.9276595744680851,
        "valid_answer_count": 218
      },
      "score": 34.811466993304066
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 458,
        "accuracy": 0.48775292864749736,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 23.8602785985648,
        "ece": 0.2210191008428979,
        "probability_coverage": 0.9265175718849841,
        "valid_answer_count": 870
      },
      "score": 37.77820982821519
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 119,
        "accuracy": 0.5085470085470085,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 27.460050462573594,
        "ece": 0.22433964059918116,
        "probability_coverage": 0.9273504273504274,
        "valid_answer_count": 217
      },
      "score": 39.29340044067897
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 289,
        "accuracy": 0.53125,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 33.018124507486206,
        "ece": 0.18674073780941497,
        "probability_coverage": 0.9025735294117647,
        "valid_answer_count": 491
      },
      "score": 44.783014043371274
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 162,
        "accuracy": 0.6113207547169811,
        "chance": 0.5,
        "chance_corrected_intelligence": 22.26415094339622,
        "ece": 0.20558199351881942,
        "probability_coverage": 0.9132075471698113,
        "valid_answer_count": 242
      },
      "score": 38.01855002582856
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 118,
        "accuracy": 0.3232876712328767,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 11.023054755043228,
        "ece": 0.2990802229418895,
        "probability_coverage": 0.9726027397260274,
        "valid_answer_count": 355
      },
      "score": 25.0530399407077
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 218,
        "accuracy": 0.42166344294003866,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 13.035047748315474,
        "ece": 0.25614290402538187,
        "probability_coverage": 0.9342359767891683,
        "valid_answer_count": 483
      },
      "score": 29.299531099639466
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 351,
        "accuracy": 0.5342465753424658,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 31.541071628323202,
        "ece": 0.2020624013149196,
        "probability_coverage": 0.9208523592085236,
        "valid_answer_count": 605
      },
      "score": 43.206189878769464
    }
  }
}
imajev 2B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 104,
        "accuracy": 0.4425531914893617,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 17.644593461860854,
        "ece": 0.12564908504129654,
        "probability_coverage": 0.9531914893617022,
        "valid_answer_count": 224
      },
      "score": 44.50510734827067
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 437,
        "accuracy": 0.4653887113951012,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 20.536091177712116,
        "ece": 0.11591882797330631,
        "probability_coverage": 0.9680511182108626,
        "valid_answer_count": 909
      },
      "score": 47.449066395274
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 116,
        "accuracy": 0.49572649572649574,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 25.567703952901606,
        "ece": 0.1224382542550698,
        "probability_coverage": 0.9743589743589743,
        "valid_answer_count": 228
      },
      "score": 49.57191951057221
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 276,
        "accuracy": 0.5073529411764706,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 29.6033622274757,
        "ece": 0.11333090283343195,
        "probability_coverage": 0.9503676470588235,
        "valid_answer_count": 517
      },
      "score": 51.54946112019294
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 170,
        "accuracy": 0.6415094339622641,
        "chance": 0.5,
        "chance_corrected_intelligence": 28.301886792452823,
        "ece": 0.050924942608096456,
        "probability_coverage": 0.9547169811320755,
        "valid_answer_count": 253
      },
      "score": 57.02490170571758
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 95,
        "accuracy": 0.2602739726027397,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 2.7377521613832823,
        "ece": 0.19842156859130225,
        "probability_coverage": 0.9945205479452055,
        "valid_answer_count": 363
      },
      "score": 31.361470765995012
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 181,
        "accuracy": 0.35009671179883944,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 2.273498473023411,
        "ece": 0.1718872548306574,
        "probability_coverage": 0.97678916827853,
        "valid_answer_count": 505
      },
      "score": 33.18644678206645
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 360,
        "accuracy": 0.547945205479452,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 33.55456952160781,
        "ece": 0.08754769304030312,
        "probability_coverage": 0.9558599695585996,
        "valid_answer_count": 628
      },
      "score": 56.201949718290905
    }
  }
}
Gevva E4B (text checkpoint, image input)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 105,
        "accuracy": 0.44680851063829785,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 18.27326068734283,
        "ece": 0.09090887860149871,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 50.04574248352154
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 414,
        "accuracy": 0.44089456869009586,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 16.895314478682995,
        "ece": 0.09125416085271387,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 49.322241154070106
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 111,
        "accuracy": 0.47435897435897434,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 22.413793103448278,
        "ece": 0.08018412069307777,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 53.18848448241636
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 293,
        "accuracy": 0.5386029411764706,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 34.06882059364329,
        "ece": 0.13864432506404617,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 53.16997779041702
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 159,
        "accuracy": 0.6,
        "chance": 0.5,
        "chance_corrected_intelligence": 19.999999999999996,
        "ece": 0.04199572666962681,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 55.80042733303732
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 67,
        "accuracy": 0.18356164383561643,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.08962525876068889,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 41.03747412393111
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 179,
        "accuracy": 0.34622823984526113,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 1.6917931067914114,
        "ece": 0.06380145531132927,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 44.46575102226277
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 340,
        "accuracy": 0.517503805175038,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 29.080129758753117,
        "ece": 0.1550548243986309,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 49.034582439513464
    }
  }
}
Liquid d1-3B (Liquid AI)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 102,
        "accuracy": 0.4340425531914894,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 16.3872590108969,
        "ece": 0.10819401720685473,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 47.37422778476297
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 410,
        "accuracy": 0.4366347177848775,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 16.262135922330096,
        "ece": 0.10130706771325647,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 48.000361189839396
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 104,
        "accuracy": 0.4444444444444444,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 17.998317914213626,
        "ece": 0.13116097675551605,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 45.88306128155521
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 260,
        "accuracy": 0.47794117647058826,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 25.400577882847386,
        "ece": 0.09781239598701864,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 52.91904934272183
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 162,
        "accuracy": 0.6113207547169811,
        "chance": 0.5,
        "chance_corrected_intelligence": 22.26415094339622,
        "ece": 0.07284573919833091,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 53.847501551865015
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 90,
        "accuracy": 0.2465753424657534,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0.9365994236311208,
        "ece": 0.15310508720042765,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 35.1577909917728
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 181,
        "accuracy": 0.35009671179883944,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 2.273498473023411,
        "ece": 0.13216539294104107,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 37.920209942407595
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 331,
        "accuracy": 0.5038051750380518,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 27.066631865468516,
        "ece": 0.08334523707326155,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 55.1987922254081
    }
  }
}
djev-distill-v4 (tarsur385)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 105,
        "accuracy": 0.44680851063829785,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 18.27326068734283,
        "ece": 0.179371542997656,
        "probability_coverage": 0.9276595744680851,
        "valid_answer_count": 218
      },
      "score": 38.88003614218673
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 428,
        "accuracy": 0.45580404685835996,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 19.11143942591811,
        "ece": 0.16080294660888803,
        "probability_coverage": 0.9265175718849841,
        "valid_answer_count": 870
      },
      "score": 40.982922742806494
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 112,
        "accuracy": 0.47863247863247865,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 23.044575273338946,
        "ece": 0.16600690447434724,
        "probability_coverage": 0.9273504273504274,
        "valid_answer_count": 217
      },
      "score": 42.4951516234501
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 271,
        "accuracy": 0.49816176470588236,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 28.289992119779352,
        "ece": 0.14067235385033663,
        "probability_coverage": 0.9025735294117647,
        "valid_answer_count": 491
      },
      "score": 46.576958239942016
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 161,
        "accuracy": 0.6075471698113207,
        "chance": 0.5,
        "chance_corrected_intelligence": 21.509433962264147,
        "ece": 0.10424008503866007,
        "probability_coverage": 0.9132075471698113,
        "valid_answer_count": 242
      },
      "score": 46.89581110212991
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 101,
        "accuracy": 0.27671232876712326,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 4.899135446685874,
        "ece": 0.2482565141734521,
        "probability_coverage": 0.9726027397260274,
        "valid_answer_count": 355
      },
      "score": 26.934208125651026
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 204,
        "accuracy": 0.3945841392649903,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 8.96311018469145,
        "ece": 0.2056654799160515,
        "probability_coverage": 0.9342359767891683,
        "valid_answer_count": 483
      },
      "score": 31.979344879685595
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 329,
        "accuracy": 0.5007610350076104,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 26.619187889183046,
        "ece": 0.13729203067721807,
        "probability_coverage": 0.9208523592085236,
        "valid_answer_count": 605
      },
      "score": 46.70964287005317
    }
  }
}
Jevify Qwen3-VL-2B T2
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 101,
        "accuracy": 0.4297872340425532,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 15.758591785414923,
        "ece": 0.14295191489361703,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 43.584104403345755
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 400,
        "accuracy": 0.42598509052183176,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 14.679189531447873,
        "ece": 0.13219872204472843,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 44.119722561251095
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 104,
        "accuracy": 0.4444444444444444,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 17.998317914213626,
        "ece": 0.14781752136752133,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 44.21740682035468
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 259,
        "accuracy": 0.47610294117647056,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 25.13790386130811,
        "ece": 0.11881231617647056,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 50.68772031300699
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 163,
        "accuracy": 0.6150943396226415,
        "chance": 0.5,
        "chance_corrected_intelligence": 23.018867924528298,
        "ece": 0.05682113207547166,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 55.82732075471698
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 79,
        "accuracy": 0.21643835616438356,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.23426410958904106,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 26.57358904109589
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 171,
        "accuracy": 0.3307543520309478,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0,
        "ece": 0.19426344294003864,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 30.57365570599614
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 330,
        "accuracy": 0.502283105022831,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 26.842909877325773,
        "ece": 0.08720563165905632,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 54.70089177275726
    }
  }
}
OmniJev 2B (tinnel123)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 97,
        "accuracy": 0.4127659574468085,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 13.24392288348701,
        "ece": 0.2109065878545796,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 35.53130265628555
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 408,
        "accuracy": 0.43450479233226835,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 15.94554664415365,
        "ece": 0.18749352082057608,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 39.22342124001921
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 110,
        "accuracy": 0.4700854700854701,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 21.783010933557616,
        "ece": 0.1624125311988401,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 44.650252346894796
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 277,
        "accuracy": 0.5091911764705882,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 29.866036249014964,
        "ece": 0.12782727884349546,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 52.15029024015794
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 140,
        "accuracy": 0.5283018867924528,
        "chance": 0.5,
        "chance_corrected_intelligence": 5.660377358490565,
        "ece": 0.20222690751619474,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 32.60749792762581
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 88,
        "accuracy": 0.2410958904109589,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0.21613832853025922,
        "ece": 0.2807980657909426,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 22.02826258517087
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 156,
        "accuracy": 0.30174081237911027,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0,
        "ece": 0.2532957744942598,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 24.670422550574024
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 349,
        "accuracy": 0.5312024353120244,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 31.093627652037732,
        "ece": 0.14408759327673493,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 51.13805449834537
    }
  }
}
jev-spatial (Fr0zencr4nE)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 110,
        "accuracy": 0.46808510638297873,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 21.41659681475273,
        "ece": 0.26223485064675817,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 34.48481334270055
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 383,
        "accuracy": 0.4078807241746539,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 11.988180666948082,
        "ece": 0.302851872760477,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 25.708903057426344
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 105,
        "accuracy": 0.44871794871794873,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 18.629100084104294,
        "ece": 0.26898178044867654,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 32.41637199718449
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 271,
        "accuracy": 0.49816176470588236,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 28.289992119779352,
        "ece": 0.19976763148513516,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 44.16823291137616
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 155,
        "accuracy": 0.5849056603773585,
        "chance": 0.5,
        "chance_corrected_intelligence": 16.981132075471695,
        "ece": 0.2619078000460092,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 32.29978603313493
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 67,
        "accuracy": 0.18356164383561643,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.46829410998892224,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 3.1705890011077766
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 171,
        "accuracy": 0.3307543520309478,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0,
        "ece": 0.31106663545127894,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 18.893336454872106
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 322,
        "accuracy": 0.4901065449010654,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 25.053133972183893,
        "ece": 0.2818594336313012,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 34.34062362296183
    }
  }
}
Gevva E2B multimodal
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 98,
        "accuracy": 0.41702127659574467,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 13.872590108968986,
        "ece": 0.11269641637521752,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 45.66665341696273
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 384,
        "accuracy": 0.40894568690095845,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 12.146475306036303,
        "ece": 0.07837206377085307,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 48.23603127593285
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 102,
        "accuracy": 0.4358974358974359,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 16.736753574432303,
        "ece": 0.09504388417804957,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 48.863988369411196
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 253,
        "accuracy": 0.4650735294117647,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 23.561859732072495,
        "ece": 0.10148345804968922,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 51.63258406106733
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 158,
        "accuracy": 0.5962264150943396,
        "chance": 0.5,
        "chance_corrected_intelligence": 19.24528301886792,
        "ece": 0.09563020381484438,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 50.05962112794953
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 71,
        "accuracy": 0.19452054794520549,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.09592541099500315,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 40.407458900499684
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 164,
        "accuracy": 0.31721470019342357,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0,
        "ece": 0.10161796740251844,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 39.838203259748155
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 318,
        "accuracy": 0.4840182648401826,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 24.158246019612953,
        "ece": 0.07911801445165827,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 54.167321564640645
    }
  }
}
JEVision (divyanshx11, visual route)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 105,
        "accuracy": 0.44680851063829785,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 18.27326068734283,
        "ece": 0.39363776051471006,
        "probability_coverage": 0.9914893617021276,
        "valid_answer_count": 233
      },
      "score": 19.682333237319313
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 391,
        "accuracy": 0.4164004259850905,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 13.254537779653868,
        "ece": 0.42217679283808485,
        "probability_coverage": 0.9893503727369543,
        "valid_answer_count": 929
      },
      "score": 14.32671079114953
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 102,
        "accuracy": 0.4358974358974359,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 16.736753574432303,
        "ece": 0.411338834255511,
        "probability_coverage": 0.9914529914529915,
        "valid_answer_count": 232
      },
      "score": 17.158714587524464
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 252,
        "accuracy": 0.4632352941176471,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 23.299185710533227,
        "ece": 0.39796640868845556,
        "probability_coverage": 0.9889705882352942,
        "valid_answer_count": 538
      },
      "score": 21.740414937180383
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 162,
        "accuracy": 0.6113207547169811,
        "chance": 0.5,
        "chance_corrected_intelligence": 22.26415094339622,
        "ece": 0.29509632574738304,
        "probability_coverage": 0.9773584905660377,
        "valid_answer_count": 259
      },
      "score": 31.158510049595392
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 82,
        "accuracy": 0.22465753424657534,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.5287572810657589,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 0
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 147,
        "accuracy": 0.28433268858800775,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0,
        "ece": 0.5496954121003053,
        "probability_coverage": 0.9961315280464217,
        "valid_answer_count": 515
      },
      "score": 0
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 349,
        "accuracy": 0.5312024353120244,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 31.093627652037732,
        "ece": 0.310809116123387,
        "probability_coverage": 0.984779299847793,
        "valid_answer_count": 647
      },
      "score": 34.17794044217847
    }
  }
}
OmniJev 0.8B (tinnel123)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 98,
        "accuracy": 0.41702127659574467,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 13.872590108968986,
        "ece": 0.15353504147438463,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 41.58279090704603
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 371,
        "accuracy": 0.3951011714589989,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 10.088644997889407,
        "ece": 0.18076728323529428,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 36.96759417541527
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 91,
        "accuracy": 0.3888888888888889,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 9.79814970563499,
        "ece": 0.18227908107876004,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 36.671166744941495
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 257,
        "accuracy": 0.4724264705882353,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 24.612555818229573,
        "ece": 0.10694918446867947,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 51.61135946224684
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 137,
        "accuracy": 0.5169811320754717,
        "chance": 0.5,
        "chance_corrected_intelligence": 3.396226415094339,
        "ece": 0.20538681473101905,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 31.159431734445263
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 75,
        "accuracy": 0.2054794520547945,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.27660085110945354,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 22.339914889054647
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 145,
        "accuracy": 0.2804642166344294,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0,
        "ece": 0.25744680174507817,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 24.255319825492183
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 324,
        "accuracy": 0.4931506849315068,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 25.500577948469363,
        "ece": 0.11653229006907434,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 51.097059967327255
    }
  }
}
PlayJev 0.8B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 91,
        "accuracy": 0.3872340425531915,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 9.471919530595141,
        "ece": 0.28238670507017716,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 26.497289258279856
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 349,
        "accuracy": 0.37167199148029817,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 6.606162937948502,
        "ece": 0.2818877447664528,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 25.114306992328974
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 96,
        "accuracy": 0.41025641025641024,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 12.952060555088309,
        "ece": 0.24047807478530506,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 32.42822279901365
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 198,
        "accuracy": 0.3639705882352941,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 9.114788547412653,
        "ece": 0.2703601249953172,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 27.521381774174607
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 134,
        "accuracy": 0.5056603773584906,
        "chance": 0.5,
        "chance_corrected_intelligence": 1.132075471698113,
        "ece": 0.2688372391135347,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 23.68231382449559
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 108,
        "accuracy": 0.2958904109589041,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 7.420749279538905,
        "ece": 0.3167552623209894,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 22.034848407670513
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 220,
        "accuracy": 0.425531914893617,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 13.616753114547482,
        "ece": 0.20525936446062804,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 36.28244011121094
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 220,
        "accuracy": 0.334855403348554,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 2.2334911816249643,
        "ece": 0.3481900159078381,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 16.297744000028672
    }
  }
}
Liquid d1-omni-600M (Liquid AI)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 74,
        "accuracy": 0.3148936170212766,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 0,
        "ece": 0.30892513245670117,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 19.107486754329884
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 363,
        "accuracy": 0.3865814696485623,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 8.822287885183629,
        "ece": 0.23872875418865291,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 30.538268523726522
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 83,
        "accuracy": 0.3547008547008547,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 4.751892346509678,
        "ece": 0.26326837784759327,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 26.04910838849551
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 181,
        "accuracy": 0.3327205882352941,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 4.649330181245067,
        "ece": 0.26624863576672914,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 25.69980151394962
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 127,
        "accuracy": 0.47924528301886793,
        "chance": 0.5,
        "chance_corrected_intelligence": 0,
        "ece": 0.26738151450637404,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 23.261848549362597
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 129,
        "accuracy": 0.35342465753424657,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 14.985590778097981,
        "ece": 0.22217459419398397,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 35.27533596965059
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 193,
        "accuracy": 0.3733075435203095,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 5.763730670415435,
        "ece": 0.25473872380445167,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 27.40799295476255
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 244,
        "accuracy": 0.3713850837138508,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 7.602818897050595,
        "ece": 0.24867385152219673,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 28.934024296305626
    }
  }
}
djev-dev BF16
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 79,
        "accuracy": 0.33617021276595743,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 1.9279128248113995,
        "ece": 0.22276083727029306,
        "probability_coverage": 0.7957446808510639,
        "valid_answer_count": 187
      },
      "score": 23.02511531898238
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 339,
        "accuracy": 0.3610223642172524,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 5.02321654706628,
        "ece": 0.1968929529844664,
        "probability_coverage": 0.7976570820021299,
        "valid_answer_count": 749
      },
      "score": 26.68915653920243
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 89,
        "accuracy": 0.3803418803418803,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 8.536585365853659,
        "ece": 0.18321782321143282,
        "probability_coverage": 0.7948717948717948,
        "valid_answer_count": 186
      },
      "score": 29.448414427659092
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 209,
        "accuracy": 0.38419117647058826,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 12.004202784344628,
        "ece": 0.15361315311803114,
        "probability_coverage": 0.7261029411764706,
        "valid_answer_count": 395
      },
      "score": 31.15335222275645
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 134,
        "accuracy": 0.5056603773584906,
        "chance": 0.5,
        "chance_corrected_intelligence": 1.132075471698113,
        "ece": 0.1407546968770855,
        "probability_coverage": 0.7283018867924528,
        "valid_answer_count": 193
      },
      "score": 26.72994094442358
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 75,
        "accuracy": 0.2054794520547945,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.3237121072586425,
        "probability_coverage": 0.9534246575342465,
        "valid_answer_count": 348
      },
      "score": 16.807722376436274
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 199,
        "accuracy": 0.3849129593810445,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 7.508846769111442,
        "ece": 0.19845445814040658,
        "probability_coverage": 0.9148936170212766,
        "valid_answer_count": 473
      },
      "score": 31.34263253341214
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 219,
        "accuracy": 0.3333333333333333,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 2.0097691934822297,
        "ece": 0.20574544201835382,
        "probability_coverage": 0.7047184170471842,
        "valid_answer_count": 463
      },
      "score": 21.74154522771557
    }
  }
}
Qevi-2B (MeerDevelopment)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 77,
        "accuracy": 0.3276595744680851,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 0.6705783738474411,
        "ece": 0.42083712721044664,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 8.251576465879056
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 331,
        "accuracy": 0.35250266240681577,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 3.7568594343604946,
        "ece": 0.39168833516230506,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 12.70959620094974
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 80,
        "accuracy": 0.3418803418803419,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 2.8595458368376825,
        "ece": 0.4062368748417775,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 10.80608543424109
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 226,
        "accuracy": 0.41544117647058826,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 16.469661150512213,
        "ece": 0.3160396176517192,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 26.630868810084188
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 130,
        "accuracy": 0.49056603773584906,
        "chance": 0.5,
        "chance_corrected_intelligence": 0,
        "ece": 0.46072920298662534,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 3.927079701337466
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 52,
        "accuracy": 0.14246575342465753,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.482830146702229,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 1.7169853297770987
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 143,
        "accuracy": 0.2765957446808511,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 0,
        "ece": 0.46085113153764795,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 3.9148868462352047
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 265,
        "accuracy": 0.4033485540334855,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 12.30098064804802,
        "ece": 0.34896223215716926,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 21.254267108307083
    }
  }
}
djev-spark NVFP4
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 73,
        "accuracy": 0.31063829787234043,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 0,
        "ece": 0.21362227965583924,
        "probability_coverage": 0.7957446808510639,
        "valid_answer_count": 187
      },
      "score": 22.788354767811942
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 322,
        "accuracy": 0.34291799787007454,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 2.332207682566487,
        "ece": 0.19546528164885066,
        "probability_coverage": 0.7976570820021299,
        "valid_answer_count": 749
      },
      "score": 25.457531322115067
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 81,
        "accuracy": 0.34615384615384615,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 3.490328006728345,
        "ece": 0.1905384831958995,
        "probability_coverage": 0.7948717948717948,
        "valid_answer_count": 186
      },
      "score": 26.34338713394652
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 206,
        "accuracy": 0.3786764705882353,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 11.216180719726813,
        "ece": 0.13331296085064798,
        "probability_coverage": 0.7261029411764706,
        "valid_answer_count": 395
      },
      "score": 32.23334412162701
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 130,
        "accuracy": 0.49056603773584906,
        "chance": 0.5,
        "chance_corrected_intelligence": 0,
        "ece": 0.08541176230601559,
        "probability_coverage": 0.7283018867924528,
        "valid_answer_count": 193
      },
      "score": 30.194539575448676
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 59,
        "accuracy": 0.16164383561643836,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.3392843305887592,
        "probability_coverage": 0.9534246575342465,
        "valid_answer_count": 348
      },
      "score": 15.323028206879947
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 192,
        "accuracy": 0.3713733075435203,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 5.472877987299435,
        "ece": 0.1853940021737022,
        "probability_coverage": 0.9148936170212766,
        "valid_answer_count": 473
      },
      "score": 31.51954092243867
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 203,
        "accuracy": 0.3089802130898021,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 0,
        "ece": 0.2130874713228293,
        "probability_coverage": 0.7047184170471842,
        "valid_answer_count": 463
      },
      "score": 20.21925430403806
    }
  }
}
OpenJev 4B NLI v2 (official image-premise path)
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 102,
        "accuracy": 0.4340425531914894,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 16.3872590108969,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 233
      },
      "score": 8.19362950544845
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 411,
        "accuracy": 0.43769968051118213,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 16.420430561418325,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 929
      },
      "score": 8.210215280709162
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 107,
        "accuracy": 0.45726495726495725,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 19.89066442388562,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 232
      },
      "score": 9.94533221194281
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 300,
        "accuracy": 0.5514705882352942,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 35.90753874441818,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 538
      },
      "score": 17.95376937220909
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 125,
        "accuracy": 0.4716981132075472,
        "chance": 0.5,
        "chance_corrected_intelligence": 0,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 259
      },
      "score": 0
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 88,
        "accuracy": 0.2410958904109589,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0.21613832853025922,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 365
      },
      "score": 0.10806916426512961
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 181,
        "accuracy": 0.35009671179883944,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 2.273498473023411,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 515
      },
      "score": 1.1367492365117056
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 332,
        "accuracy": 0.5053272450532724,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 27.290353853611236,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 647
      },
      "score": 13.645176926805618
    }
  }
}
OpenJev 4B NLI v5
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 127,
        "accuracy": 0.5404255319148936,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 32.103939647946355,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 233
      },
      "score": 16.051969823973177
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 517,
        "accuracy": 0.5505857294994675,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 33.19966230476994,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 929
      },
      "score": 16.59983115238497
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 138,
        "accuracy": 0.5897435897435898,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 39.44491169049622,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 232
      },
      "score": 19.72245584524811
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 330,
        "accuracy": 0.6066176470588235,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 43.787759390596264,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 538
      },
      "score": 21.893879695298132
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 177,
        "accuracy": 0.6679245283018868,
        "chance": 0.5,
        "chance_corrected_intelligence": 33.58490566037735,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 259
      },
      "score": 16.792452830188676
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 137,
        "accuracy": 0.37534246575342467,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 17.867435158501443,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 365
      },
      "score": 8.933717579250722
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 228,
        "accuracy": 0.4410058027079304,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 15.943574579475499,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 515
      },
      "score": 7.971787289737749
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 416,
        "accuracy": 0.6331811263318112,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 46.08300085760094,
        "ece": null,
        "probability_coverage": 0,
        "valid_answer_count": 647
      },
      "score": 23.04150042880047
    }
  }
}
Gemini 3.8 Flash
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 86,
        "accuracy": 0.3659574468085106,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 6.328583403185245,
        "ece": 0.09715116279069767,
        "probability_coverage": 0.3659574468085106,
        "valid_answer_count": 86
      },
      "score": 17.90684489308198
    },
    "S": {
      "n": 0,
      "metrics": null,
      "score": null
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 93,
        "accuracy": 0.3974358974358974,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 11.059714045416316,
        "ece": 0.06817021276595751,
        "probability_coverage": 0.4017094017094017,
        "valid_answer_count": 94
      },
      "score": 22.876865569716703
    }
  },
  "per_type": {
    "choice": {
      "n": 214,
      "metrics": {
        "n": 214,
        "correct": 103,
        "accuracy": 0.48130841121495327,
        "chance": 0.29042056074766354,
        "chance_corrected_intelligence": 26.901547579848536,
        "ece": 0.06363461538461535,
        "probability_coverage": 0.48598130841121495,
        "valid_answer_count": 104
      },
      "score": 34.65731584599904
    },
    "noul": {
      "n": 108,
      "metrics": {
        "n": 108,
        "correct": 39,
        "accuracy": 0.3611111111111111,
        "chance": 0.5,
        "chance_corrected_intelligence": 0,
        "ece": 0.12897435897435894,
        "probability_coverage": 0.3611111111111111,
        "valid_answer_count": 39
      },
      "score": 13.39814814814815
    },
    "score": {
      "n": 147,
      "metrics": {
        "n": 147,
        "correct": 37,
        "accuracy": 0.25170068027210885,
        "chance": 0.23979591836734693,
        "chance_corrected_intelligence": 1.565995525727071,
        "ece": 0.07943243243243245,
        "probability_coverage": 0.25170068027210885,
        "valid_answer_count": 37
      },
      "score": 11.368712048577821
    }
  },
  "hard_slices": {
    "hard": {
      "n": 206,
      "metrics": {
        "n": 206,
        "correct": 26,
        "accuracy": 0.1262135922330097,
        "chance": 0.33001618122977344,
        "chance_corrected_intelligence": 0,
        "ece": 0.11074074074074068,
        "probability_coverage": 0.13106796116504854,
        "valid_answer_count": 27
      },
      "score": 5.101941747572816
    },
    "non-hard": {
      "n": 263,
      "metrics": {
        "n": 263,
        "correct": 153,
        "accuracy": 0.5817490494296578,
        "chance": 0.3171736375158429,
        "chance_corrected_intelligence": 38.74709976798144,
        "ece": 0.08507843137254904,
        "probability_coverage": 0.5817490494296578,
        "valid_answer_count": 153
      },
      "score": 43.51157269767893
    }
  }
}
Standard One 3B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 100,
        "accuracy": 0.425531914893617,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 15.129924559932945,
        "ece": 0.04980213132130396,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 52.58474914783608
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 387,
        "accuracy": 0.41214057507987223,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 12.621359223300978,
        "ece": 0.08389594827742633,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 47.92108478390786
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 103,
        "accuracy": 0.44017094017094016,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 17.367535744322964,
        "ece": 0.07818868139768935,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 50.864899732392544
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 246,
        "accuracy": 0.4522058823529412,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 21.723141581297607,
        "ece": 0.026257317032092508,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 58.23583908743956
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 167,
        "accuracy": 0.630188679245283,
        "chance": 0.5,
        "chance_corrected_intelligence": 26.0377358490566,
        "ece": 0.02900700366825616,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 60.118167557702684
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 74,
        "accuracy": 0.20273972602739726,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 0,
        "ece": 0.21005629637912387,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 28.99437036208761
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 178,
        "accuracy": 0.344294003868472,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 1.4009404236754115,
        "ece": 0.11827983591400583,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 38.87248662043712
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 309,
        "accuracy": 0.4703196347031963,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 22.144748126328345,
        "ece": 0.06802229040734738,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 54.27014502242943
    }
  }
}
Standard One 8B
{
  "sets": {
    "P": {
      "n": 235,
      "metrics": {
        "n": 235,
        "correct": 119,
        "accuracy": 0.5063829787234042,
        "chance": 0.3231205673758865,
        "chance_corrected_intelligence": 27.074601844090523,
        "ece": 0.12514218199138863,
        "probability_coverage": 1,
        "valid_answer_count": 235
      },
      "score": 51.0230827229064
    },
    "S": {
      "n": 939,
      "metrics": {
        "n": 939,
        "correct": 484,
        "accuracy": 0.5154419595314164,
        "chance": 0.3272275470358537,
        "chance_corrected_intelligence": 27.975939214858588,
        "ece": 0.12155429229251659,
        "probability_coverage": 1,
        "valid_answer_count": 939
      },
      "score": 51.83254037817763
    },
    "A": {
      "n": 234,
      "metrics": {
        "n": 234,
        "correct": 127,
        "accuracy": 0.5427350427350427,
        "chance": 0.3225071225071225,
        "chance_corrected_intelligence": 32.506307821698904,
        "ece": 0.1064742053234561,
        "probability_coverage": 1,
        "valid_answer_count": 234
      },
      "score": 55.60573337850384
    }
  },
  "per_type": {
    "choice": {
      "n": 544,
      "metrics": {
        "n": 544,
        "correct": 314,
        "accuracy": 0.5772058823529411,
        "chance": 0.3001838235294118,
        "chance_corrected_intelligence": 39.584975045967944,
        "ece": 0.05267302359519717,
        "probability_coverage": 1,
        "valid_answer_count": 544
      },
      "score": 64.52518516346426
    },
    "noul": {
      "n": 265,
      "metrics": {
        "n": 265,
        "correct": 181,
        "accuracy": 0.6830188679245283,
        "chance": 0.5,
        "chance_corrected_intelligence": 36.60377358490565,
        "ece": 0.08812818203452755,
        "probability_coverage": 1,
        "valid_answer_count": 265
      },
      "score": 59.48906858900007
    },
    "score": {
      "n": 365,
      "metrics": {
        "n": 365,
        "correct": 108,
        "accuracy": 0.2958904109589041,
        "chance": 0.23945205479452056,
        "chance_corrected_intelligence": 7.420749279538905,
        "ece": 0.26569936975990244,
        "probability_coverage": 1,
        "valid_answer_count": 365
      },
      "score": 27.140437663779206
    }
  },
  "hard_slices": {
    "hard": {
      "n": 517,
      "metrics": {
        "n": 517,
        "correct": 213,
        "accuracy": 0.4119922630560928,
        "chance": 0.33497743391360413,
        "chance_corrected_intelligence": 11.580784332735465,
        "ece": 0.17764796381392636,
        "probability_coverage": 1,
        "valid_answer_count": 517
      },
      "score": 38.025595784975096
    },
    "non-hard": {
      "n": 657,
      "metrics": {
        "n": 657,
        "correct": 390,
        "accuracy": 0.593607305936073,
        "chance": 0.31966007102993405,
        "chance_corrected_intelligence": 40.26622916588985,
        "ece": 0.0857922204164513,
        "probability_coverage": 1,
        "valid_answer_count": 657
      },
      "score": 61.55389254129979
    }
  }
}

Computer-use track

Separate from the core Capability headline. No mixing of track scores.

SystemCapabilityCompositePublic / sealed items
Wity-1 (AlphaNimble), reasoning autoAPI90.8579.0065 / 66
Imajev-4B92.8477.9465 / 261
Surogate Rune 26B-A4B v386.3164.9165 / 261
Autoloops – Gemma 4 31B ITAPI82.9847.1065 / 66
imajev 9B87.3950.8365 / 261
H2O-Lightning-4B (image)89.3160.9965 / 261
Wity-1 (AlphaNimble), reasoning offAPI84.5978.7765 / 66
Jev-Omni76.5873.9965 / 261
JPT-9B (kirp / llm2jev)89.2857.8165 / 261
NeoHorse Jev 4B85.7173.6365 / 261
Jevify Gemma 4 26B-A4B81.9463.8865 / 261
AutoJev-27B80.8166.2165 / 261
JPT-4B (kirp / llm2jev)90.2768.3865 / 261
Jev-Vision 8B (SeanLiu)74.3061.8765 / 261
System1 s1-vision (System1 Models)API87.5542.2265 / 66
shisa-de-180.7263.2765 / 261
Clef-Flash (Cloudflare)83.7455.2965 / 261
Clef (Cloudflare)91.4329.9065 / 261
GPT-6 Luna via OpenAI Decisions (OpenAI)API89.2527.4465 / 66
Winnow-12B (Q8_0 + F16 vision projector)91.9748.8765 / 261
Image Hopper (HopitAI)83.4962.0165 / 261
OmniJev-Qwen3.5-9B-v4 (tzcfly)69.8270.3265 / 261
OmniJev 4B (tinnel123)66.2062.9865 / 261
Reflex 4B (released stable configuration)81.0067.2865 / 261
CUA-S1 4B83.0956.4965 / 261
GPT-6 Luna (low reasoning effort)API96.4916.9265 / 66
Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj85.5822.5865 / 261
Visual-Jev generic (zero-shot Qwen3.5-4B baseline)83.5751.3765 / 261
Mapika decider-2b-vision BF1676.3072.0165 / 261
vjev-vision (yah01)65.2659.8765 / 261
Visual-Jev 4B Answer-SFT81.5571.4165 / 261
GPT-5.6 LunaAPI93.2212.2065 / 66
Jevify Gemma 4 E4B69.8669.8865 / 261
JPT-0.8B (kirp / llm2jev)77.2871.8765 / 261
Glance (frozen Qwen3-VL-4B)68.8768.1365 / 261
Gemini 3.1 Flash LiteAPI93.319.8965 / 66
diffusiongemma-26b djev v10 step160 (snowicarus)71.4253.8565 / 261
imajev 2B77.1269.7165 / 261
Gevva E4B (text checkpoint, image input)57.0856.0865 / 261
Liquid d1-3B (Liquid AI)68.0366.2265 / 261
djev-distill-v4 (tarsur385)74.5054.2465 / 261
Jevify Qwen3-VL-2B T269.2467.8065 / 261
OmniJev 2B (tinnel123)58.7461.5065 / 261
jev-spatial (Fr0zencr4nE)60.0365.2865 / 261
Gevva E2B multimodal56.5159.9265 / 261
JEVision (divyanshx11, visual route)53.1748.8365 / 261
OmniJev 0.8B (tinnel123)48.3725.2165 / 261
PlayJev 0.8B31.590.8765 / 261
Liquid d1-omni-600M (Liquid AI)34.655.0465 / 261
djev-dev BF1621.100.0265 / 261
Qevi-2B (MeerDevelopment)44.6037.8065 / 261
djev-spark NVFP422.590.0465 / 261
OpenJev 4B NLI v2 (official image-premise path)25.850.0065 / 261
OpenJev 4B NLI v537.860.0065 / 261
Gemini 3.8 FlashAPI56.440.0065 / 66
Standard One 3B60.89—65 / 261
Standard One 8B81.65—65 / 261

API: Wity-1 (AlphaNimble), reasoning auto, Autoloops – Gemma 4 31B IT, Wity-1 (AlphaNimble), reasoning off, System1 s1-vision (System1 Models), GPT-6 Luna via OpenAI Decisions (OpenAI), GPT-6 Luna (low reasoning effort), GPT-5.6 Luna, Gemini 3.1 Flash Lite, Gemini 3.8 Flash ran the hosted-API subset (66 sealed items instead of 261), so its computer-use scores are not comparable with the self-hosted rows.

Dated carry — v0.1.5 scores

Hosted APIs carry their previous score between rotation-eligible refreshes. Unavailable sources and rows not yet measured remain visible. All rows below are carried, not ranked on v0.3 scale.

SystemCapability (v0.1.5 all track)Composite (v0.1.5)Last measured / release dateStatus
JevAny-27B SFT79.8063.792026-09-26Source unavailable since 30 Sep 2026: the Hugging Face repository returns 401 and the author lists no public models. Carried, not re-measured.
JevAny-27B RLCR80.1163.462026-09-26Source unavailable since 30 Sep 2026: the Hugging Face repository returns 401 and the author lists no public models. Carried, not re-measured.

The examples below are retained v0.1.5 public examples for illustration; they are not v0.3 sealed items.

Examples from public items

These eight examples are from the public split. Each card shows the image, question, options and correct answer; licensed sources are credited below their image.

A cardboard parcel with a visibly torn corner on a conveyor.

Everyday photo

Parcel condition

Question: Is the parcel visibly damaged?

  • A. yesCorrect
  • B. no

Correct answer: A. yes

Source: Synthetic image created for ImageJevBench · public item promo-parcel-01

A printed cafe receipt with item names, amounts, tax and total visible.

Everyday photo

Receipt legibility

Question: Is the receipt readable?

  • A. yesCorrect
  • B. no

Correct answer: A. yes

Source: Synthetic image created for ImageJevBench · public item promo-receipt-00

A spreadsheet with a number-format dialog and five labelled target markers.

Computer use · spreadsheet

Change a cell format

Question: Goal: change selected cells to type “Text”. Which labelled marker should be clicked?

  • A. Click marker A
  • B. Click marker BCorrect
  • C. Click marker C
  • D. Click marker D
  • E. Click marker E

Correct answer: B. Click marker B

Source: ScreenSpot-Pro · excel_macos_0 · MIT · 210e78d38442 · public item mm-195 · Licence text: MITAdapted: five labelled click markers were drawn on the source screenshot.

A desktop browser showing a sports page, browser tabs and five labelled markers.

Browser

Open a new tab

Question: Goal: open a new tab. Which labelled marker should be clicked?

  • A. Click marker A
  • B. Click marker B
  • C. Click marker C
  • D. Click marker D
  • E. Click marker ECorrect

Correct answer: E. Click marker E

Source: ScreenSpot · item 15 · Apache-2.0 · 0be08781e2e1 · public item mm-135 · Licence text: Apache-2.0Adapted: five labelled click markers were drawn on the source screenshot.

A mobile translation interface with “Hello, world” and five labelled target markers.

Mobile app · translation

Use the translate control

Question: Goal: translate. Which labelled marker should be clicked?

  • A. Click marker A
  • B. Click marker B
  • C. Click marker C
  • D. Click marker DCorrect
  • E. Click marker E

Correct answer: D. Click marker D

Source: ScreenSpot · item 341 · Apache-2.0 · 0be08781e2e1 · public item mm-118 · Licence text: Apache-2.0Adapted: five labelled click markers were drawn on the source screenshot.

A Windows file-manager context menu with five labelled target markers.

Computer use · file manager

Copy a file

Question: Goal: copy the file. Which labelled marker should be clicked?

  • A. Click marker ACorrect
  • B. Click marker B
  • C. Click marker C
  • D. Click marker D
  • E. Click marker E

Correct answer: A. Click marker A

Source: ScreenSpot · item 19 · Apache-2.0 · 0be08781e2e1 · public item mm-139 · Licence text: Apache-2.0Adapted: five labelled click markers were drawn on the source screenshot.

A compact financial table with 2014 and 2015 net revenue values.

FinQA · financial table

Calculate a revenue change

Question: What is the net change in net revenue during 2015 for Entergy Corporation?

  • A. 103.4
  • B. 84.6
  • C. 94Correct
  • D. 112.8

Correct answer: C. 94

Source: FinQA · table item · MIT annotations; CDLA-Permissive-1.0 table data · 3d6a736bc67e · public item mm-061 · Licence text: MIT, CDLA-Permissive-1.0Table data: IBM FinTabNet (CDLA-Permissive-1.0). The table was re-rendered as an image by ImageJevBench; no original filing page is shown.

Screenshot images are adapted from the credited datasets (labelled markers added); the FinQA table is re-rendered by ImageJevBench. Licences: Apache-2.0 (ScreenSpot), MIT (ScreenSpot-Pro, Geometry3K, FinQA annotations), CDLA-Permissive-1.0 (FinTabNet table data). App and website content shown in screenshots belongs to its respective owners. No Mind2Web or Android-in-the-Wild image is shown.

v0.3.0 method notes

The new reserve has 2,441 items: 713 real Hugging Face images/items with CC-BY/MIT provenance and 1,728 synthetic items. The frozen draw has 300 public and 1,200 sealed items; hosted APIs receive the 300-item API subset plus public items. The sealed draw uses 600 choice / 300 Noul / 300 score tasks and a 40% hard share, across 14 families, 14 languages and five use cases.

Pass 1 is the official prediction pass; pass 2 is diagnostic and is never included in these rankings. Reproducibility: a full second pass of the 29 common self-hosted systems on a fresh pod returned the same answer for 1,500 of 1,500 items in every system (100 % agreement). Frozen v0.2 scoring math is applied to the new draw. Core is the headline; computer-use is separate. Pending field-median gap/API equating holds remain visible per row. Scores from v0.1.5 are not directly ranked on the v0.3 scale. GPT-6 Luna via OpenAI Decisions was measured on 6 Oct 2026 through the native Decisions choice endpoint (same option text, native option probabilities, images re-encoded losslessly as PNG, refusals and provider errors counted wrong, no retries); the GPT-6 Luna chat-completions row was re-measured separately on 7 Oct 2026 (see below). Wity-1 (AlphaNimble) was measured on 7 Oct 2026 on the same A300+P300 input through its native SystemOne choice endpoint (reasoning=auto, native option probabilities, no retries; failed requests and timeouts counted wrong). Images that would exceed Wity's 12 MB request limit were downscaled to 1,500 px on the long side, as Wity's docs advise (2 images). Latency is measured end to end; on this run large images took much longer on the Wity side than the model time Wity reports. A second Wity-1 row uses reasoning off (the server default); the first row uses reasoning auto. Dated-carry re-measurement (7 Oct 2026, requested by Florian): GPT-6 Luna (chat), GPT-5.6 Luna, Gemini 3.1 Flash Lite, Gemini 3.8 Flash and Autoloops Gemma 4 31B IT were re-measured on the same A300+P300 input as an owner exception to the N=3 API refresh cadence (no provider version change is claimed; exposure logged before dispatch). OpenRouter and Autoloops requests used the unchanged v0.2 prompts and parsers; images were sent with their real MIME type, 7 BMP images were re-encoded losslessly as PNG and 5 images over 6 MiB were downscaled to 2,048 px on the long side. Provider failures count wrong; no retries. Gemini 3.8 Flash reasons by default and ran out of the 384-token answer budget on 329 of 600 items, which count wrong under the unchanged method. NeoHorse Jev 4B was re-measured on its self-hosted runtime at Hugging Face revision 434cb21d (identical weights and code; the reviewed revision 56c36ae6 was removed by the author). The two JevAny rows remain carries because their source is unavailable.

TypeSafe-20 use-case radar. A separate sealed bank of 600 image decision items, 30 for each TypeSafe use case, four options each, frozen on 10 October 2026 after blind Gold checks. Self-hosted systems answer all 600 offline with their pinned v0.3 runtimes; hosted APIs answer the same 600, with exposure logged before dispatch. Values are chance-corrected accuracy per use case; errors and unparseable answers count as wrong (Gemini 3.8 Flash: 136 of 600, GPT-6 Luna (low reasoning effort): 9 of 600, GPT-5.6 Luna: 8 of 600, GPT-6 Luna via OpenAI Decisions (OpenAI): 4 of 600; every other measured system answered all 600). They are not part of the v0.3 score or rank. 56 of 57 listed systems are measured so far; the others show gaps until measured.

Each refresh draws fresh items. Items retire after three scored uses; hosted API exposure is at most once per three refresh releases or on a new model version. Exposure is logged before dispatch, items are not reused for that provider, and retire globally after two providers.

Image-type and TypeSafe use-case radar panels are retained. Their values are chance-corrected category competence measured on the v0.3.0 draw (categories with fewer than 30 items are listed in a table, not drawn). No v0.1.5 category values or inferred scores are substituted.

Aggregate source receipts and holds
{
  "sources": [
    {
      "path": "run03/final-gpu/autojev_27b.json",
      "sha256": "a5b31cb684a606a00a7ba9e9c9c8bf6b2344c335de6daa7ea7545580ce115aea"
    },
    {
      "path": "run03/final-gpu/bonsai_2_27b_pq2_0.json",
      "sha256": "40a12210c95a3804facef75dccabbc7c4738b4da4f948cc76bb7379c340e401c"
    },
    {
      "path": "run03/final-gpu/clef.json",
      "sha256": "5df29672d9fcaad89a4ef47135ab109b1476ae64afc54ebb5efd27b6e9b8c688"
    },
    {
      "path": "run03/final-gpu/clef_flash.json",
      "sha256": "73e41b8eb4651710379bc261c81bd25e768dc0a8fe613f1f5040dabde6075d68"
    },
    {
      "path": "run03/final-gpu/cua_s1_4b.json",
      "sha256": "47e9287ce9d06f76eaac51fe5e2aa91e081f16f82cc5bae6b58e4c582fd2d2be"
    },
    {
      "path": "run03/final-gpu/decider_2b_vision.json",
      "sha256": "5f1c07ea1ae81f5d4340537b09f5cda80dd3b0a1c43d12e22e4891dd1bfa82f8"
    },
    {
      "path": "run03/final-gpu/djev_dev_bf16.json",
      "sha256": "88f3f4800fb9d00269695321f8cb4a76ec758dad29b55f5f0d687445f1f2bc8a"
    },
    {
      "path": "run03/final-gpu/djev_distill_v4.json",
      "sha256": "5d0c425d020f879b101cf4cc9f90a5bfd413483373c69187ba28d91dd169470b"
    },
    {
      "path": "run03/final-gpu/djev_spark_nvfp4.json",
      "sha256": "2efefcf23789b1294ab0fcbd27b46ee3f4b7c3c19e7f93ede32b9d969e7a09b5"
    },
    {
      "path": "run03/final-gpu/djev_v10_step160_snowicarus.json",
      "sha256": "f11f3c9db72ef1adf4a76327a7b84c4d179b5841ae440adcee54d78efebe1fc1"
    },
    {
      "path": "run03/final-gpu/gevva_e2b_multimodal.json",
      "sha256": "e54c157011f833d1f0be42ba33fd659ad09bb64776c0d5651adb69bcf62017c7"
    },
    {
      "path": "run03/final-gpu/gevva_e4b.json",
      "sha256": "5aef4ce94e9ffbea5c15056c6e93338120148ddde0ee05f64d746d079201791d"
    },
    {
      "path": "run03/final-gpu/glance_qwen3vl_4b.json",
      "sha256": "c151b664a54a3cc8e7fc9e8e29b114af7a0292c22127ed6b385c4caac7922276"
    },
    {
      "path": "run03/final-gpu/h2o_lightning_4b_image.json",
      "sha256": "ef959a912489c9daef699505924bfab34b4827d42cc8869503f3490ca2c437af"
    },
    {
      "path": "run03/final-gpu/image_hopper.json",
      "sha256": "1d606731451f1dd4978deac243fc22d7b01408f97f8903e01721842d417a6d35"
    },
    {
      "path": "run03/final-gpu/imajev_2b.json",
      "sha256": "628eaab837889bff612f6ffe02784dc9403113b7cb1a2c6e81cf4ba465ad1317"
    },
    {
      "path": "run03/final-gpu/imajev_4b.json",
      "sha256": "6ffb09e2f7c455f702db5b26d252205925b444adc7c147706d1deb166f0e21a1"
    },
    {
      "path": "run03/final-gpu/imajev_9b.json",
      "sha256": "1dd712bc8d5fe4669c17da48d9ad245313e267cb6fd07d090c5724c2dab17f2d"
    },
    {
      "path": "run03/final-gpu/jev_omni.json",
      "sha256": "2c12b716b37de603e18689ea5053e0316fe10c5901a497ba69100d4a69052239"
    },
    {
      "path": "run03/final-gpu/jev_spatial.json",
      "sha256": "42ec7295865653e263036aff13a2583c009de42cc775d6a8dd23d4a1c689922e"
    },
    {
      "path": "run03/final-gpu/jevify_gemma4_26b_a4b.json",
      "sha256": "eff7d61fb9d3d12e6187104567fae9fae0f4f042f8a8fc55ecc9b9450e35faf0"
    },
    {
      "path": "run03/final-gpu/jevify_gemma4_e4b.json",
      "sha256": "63d9a63e9bfabf0004d67f5cc412344d4a4f0c596388ba65ce52063ac9463811"
    },
    {
      "path": "run03/final-gpu/jevify_qwen3vl_2b_t2.json",
      "sha256": "29fcf9bd8d6f71b62e2cbf2645035aeca7a09a28044afce7582d95c887575882"
    },
    {
      "path": "run03/final-gpu/jevision.json",
      "sha256": "5efd9fdf3068f33e3fc020e62d543a78dc64ba266a55e321ea46ad8e042cb998"
    },
    {
      "path": "run03/final-gpu/jpt_0p8b.json",
      "sha256": "4850602fca1f18c2220fe2789746fbc75d48bb85566be2914ebd3cdeefb4b022"
    },
    {
      "path": "run03/final-gpu/jpt_4b.json",
      "sha256": "8bd7a686edbf061cd7b8dd254d2c1d3c1a41fb2169a41409d0735ffad932d33a"
    },
    {
      "path": "run03/final-gpu/jpt_9b.json",
      "sha256": "adb645ad2b334791a1492dc0acccaa8e7edbe2c2378397de3666c03e99286d71"
    },
    {
      "path": "run03/final-gpu/liquid_d1_3b.json",
      "sha256": "9cc90787efa9d8fcf3012e04286da5ecd68bc5124bad9a1769e407cab6a4e81f"
    },
    {
      "path": "run03/final-gpu/liquid_d1_omni_600m.json",
      "sha256": "08ea73e019815467216339490e2487f2faf8368c99e928d32c1ee57d4a822963"
    },
    {
      "path": "run03/final-gpu/neohorse_jev_4b.json",
      "sha256": "7050d7b9217d5a64ea6e8b3e9947c624e0a76beeeb9472eedc05d4b8ea00c635"
    },
    {
      "path": "run03/final-gpu/omnijev_0p8b.json",
      "sha256": "743230b680f07f5556310ad00960499fc72e43886f724e817937cacdb825a534"
    },
    {
      "path": "run03/final-gpu/omnijev_2b.json",
      "sha256": "0c08abd85538e77c84de0a1aa39f1416848e430d6aaf5c356cf3156db372f091"
    },
    {
      "path": "run03/final-gpu/omnijev_4b.json",
      "sha256": "c13c07688e4f792240a2906abab9d2f2bb2c9208570cfc68749824ad5f0f5c00"
    },
    {
      "path": "run03/final-gpu/omnijev_9b_v4_tzcfly.json",
      "sha256": "bd6128d552e5433c7473be59bec60f94385e5c493af45bc282313d416aeb2a9d"
    },
    {
      "path": "run03/final-gpu/openjev_4b_nli.json",
      "sha256": "f9271bea85d1a9f03bdb608632ced8cd197cd492cecc0d9559377c58edf56d59"
    },
    {
      "path": "run03/final-gpu/openjev_4b_v5.json",
      "sha256": "1a020b527b3b3584ebf0a93fd7597552126cc2b7cda462c9b9f4936b2bd5457a"
    },
    {
      "path": "run03/final-gpu/playjev_0p8b.json",
      "sha256": "b91277d126b1ba9dbddb2bf9622996b6f41987649ba4b8b2741db72faebe5b91"
    },
    {
      "path": "run03/final-gpu/qevi_2b.json",
      "sha256": "dcceed27eb5e2ea4feb0449f7db92f80de84285f5f94b8f88d09b35909c69e37"
    },
    {
      "path": "run03/final-gpu/reflex_4b.json",
      "sha256": "89185fbe0c11afd188132a8a0dfe287f1e31d43843c82bd8d015fe2ef87f55b1"
    },
    {
      "path": "run03/final-gpu/seanliu_jev_vision_8b.json",
      "sha256": "75d43e09e664cf5bc4deef04ec8eb72b1e1454b558b8e06ae5e619db027167d4"
    },
    {
      "path": "run03/final-gpu/shisa_de_1.json",
      "sha256": "75b2999c5f0f456bdc1b76b5d1c698b8f5fdb4a2224ae071c4e8d26d86afe1b0"
    },
    {
      "path": "run03/final-gpu/standard_one_3b.json",
      "sha256": "0737014fb08f54c87034e7d75ab02680b0bae8e9d75e1112f36710cdd86a3411"
    },
    {
      "path": "run03/final-gpu/standard_one_8b.json",
      "sha256": "eb30e85eb0e5724b4c09abcfeaf03bfdc42f488e4df21c9d11f3cef691ab6066"
    },
    {
      "path": "run03/final-gpu/surogate_rune_26b_v3.json",
      "sha256": "088dfb1cbe44c98926d9e2c92ab1279477ec646049f5770e33d7839687a8fa30"
    },
    {
      "path": "run03/final-gpu/visual_jev_4b.json",
      "sha256": "be8f00bcf253c1aaae793febc326784344c452e69772fe477b8dcab2568431cf"
    },
    {
      "path": "run03/final-gpu/visual_jev_generic_baseline.json",
      "sha256": "33d721a19ab47af122a763f8fd49ba8a2f90cff8d27d2832111aa78dc71f0282"
    },
    {
      "path": "run03/final-gpu/vjev_vision.json",
      "sha256": "3f377c27c968c719237d0b9899852b1cd1e8957cd928e48c1b452ee7c00995b4"
    },
    {
      "path": "run03/final-gpu/winnow_12b_q8.json",
      "sha256": "3c9e38ae9c12d7d3ca4ff8a4c121769c5994bc14e9caa0a95fe014a73170dcdd"
    },
    {
      "path": "run03/final-api/gemini31_flash_lite.json",
      "sha256": "73ebfb4c8008ba8423624b088d4bdaad6c6a808a8d1db8c6be9cbf321e3e3e63"
    },
    {
      "path": "run03/final-api/gemini38_flash.json",
      "sha256": "0a8903728d2e951d3c04ff152b9d6ab15cf5b76d6796effcef1550d43f0bbe09"
    },
    {
      "path": "run03/final-api/gpt56_luna.json",
      "sha256": "72a28e195b794ec918a6c2d1a1e93036b6b3cde4091bb70a17f16cd5dd2e26de"
    },
    {
      "path": "run03/final-api/gpt6_luna.json",
      "sha256": "99c186e38f92c8788a47c9f5aee8fc57d04743a9a0d6a4c9b8b9b0397d864a3a"
    },
    {
      "path": "run03/final-api/gpt6_luna_decisions.json",
      "sha256": "89d1f5cf97f17f007432ab7bae7564dc7132b6f6603a05a7e76fc333f71992f2"
    },
    {
      "path": "run03/final-api/kushal_gemma4_31b_it_autoloops.json",
      "sha256": "949ddf6d6560e56450724f50bd8197fd5f47f1cc7edcbbff47f3500ee72cae87"
    },
    {
      "path": "run03/final-api/s1-vision.json",
      "sha256": "45249a89358c1c0164443d1bae27deaa91dbc4154c0a9bbc26bd59de80cebcb5"
    },
    {
      "path": "run03/final-api/wity_1.json",
      "sha256": "9943057d9375f272916b5af6071f1380a414150c0a9483056b8cda91ebee44c2"
    },
    {
      "path": "run03/final-api/wity_1_off.json",
      "sha256": "2d60e36f34cf409e7bd72c4cf308b97bc2475862e1ce7cf1249c8a1b47732f2b"
    }
  ],
  "warnings": []
}

Revision history

v0.3.0 ·
  • New item pool of 2,441 items: 713 real and 1,728 synthetic; fresh 300 public / 1,200 sealed draw.
  • 57 measured systems, 55 ranked; 2 dated v0.1.5 carries, not ranked on the v0.3 scale.
  • Same-day fix: the lists follow the official rank order.

Earlier revisions (v0.1 to v0.1.5) are in the archive's revision history.

v0.1.5 archive — full live page and earlier revision history

Image benchmark · v0.1.5

Image JevBench

The JevImageBench Capability Score compares intelligence and calibration inside the same cost and speed budget as JevBench.

JevImageBench v0.1.5 · headline

JevImageBench Capability Score

Capability ranking of Jev-class systems

Capability Score averages Intelligence and Calibration. Wity-1 leads the Jev-class systems with 85.7.

Jev-class means at most 2× the cost and median latency of Jev 1.13.0 (JevBench). How we choose ↘

Adjust cost / latency caps · 2× official
Official
#SystemICCap.$/1k
  1. 1Wity-1API83.773.985.7$0.0074API price · eligibility checked at the developer's list price; at base-model pricing it would still fit within the cost cap (0.82× Jev 1.13.0)
  2. 2JPT-9B75.748.283.6$0.053
  3. 3Imajev-4B73.861.282.1$0.020
  4. 4shisa-de-172.447.682.1$0.056
  5. 5NeoHorse Jev 4B72.951.682.0$0.041
  6. 6imajev 9B74.746.182.0$0.063
  7. 7Surogate Rune 26B-A4B v374.947.881.3$0.055
  8. 8JevAny-27B RLCR70.247.980.1$0.055
  9. 9Jevify Gemma 4 26B-A4B71.648.479.8$0.053
  10. 10JevAny-27B SFT69.648.179.8$0.054
Show all 38 Jev-class systems (28 more)
  1. 11JPT-4B66.951.177.1$0.043
  2. 12Jev-Omni63.959.576.9$0.022
  3. 13AutoJev-27B69.349.475.9$0.049
  4. 14Visual-Jev 4B Answer-SFT65.453.574.8$0.036
  5. 15CUA-S1 4B60.248.574.4$0.052
  6. 16Visual-Jev generic60.247.074.2$0.059
  7. 17imajev 2B57.754.274.1$0.034
  8. 18Reflex 4B63.349.473.5$0.049
  9. 19Standard One 8B52.848.372.4$0.053
  10. 20OmniJev 4B56.950.669.7$0.044
  11. 21Mapika decider-2b-vision BF1652.857.667.1$0.026
  12. 22Jevify Qwen3-VL-2B T247.359.467.1$0.023
  13. 23djev-spark NVFP449.145.966.7$0.063
  14. 24Glance59.455.566.2$0.030
  15. 25OmniJev 2B49.554.465.0$0.033
  16. 26Jev-Vision 8B64.250.063.7$0.046
  17. 27vjev-vision43.260.863.5$0.020
  18. 28jev-spatial52.059.662.5$0.022
  19. 29OmniJev-Qwen3.5-9B-v452.259.560.9$0.022
  20. 30Standard One 3B35.852.459.8$0.039
  21. 31JPT-0.8B36.357.557.2$0.026
  22. 32Jevify Gemma 4 E4B35.760.956.2$0.020
  23. 33OmniJev 0.8B34.655.455.5$0.031
  24. 34Gevva E2B multimodal32.151.054.0$0.043
  25. 35Gevva E4B35.949.152.9$0.050
  26. 36Qevi-2B29.558.642.6$0.024
  27. 37JEVision35.447.339.9$0.057
  28. 38PlayJev 0.8B4.459.424.4$0.023

Wide coloured bar = Capability Score (0–100). Thin bars: cost above, median latency below; shared log ratio scale 0.25× … 64× Jev 1.13.0 (JevBench), ticks at 1× and 2× (cap). Shorter is cheaper or faster. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ or hover a row or thin bar for details.

Cost and latency correlate here (Spearman ρ = 0.90, n = 50), mostly because self-hosted cost is computed from measured GPU time; both bars are kept for consistency with JevBench.

  • Closed API (weights not public) (6)
  • Open weights · LLM decoder (40)
  • Open weights · diffusion LM (4)
  • green: ≤ reference
  • amber: ≤ cap (2× reference)
  • red: > cap
Show general-purpose LLMs and other systems outside the limits

Sorted by Capability Score, not numbered. Each row says which limit it misses, measured against Jev 1.13.0 ($0.032 per 1,000 decisions, median 0.62 s).

  1. –GPT-5.6 LunaAPI91.823.593.8$0.35Outside: cost 10.9× Jev 1.13.0 (JevBench), latency 4.7× Jev 1.13.0 (JevBench)
  2. –Gemini 3.1 Flash LiteAPI84.322.387.9$0.39Outside: cost 12.0× Jev 1.13.0 (JevBench), latency 2.9× Jev 1.13.0 (JevBench)
  3. –GPT-6 Luna (low)API81.127.584.7$0.22Outside: cost 6.7× Jev 1.13.0 (JevBench), latency 4.7× Jev 1.13.0 (JevBench)
  4. –Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj74.929.482.8$0.23Outside: cost 7.0× Jev 1.13.0 (JevBench), latency 2.8× Jev 1.13.0 (JevBench)
  5. –Autoloops – Gemma 4 31B ITAPI79.342.680.8$0.082Outside: cost 2.5× Jev 1.13.0 (JevBench), latency 2.3× Jev 1.13.0 (JevBench)
  6. –Gemini 3.8 FlashAPI88.50.680.7$2.06Outside: cost 63.8× Jev 1.13.0 (JevBench), latency 7.8× Jev 1.13.0 (JevBench)
  7. –Winnow-12B67.441.376.8$0.090Outside: cost 2.8× Jev 1.13.0 (JevBench)
  8. –djev-distill-v452.545.168.7$0.068Outside: cost 2.1× Jev 1.13.0 (JevBench)
  9. –diffusiongemma-26b djev v10 step16057.244.665.7$0.070Outside: cost 2.2× Jev 1.13.0 (JevBench)
  10. –djev-dev BF1649.644.763.8$0.070Outside: cost 2.2× Jev 1.13.0 (JevBench)
  11. –OpenJev 4B NLI v567.241.133.6$0.092Outside: cost 2.8× Jev 1.13.0 (JevBench)
  12. –OpenJev 4B NLI v263.640.931.8$0.093Outside: cost 2.9× Jev 1.13.0 (JevBench)

Jev itself cannot read images, so there is no Jev row to anchor on. We use the same absolute budget as JevBench: at most 2× Jev 1.13.0's JevBench v1.5.4 cost (≤ USD 0.0646 per 1,000 decisions) and at most 2× its median latency (≤ 1.23 s, adjusted p50), frozen in this release. A decision model in this class should fit the same budget whether the input is text or an image. 38 of 50 systems qualify, a larger share than on JevBench, so we keep 2×. Loosening to 3× would add 7 open-weight fine-tunes that sit just above the cost line; the general-purpose hosted LLMs stay outside even at 3×. The charts below show speed and cost beside Capability Score; the official JevImageBench Score weighs all four axes.

Capability against cost and speed

Jev-class systems are shown by default. These charts use the official 2× caps. Bubble size follows the official Image JevBench Score; official rank stays unchanged. The five most capable Jev-class systems are labelled.

Capability vs cost

Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.

Capability vs cost: 38 systems. Upper right is best: more capable and cheaper.2030405060708090100$0.010$0.10$ per 1,000 decisions (log)Capability↑2× Jev 1.13.0 (JevBench)← priciercheaper →1. Wity-12. JPT-9B3. Imajev-4B4. shisa-de-15. NeoHorse Jev 4B
38 systems. Tap a bubble for its values.

Capability vs speed

Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). Self-hosted systems are plotted at the est. incl. network latency (2x measured + 150 ms); their measured loopback p50 (on-GPU, no network; not directly comparable with API latency) is in the tooltip. The dashed line is 2× Jev’s median latency.

Capability vs speed: 38 systems. Upper right is best: more capable and faster.203040506070809010070≈3.2 s80≈1.0 s90≈316 ms100≈100 msMedian-latency speedCapability↑2× Jev 1.13.0 (JevBench)← slowerfaster →1. Wity-12. JPT-9B3. Imajev-4B4. shisa-de-15. NeoHorse Jev 4B
38 systems. Tap a bubble for its values.
  • Closed API (weights not public) (6)
  • Open weights · LLM decoder (40)
  • Open weights · diffusion LM (4)
  • faint = outside Jev-class
Explore capability, cost and speed in 3D

Image JevBench v0.1.5

Image JevBench Score: 50 ranked systems

Official· four axes 0–100, equal-weight harmonic mean · Method notes ↓

Weights:
Adjust weights ↓

View by:Capability ↑

The official order weighs Intelligence, Calibration, Speed and Cost equally. Each button re-sorts the same systems by one axis.

Greener = stronger within its column.

50 of 50 systems, sorted by official rank, #1 first.

  1. 1Wity-1APIdetailsClosed APIBase model: undisclosed*1Base-model price: 74.4 (would be #2)80.2I 84C 88S 77K 74ann.$0.0074
  2. 2Imajev-4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource76.4I 74C 91S 88K 61$0.020
  3. 3Jev-Omnidetailsfine-tune · 12BBase model: Gemma 4 12B ITsource73.1I 64C 90S 90K 60$0.022
  4. 4NeoHorse Jev 4Bdetails4BBase model: NeoHorse-1-4Bsource71.9I 73C 91S 87K 52$0.041
  5. 5Visual-Jev 4B Answer-SFTdetailsadapter · 4BBase model: Qwen/Qwen3-VL-4B-Instructsource69.8I 65C 84S 88K 53$0.036
  6. 6JPT-4Bdetailsadapter · 4BBase model: Qwen3.5-4Bsource69.5I 67C 87S 87K 51$0.043
  7. 7imajev 2Bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2Bsource68.7I 58C 90S 88K 54$0.034
  8. 8Glancedetails4BBase model: Qwen3-VL-4Bsource66.9I 59C 73S 89K 56$0.030
  9. 9AutoJev-27Bdetailsfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsource66.8I 69C 82S 87K 49$0.049
  10. 10Mapika decider-2b-vision BF16detailsfine-tune · 2BBase model: Qwen/Qwen3.5-2B-Basesource66.8I 53C 81S 88K 58$0.026
  11. 11jev-spatialdetailsadapterBase model: allenai/Molmo2-ERsource65.9I 52C 73S 91K 60$0.022
  12. 12JPT-9Bdetailsadapter · 9BBase model: Qwen3.5-9Bsource65.8I 76C 92S 85K 48$0.053
  13. 13OmniJev 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource65.7I 57C 83S 87K 51$0.044
  14. 14Reflex 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource65.4I 63C 84S 86K 49$0.049
  15. 15OmniJev-Qwen3.5-9B-v4detailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource65.1I 52C 70S 90K 60$0.022
  16. 16Jevify Gemma 4 26B-A4Bdetailsadapter · 26B-A4BBase model: google/gemma-4-26B-A4B-itsource65.0I 72C 88S 86K 48$0.053
  17. 17Surogate Rune 26B-A4B v3details26B-A4BBase model: google/gemma-4-26B-A4B-itsource63.8I 75C 88S 86K 48$0.055
  18. 18JevAny-27B SFTdetailsadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 263.8I 70C 90S 86K 48$0.054
  19. 19JevAny-27B RLCRdetailsadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 263.5I 70C 90S 86K 48$0.055
  20. 20Jev-Vision 8Bdetailsadapter · 8BBase model: Qwen/Qwen3-VL-8B-Instructsource63.4I 64C 63S 86K 50$0.046
Show all 50 systems (30 more ranked)
  1. 21shisa-de-1details26B-A4BBase model: Gemma 4 26B-A4Bsource63.2I 72C 92S 86K 48$0.056
  2. 22OmniJev 2Bdetailsadapter · 2BBase model: Qwen/Qwen3.5-2Bsource62.8I 49C 81S 89K 54$0.033
  3. 23CUA-S1 4Bdetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource62.6I 60C 89S 85K 49$0.052
  4. 24Standard One 8Bdetailsadapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source59.8I 53C 92S 85K 48$0.053
  5. 25Jevify Qwen3-VL-2B T2detailsadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsource59.0I 47C 87S 90K 59$0.023
  6. 26imajev 9Bdetailsadapter · 9BBase model: Qwen/Qwen3.5-9Bsource58.3I 75C 89S 84K 46$0.063
  7. 27Visual-Jev genericdetails4BBase model: undisclosed57.7I 60C 88S 84K 47$0.059
  8. 28djev-distill-v4detailsdistilled · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource50.0I 53C 85S 83K 45$0.068
  9. 29djev-spark NVFP4details26B-A4B · NVFP4Base model: google/diffusiongemma-26B-A4B-itsource 1source 249.4I 49C 84S 84K 46$0.063
  10. 30diffusiongemma-26b djev v10 step160detailsfine-tune · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource48.8I 57C 74S 83K 45$0.070
  11. 31vjev-visiondetailsadapter · 4BBase model: Qwen/Qwen3.5-4Bsource47.7I 43C 84S 91K 61$0.020
  12. 32Autoloops – Gemma 4 31B ITAPIdetailsClosed APIBase model: undisclosed47.1I 79C 82S 74K 43ann.$0.082
  13. 33djev-dev BF16details26B-A4BBase model: google/diffusiongemma-26B-A4B-itsource46.6I 50C 78S 83K 45$0.070
  14. 34Winnow-12Bdetailsmerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsource43.6I 67C 86S 82K 41$0.090
  15. 35JPT-0.8Bdetailsadapter · 0.8BBase model: Qwen3.5-0.8Bsource30.5I 36C 78S 89K 58$0.026
  16. 36Jevify Gemma 4 E4Bdetailsadapter · 4BBase model: google/gemma-4-E4B-itsource29.8I 36C 77S 91K 61$0.020
  17. 37Standard One 3Bdetailsadapter · 3BBase model: mistralai/Ministral-3-3B-Instruct-2512-BF16source29.0I 36C 84S 87K 52$0.039
  18. 38Gevva E4Bdetails4BBase model: Gemma 4 E4Bsource26.8I 36C 70S 86K 49$0.050
  19. 39OmniJev 0.8Bdetailsadapter · 0.8BBase model: Qwen/Qwen3.5-0.8Bsource26.8I 35C 76S 89K 55$0.031
  20. 40Gevva E2B multimodaldetails2BBase model: Gemma 4 E2Bsource21.9I 32C 76S 87K 51$0.043
  21. 41JEVisiondetails0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource21.5I 35C 44S 85K 47$0.057
  22. 42Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjdetailsLLM decoderBase model: undisclosedsource19.3I 75C 91S 74K 29$0.225
  23. 43Qevi-2Bdetailsadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsource17.4I 30C 56S 89K 59$0.024
  24. 44GPT-6 LunaAPIdetailsClosed APIBase model: undisclosed16.1I 81C 88S 66K 27$0.217
  25. 45GPT-5.6 LunaAPIdetailsClosed APIBase model: undisclosed11.0I 92C 96S 63K 23$0.352
  26. 46Gemini 3.1 Flash LiteAPIdetailsClosed APIBase model: undisclosed9.6I 84C 91S 64K 22$0.389
  27. 47PlayJev 0.8Bdetailsfine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesource0.1I 4C 44S 90K 59$0.023
  28. 48Gemini 3.8 FlashAPIdetailsClosed APIBase model: undisclosed0.0I 88C 73S 59K 1$2.060
  29. 49OpenJev 4B NLI v2details4BBase model: Qwen/Qwen3.5-4Bsource0.0I 64C 0S 80K 41$0.093
  30. 50OpenJev 4B NLI v5details4BBase model: Qwen/Qwen3.5-4Bsource0.0I 67C 0S 79K 41$0.092

Base-model notes

  1. *1 Wity-1 (base model: undisclosed): Base model: undisclosed (author request).
Weights:
Adjust weights ↓

Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; the gates still apply when an axis sits at 0. Only equal weights give the official Image JevBench Score and rank.

Score = 4 / (1/I + 1/C + 1/S + 1/K) (each 0–100; × (axis / 50)² for Intelligence, Speed or Cost below 50)

  • Closed API (weights not public) (6)
  • Open weights · LLM decoder (40)
  • Open weights · diffusion LM (4)
  • Striped bar = same system under the labelled alternative price assumption
I, C, S, K = Intelligence, Calibration, Speed, Cost; the est. pill = estimated cost; ann. = announced price; API = the operator's endpoint saw held-out benchmark inputs, without answers; $/1k decisions = US dollars per 1,000 decisions (not heat-shaded). Names link to each project.

Wity-1 is ranked at Wity's own stated API price (USD 0.042 per million input tokens, output free). The striped bar shows the score at the base-model reference price we use for self-hosted open weights of the same base (base model undisclosed at the author's request). See pricing note ↓

Compare two systems

Pick any two. Compare the four score axes, capability by image type and use cases; request-type and tier aggregates are not published for this benchmark. Further out is better on every spoke; the link keeps the pair.

  • A: Wity-1 — Closed API (weights not public) · Score 80.2 (#1)
  • B: Imajev-4B — Open weights · LLM decoder · Score 76.4 (#2)

The four score axes

Radar: the four score axes, two systemsThe four score axes, Wity-1 vs Imajev-4B. Intelligence: 83.7 vs 73.8; Calibration: 87.7 vs 90.5; Speed: 76.9 vs 87.6; Cost: 73.9 vs 61.2.50100Intelligence: Wity-1: 83.7; Imajev-4B: 73.8Intelligence83.7 · 73.8Calibration: Wity-1: 87.7; Imajev-4B: 90.5Calibration87.7 · 90.5Speed: Wity-1: 76.9; Imajev-4B: 87.6Speed76.9 · 87.6Cost: Wity-1: 73.9; Imajev-4B: 61.2Cost73.9 · 61.2
0–100, the values in the table. An axis a system has no published value for is left as a gap (it counts as 0 in the composite).

Capability by image type

Radar: capability by image type, two systemsCapability by image type, Wity-1 vs Imajev-4B. Everyday photos: 90.6 vs 86.5; Documents: 83.5 vs 49.1; Charts: 96.4 vs 87.4; Inventory: 76.7 vs 83.7; Safety inspection: 82.5 vs 76.7; Screens & apps: 78.9 vs 71.4; Web tasks: 79.4 vs 69.9. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Everyday photos: Wity-1: 90.6; Imajev-4B: 86.5 — Everyday photos: yes/no and counting questions about ordinary photos: damage, safety, shelves, signs. 184 items (89 public, 95 sealed).Everydayphotos90.6 · 86.5Documents: Wity-1: 83.5; Imajev-4B: 49.1 — Documents: reading fields and facts from scanned or rendered business documents. 102 items (0 public, 102 sealed).Documents83.5 · 49.1Charts: Wity-1: 96.4; Imajev-4B: 87.4 — Charts: reading values and trends from charts. 79 items (0 public, 79 sealed).Charts96.4 · 87.4Inventory: Wity-1: 76.7; Imajev-4B: 83.7 — Inventory: counting and checking stock on shelves and in storage. 61 items (0 public, 61 sealed).Inventory76.7 · 83.7Safety inspection: Wity-1: 82.5; Imajev-4B: 76.7 — Safety inspection: spotting hazards and missing protective equipment. 57 items (0 public, 57 sealed).Safetyinspection82.5 · 76.7Screens & apps: Wity-1: 78.9; Imajev-4B: 71.4 — Screens & apps: finding the right element on desktop, web and phone screenshots. 83 items (21 public, 62 sealed).Screens &apps78.9 · 71.4Web tasks: Wity-1: 79.4; Imajev-4B: 69.9 — Web tasks: choosing the next element for a multi-step browser goal. 79 items (79 public, 0 sealed).Web tasks79.4 · 69.9

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

The kind of image and task: photos, documents, charts, screens, web tasks, tables, geometry. Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Wity-1: public+sealed; Imajev-4B: public+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.

Low sample, n < 30 — indicative only

Category (items)A: Wity-1B: Imajev-4B
Financial tables (20)51.7 n=2037.9 n=20
Geometry (19)86.0 n=1979.0 n=19

Not drawn on the radar: with so few items a single answer moves a category score by several points, so these values are noise-prone. These values become spokes once the item pool reaches 30 items per category.

What each category means · items per category
  • Everyday photos — yes/no and counting questions about ordinary photos: damage, safety, shelves, signs. 184 items (89 public / 95 sealed)
  • Documents — reading fields and facts from scanned or rendered business documents. 102 items (0 public / 102 sealed)
  • Charts — reading values and trends from charts. 79 items (0 public / 79 sealed)
  • Inventory — counting and checking stock on shelves and in storage. 61 items (0 public / 61 sealed)
  • Safety inspection — spotting hazards and missing protective equipment. 57 items (0 public / 57 sealed)
  • Screens & apps — finding the right element on desktop, web and phone screenshots. 83 items (21 public / 62 sealed)
  • Web tasks — choosing the next element for a multi-step browser goal. 79 items (79 public / 0 sealed)

Use cases (TypeSafe categories)

Radar: use cases (typesafe categories), two systemsUse cases (TypeSafe categories), Wity-1 vs Imajev-4B. Customer support: 76.3 vs 48.3; Insurance claims: 94.4 vs 97.2; Legal & compliance: 73.5 vs 68.5; E-commerce: 84.0 vs 78.3; Risk assessment: 90.2 vs 73.8; Demand forecasting: 96.8 vs 93.5. Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.−1000100Customer support: Wity-1: 76.3; Imajev-4B: 48.3 — Customer support: classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 78 items (19 public, 59 sealed).Customersupport76.3 · 48.3Insurance claims: Wity-1: 94.4; Imajev-4B: 97.2 — Insurance claims: first-notice-of-loss reports, claim complexity, missing information, claim triage. 64 items (20 public, 44 sealed).Insuranceclaims94.4 · 97.2Legal & compliance: Wity-1: 73.5; Imajev-4B: 68.5 — Legal and compliance: contracts, policies, regulations, eligibility rules, checking documents against written requirements. 138 items (28 public, 110 sealed).Legal &compliance73.5 · 68.5E-commerce: Wity-1: 84.0; Imajev-4B: 78.3 — E-commerce marketplaces: product listings, product attributes, orders and deliveries, seller catalog policy. 128 items (36 public, 92 sealed).E-commerce84.0 · 78.3Risk assessment: Wity-1: 90.2; Imajev-4B: 73.8 — Risk assessment: estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 106 items (27 public, 79 sealed).Riskassessment90.2 · 73.8Demand forecasting: Wity-1: 96.8; Imajev-4B: 93.5 — Demand forecasting: purchase intent, product interest, demand and supply signals for forecasting. 41 items (0 public, 41 sealed).Demandforecasting96.8 · 93.5

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

The real-world application area of each question, after the TypeSafe use-case map; an item can count in two. Chance-corrected competence per category (0 = chance, below chance negative, 100 = perfect), Wity-1: public+sealed; Imajev-4B: public+sealed items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.

Low sample, n < 30 — indicative only

Category (items)A: Wity-1B: Imajev-4B
Search and retrieval (19)93.4 n=1986.8 n=19

No published value for either system (under 15 answered items, or no per-category values): Semantic code linting (1), Feature extraction for predictive modeling (2), Recruiting (1), Financial crime (5), Moderation and trust and safety (12), Advertising (4), Graphs and knowledge graphs (1).

Not drawn on the radar: with so few items a single answer moves a category score by several points, so these values are noise-prone. These values become spokes once the item pool reaches 30 items per category.

What each category means · items per category
  • Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 78 items (19 public / 59 sealed)
  • Insurance claims — first-notice-of-loss reports, claim complexity, missing information, claim triage. 64 items (20 public / 44 sealed)
  • Legal and compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 138 items (28 public / 110 sealed)
  • E-commerce marketplaces — product listings, product attributes, orders and deliveries, seller catalog policy. 128 items (36 public / 92 sealed)
  • Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 106 items (27 public / 79 sealed)
  • Demand forecasting — purchase intent, product interest, demand and supply signals for forecasting. 41 items (0 public / 41 sealed)

Not drawn: Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 307 items (141 public / 166 sealed) — not a use case of its own, so it is counted but not drawn.

No items in this release: Scientific discovery, Model routing, LLM guardrails, Lead generation, Gaming.

How the categories were made

Chance-corrected accuracy (0 = chance, 100 = perfect) over the category's items, public and sealed pooled. Not part of the score; compare systems within a category.

Capabilities are the item source families (fixed, no model). Use cases: each item's question and options (not the image) were labelled with one or two TypeSafe use-case categories by Winnow-12B Q8 on our own GPU pod; about 5 % of public items were checked by hand.

  • A second use case is kept when the labeller gave it probability >= 0.15 and it is not Other.
All values as a table
SpokeA: Wity-1B: Imajev-4B
The four score axes
Intelligence83.773.8
Calibration87.790.5
Speed76.987.6
Cost73.961.2
Capability by image type
Everyday photos90.686.5
Documents83.549.1
Charts96.487.4
Inventory76.783.7
Safety inspection82.576.7
Screens & apps78.971.4
Web tasks79.469.9
Use cases (TypeSafe categories)
Customer support76.348.3
Insurance claims94.497.2
Legal & compliance73.568.5
E-commerce84.078.3
Risk assessment90.273.8
Demand forecasting96.893.5

Full ranking

Ranked by the composite score: equal-weight Intelligence, Calibration, Speed and Cost axes, then the unchanged Jev-class gates. Matched gap is signed public-minus-sealed accuracy within the matched families. No system currently exceeds the 15 pp allowance. Hosted systems are marked API because their providers received sealed images and questions; the Gemma 4 endpoint is identified separately in the exposure note.

#SystemCompositeIntelligenceCalibrationSpeedCostGap (matched)Earlier splitPublic accuracySealed accuracyUSD / 1,000p50 / p95
1Wity-1

Under author review: server build ID was not recorded; this score may change after verification.

Setting: Wity SystemOne, reasoning=auto

APIClosed APIBase model: undisclosed · Base model: undisclosed (author request).details		80.2083.6887.7176.9573.92+2.1 pp—200/228 · 87.7%410/456 · 89.9%USD 0.00740.710 s / 2.844 s
2Imajev-4B

Setting: PyTorch; one rotation; --fast; --merge-lora; calibration.json; server 8501f5c3; adapter c9e5f132.

adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails		76.3973.7790.5287.5961.18-4.2 pp#1 · 76.39185/228 · 81.1%380/456 · 83.3%USD 0.01970.099 s / 0.174 s
3Jev-Omnifine-tune · 12BBase model: Gemma 4 12B ITsourcedetails73.1063.9389.9089.6859.52-1.0 pp#2 · 73.10153/228 · 67.1%368/456 · 80.7%USD 0.02240.076 s / 0.103 s
4NeoHorse Jev 4B4BBase model: NeoHorse-1-4Bsourcedetails71.9472.8991.2086.7651.56-0.8 pp#3 · 71.94184/228 · 80.7%377/456 · 82.7%USD 0.04120.140 s / 0.170 s
5Visual-Jev 4B Answer-SFTadapter · 4BBase model: Qwen/Qwen3-VL-4B-Instructsourcedetails69.8165.4384.1387.5253.48-6.2 pp#4 · 69.81175/228 · 76.8%352/456 · 77.2%USD 0.03550.101 s / 0.176 s
6JPT-4Bkirp / llm2jevadapter · 4BBase model: Qwen3.5-4Bsourcedetails69.5566.9287.3186.5251.13+3.3 pp#5 · 69.55172/228 · 75.4%362/456 · 79.4%USD 0.04260.123 s / 0.206 s
7imajev 2Badapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails68.7257.7390.4687.8554.19-1.1 pp#6 · 68.72164/228 · 71.9%328/456 · 71.9%USD 0.03360.096 s / 0.164 s
8Glancefrozen Qwen3-VL-4B4BBase model: Qwen3-VL-4Bsourcedetails66.8659.3772.9788.7355.54-2.8 pp#7 · 66.86177/228 · 77.6%322/456 · 70.6%USD 0.03030.089 s / 0.129 s
9AutoJev-27Bfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsourcedetails66.8569.3382.4386.7949.37+0.1 pp#8 · 66.85174/228 · 76.3%371/456 · 81.4%USD 0.04870.141 s / 0.167 s
10Mapika decider-2b-vision BF16fine-tune · 2BBase model: Qwen/Qwen3.5-2B-Basesourcedetails66.7752.8481.4688.1757.59-4.4 pp#9 · 66.77151/228 · 66.2%319/456 · 70.0%USD 0.02590.091 s / 0.154 s
11jev-spatialFr0zencr4nEadapterBase model: allenai/Molmo2-ERsourcedetails65.8752.0473.0490.5259.63-0.7 pp#10 · 65.87159/228 · 69.7%307/456 · 67.3%USD 0.02220.058 s / 0.092 s
12JPT-9Bkirp / llm2jevadapter · 9BBase model: Qwen3.5-9Bsourcedetails65.8275.7491.5485.1848.25+1.4 pp#11 · 65.82187/228 · 82.0%387/456 · 84.9%USD 0.05310.155 s / 0.255 s
13OmniJev 4Btinnel123adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails65.6556.8582.6187.0650.65-3.0 pp#12 · 65.65163/228 · 71.5%325/456 · 71.3%USD 0.04420.131 s / 0.164 s
14Reflex 4Breleased stable configurationadapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails65.3863.3483.6886.1449.35-0.6 pp#13 · 65.38184/228 · 80.7%333/456 · 73.0%USD 0.04880.154 s / 0.191 s
15OmniJev-Qwen3.5-9B-v4tzcflyadapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails65.1452.1769.5990.3459.54+0.1 pp#14 · 65.14149/228 · 65.4%318/456 · 69.7%USD 0.02230.063 s / 0.093 s
16Jevify Gemma 4 26B-A4Badapter · 26B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails65.0071.6487.9786.2448.37-1.2 pp#15 · 65.00189/228 · 82.9%366/456 · 80.3%USD 0.05260.156 s / 0.182 s
17Surogate Rune 26B-A4B v326B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails63.8274.9187.7785.9347.81+0.9 pp#16 · 63.82191/228 · 83.8%379/456 · 83.1%USD 0.05490.163 s / 0.193 s
18JevAny-27B SFTadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details63.7969.5790.0286.1148.05+4.7 pp#17 · 63.79177/228 · 77.6%369/456 · 80.9%USD 0.05390.161 s / 0.185 s
19JevAny-27B RLCRadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details63.4670.2389.9986.0647.90+4.7 pp#18 · 63.46178/228 · 78.1%371/456 · 81.4%USD 0.05450.162 s / 0.186 s
20Jev-Vision 8BSeanLiuadapter · 8BBase model: Qwen/Qwen3-VL-8B-Instructsourcedetails63.3964.1863.1485.8649.99-1.0 pp#19 · 63.39181/228 · 79.4%340/456 · 74.6%USD 0.04650.149 s / 0.215 s
21shisa-de-126B-A4BBase model: Gemma 4 26B-A4Bsourcedetails63.2072.4491.8285.8547.60+1.7 pp#20 · 63.20182/228 · 79.8%377/456 · 82.7%USD 0.05580.165 s / 0.195 s
22OmniJev 2Btinnel123adapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails62.8449.4780.5388.5354.40-3.9 pp#21 · 62.84163/228 · 71.5%291/456 · 63.8%USD 0.03310.097 s / 0.130 s
23CUA-S1 4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails62.6360.1788.5785.3948.55+0.5 pp#22 · 62.63170/228 · 74.6%333/456 · 73.0%USD 0.05190.150 s / 0.246 s
24Standard One 8Badapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16sourcedetails59.8052.7792.0684.9148.26-1.6 pp#23 · 59.80168/228 · 73.7%301/456 · 66.0%USD 0.05300.142 s / 0.298 s
25Jevify Qwen3-VL-2B T2adapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails59.0247.3186.8389.5259.38+1.6 pp#24 · 59.02164/228 · 71.9%280/456 · 61.4%USD 0.02260.062 s / 0.129 s
26imajev 9Badapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails58.3374.7589.2784.1346.06+1.3 pp#25 · 58.33197/228 · 86.4%372/456 · 81.6%USD 0.06280.188 s / 0.292 s
27Visual-Jev genericzero-shot Qwen3.5-4B baseline4BBase model: undiscloseddetails57.7360.2488.0983.9846.97+0.3 pp#26 · 57.73178/228 · 78.1%325/456 · 71.3%USD 0.05860.179 s / 0.319 s
28djev-distill-v4tarsur385distilled · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails50.0452.5484.8182.9845.11+7.9 pp#27 · 50.04142/228 · 62.3%327/456 · 71.7%USD 0.06760.195 s / 0.392 s
29djev-spark NVFP426B-A4B · NVFP4Base model: google/diffusiongemma-26B-A4B-itsource 1source 2details49.3849.1084.3083.5145.95+3.4 pp#28 · 49.38146/228 · 64.0%307/456 · 67.3%USD 0.06330.173 s / 0.374 s
30diffusiongemma-26b djev v10 step160snowicarusfine-tune · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails48.7657.1974.3082.8044.65+2.6 pp#29 · 48.76152/228 · 66.7%338/456 · 74.1%USD 0.07000.203 s / 0.397 s
31vjev-visionyah01adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails47.6743.1883.8090.6260.78-2.2 pp#30 · 47.67140/228 · 61.4%286/456 · 62.7%USD 0.02030.058 s / 0.088 s
32Autoloops – Gemma 4 31B ITAPIClosed APIBase model: undiscloseddetails47.1479.2782.3873.8942.65+0.7 pp#31 · 47.14193/228 · 84.6%397/456 · 87.1%USD 0.08161.425 s / 2.868 s
33djev-dev BF1626B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails46.6449.6078.0283.0044.69+6.6 pp#32 · 46.64153/228 · 67.1%302/456 · 66.2%USD 0.06980.193 s / 0.392 s
34Winnow-12BQ8_0 + F16 vision projectormerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsourcedetails43.5667.4086.2382.1241.35+2.9 pp#33 · 43.56177/228 · 77.6%359/456 · 78.7%USD 0.09020.268 s / 0.372 s
35JPT-0.8Bkirp / llm2jevadapter · 0.8BBase model: Qwen3.5-0.8Bsourcedetails30.5136.2778.0689.1457.50-0.3 pp#34 · 30.51120/228 · 52.6%275/456 · 60.3%USD 0.02610.074 s / 0.129 s
36Jevify Gemma 4 E4Badapter · 4BBase model: google/gemma-4-E4B-itsourcedetails29.7635.7076.6890.6760.86-1.7 pp#35 · 29.76129/228 · 56.6%263/456 · 57.7%USD 0.02020.057 s / 0.088 s
37Standard One 3Badapter · 3BBase model: mistralai/Ministral-3-3B-Instruct-2512-BF16sourcedetails29.0135.7683.9086.5252.36-1.1 pp#36 · 29.01136/228 · 59.6%256/456 · 56.1%USD 0.03870.100 s / 0.244 s
38Gevva E4Btext checkpoint, image input4BBase model: Gemma 4 E4Bsourcedetails26.8335.9469.9185.8249.06+1.4 pp#37 · 26.83132/228 · 57.9%261/456 · 57.2%USD 0.04990.158 s / 0.206 s
39OmniJev 0.8Btinnel123adapter · 0.8BBase model: Qwen/Qwen3.5-0.8Bsourcedetails26.8134.5776.4788.9555.39+4.4 pp#38 · 26.81148/228 · 64.9%238/456 · 52.2%USD 0.03070.088 s / 0.120 s
40Gevva E2B multimodal2BBase model: Gemma 4 E2Bsourcedetails21.8532.1075.9786.7751.03-3.5 pp#39 · 21.85115/228 · 50.4%261/456 · 57.2%USD 0.04290.132 s / 0.179 s
41JEVisiondivyanshx11, visual route0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails21.5035.4344.4085.4547.30+2.3 pp#40 · 21.50123/228 · 53.9%268/456 · 58.8%USD 0.05710.170 s / 0.216 s
42Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjLLM decoderBase model: undisclosedsourcedetails19.2674.9190.7173.6229.43+1.9 pp#41 · 19.26191/228 · 83.8%379/456 · 83.1%USD 0.22520.799 s / 1.167 s
43Qevi-2BMeerDevelopmentadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails17.4429.5455.6789.3758.64+0.1 pp#42 · 17.44144/228 · 63.2%219/456 · 48.0%USD 0.02390.068 s / 0.127 s
44GPT-6 Lunalow reasoning effort

Setting: OpenRouter reasoning.effort=low

Cost receipts cover 91.8% of calls

APIClosed APIBase model: undiscloseddetails		16.0681.1088.3165.7027.48+12.0 pp#43 · 16.06203/228 · 89.0%395/456 · 86.6%USD 0.21662.905 s / 9.256 s
45GPT-5.6 Luna

Cost receipts cover 99.6% of calls

APIClosed APIBase model: undiscloseddetails		11.0591.8195.7862.5823.49-0.8 pp#44 · 11.05211/228 · 92.5%436/456 · 95.6%USD 0.35242.878 s / 19.176 s
46Gemini 3.1 Flash LiteAPIClosed APIBase model: undiscloseddetails9.5884.2891.4464.4922.31-0.2 pp#45 · 9.58194/228 · 85.1%419/456 · 91.9%USD 0.38881.799 s / 19.775 s
47PlayJev 0.8Bgated by Intelligence (below the Jev-class floor)fine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails0.114.4244.4789.6959.35-1.0 pp#46 · 0.1180/228 · 35.1%170/456 · 37.3%USD 0.02260.067 s / 0.115 s
48Gemini 3.8 Flashgated by Cost (USD 2.06 per 1,000 decisions)APIClosed APIBase model: undiscloseddetails0.0088.4772.9858.780.58+0.8 pp#47 · 0.00202/228 · 88.6%430/456 · 94.3%USD 2.05994.832 s / 27.423 s
49OpenJev 4B NLI v2official image-premise pathgated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.0063.580.0079.5440.89+3.9 pp#48 · 0.00187/228 · 82.0%331/456 · 72.6%USD 0.09340.317 s / 0.634 s
50OpenJev 4B NLI v5gated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.0067.210.0079.4941.09+0.4 pp#49 · 0.00180/228 · 78.9%355/456 · 77.9%USD 0.09200.321 s / 0.635 s

The frozen benchmark has 684 scored items: 228 public and 456 sealed; 93 further items are retired and not scored. This page shows aggregate sealed results only. It contains no sealed task, image, answer key, or per-item prediction.

Wity-1 under author review. The run used the listed production endpoint, but its response did not identify the deployed build. The author is checking the build; this score may change after a full rerun.

Caveat: The 333 fresh sealed items are our own synthetic images and renders. They are easier for frontier API models than the older real-source items, so sealed accuracy is higher than on the earlier split for most hosted systems.

Examples from public items

These eight examples are from the public split. Each card shows the image, question, options and correct answer; licensed sources are credited below their image.

A cardboard parcel with a visibly torn corner on a conveyor.

Everyday photo

Parcel condition

Question: Is the parcel visibly damaged?

  • A. yesCorrect
  • B. no

Correct answer: A. yes

Source: Synthetic image created for ImageJevBench · public item promo-parcel-01

A printed cafe receipt with item names, amounts, tax and total visible.

Everyday photo

Receipt legibility

Question: Is the receipt readable?

  • A. yesCorrect
  • B. no

Correct answer: A. yes

Source: Synthetic image created for ImageJevBench · public item promo-receipt-00

A spreadsheet with a number-format dialog and five labelled target markers.

Computer use · spreadsheet

Change a cell format

Question: Goal: change selected cells to type “Text”. Which labelled marker should be clicked?

  • A. Click marker A
  • B. Click marker BCorrect
  • C. Click marker C
  • D. Click marker D
  • E. Click marker E

Correct answer: B. Click marker B

Source: ScreenSpot-Pro · excel_macos_0 · MIT · 210e78d38442 · public item mm-195 · Licence text: MITAdapted: five labelled click markers were drawn on the source screenshot.

A desktop browser showing a sports page, browser tabs and five labelled markers.

Browser

Open a new tab

Question: Goal: open a new tab. Which labelled marker should be clicked?

  • A. Click marker A
  • B. Click marker B
  • C. Click marker C
  • D. Click marker D
  • E. Click marker ECorrect

Correct answer: E. Click marker E

Source: ScreenSpot · item 15 · Apache-2.0 · 0be08781e2e1 · public item mm-135 · Licence text: Apache-2.0Adapted: five labelled click markers were drawn on the source screenshot.

A mobile translation interface with “Hello, world” and five labelled target markers.

Mobile app · translation

Use the translate control

Question: Goal: translate. Which labelled marker should be clicked?

  • A. Click marker A
  • B. Click marker B
  • C. Click marker C
  • D. Click marker DCorrect
  • E. Click marker E

Correct answer: D. Click marker D

Source: ScreenSpot · item 341 · Apache-2.0 · 0be08781e2e1 · public item mm-118 · Licence text: Apache-2.0Adapted: five labelled click markers were drawn on the source screenshot.

A Windows file-manager context menu with five labelled target markers.

Computer use · file manager

Copy a file

Question: Goal: copy the file. Which labelled marker should be clicked?

  • A. Click marker ACorrect
  • B. Click marker B
  • C. Click marker C
  • D. Click marker D
  • E. Click marker E

Correct answer: A. Click marker A

Source: ScreenSpot · item 19 · Apache-2.0 · 0be08781e2e1 · public item mm-139 · Licence text: Apache-2.0Adapted: five labelled click markers were drawn on the source screenshot.

A compact financial table with 2014 and 2015 net revenue values.

FinQA · financial table

Calculate a revenue change

Question: What is the net change in net revenue during 2015 for Entergy Corporation?

  • A. 103.4
  • B. 84.6
  • C. 94Correct
  • D. 112.8

Correct answer: C. 94

Source: FinQA · table item · MIT annotations; CDLA-Permissive-1.0 table data · 3d6a736bc67e · public item mm-061 · Licence text: MIT, CDLA-Permissive-1.0Table data: IBM FinTabNet (CDLA-Permissive-1.0). The table was re-rendered as an image by ImageJevBench; no original filing page is shown.

Screenshot images are adapted from the credited datasets (labelled markers added); the FinQA table is re-rendered by ImageJevBench. Licences: Apache-2.0 (ScreenSpot), MIT (ScreenSpot-Pro, Geometry3K, FinQA annotations), CDLA-Permissive-1.0 (FinTabNet table data). App and website content shown in screenshots belongs to its respective owners. No Mind2Web or Android-in-the-Wild image is shown.

Results by track

Each track is ranked on its own public and sealed items. The licensed core and synthetic everyday-photo results remain separately visible.

Core · 500 items

139 public · 361 sealed: 62 real-source items and 299 fresh synthetic pool items (documents, charts, inventory, safety).

#SystemCompositeIntelligenceCalibrationSpeedCostPublic accuracySealed accuracyUSD / 1,000
1Wity-1

Under author review: server build ID was not recorded; this score may change after verification.

Setting: Wity SystemOne, reasoning=auto

APIClosed APIBase model: undisclosed · Base model: undisclosed (author request).details		78.6381.5384.9476.4572.70114/139 · 82.0%321/361 · 88.9%USD 0.0081
2Imajev-4B

Setting: PyTorch; one rotation; --fast; --merge-lora; calibration.json; server 8501f5c3; adapter c9e5f132.

adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails		75.4970.2386.0388.0263.44104/139 · 74.8%290/361 · 80.3%USD 0.0165
3NeoHorse Jev 4B4BBase model: NeoHorse-1-4Bsourcedetails70.8569.6486.1587.2452.54103/139 · 74.1%289/361 · 80.1%USD 0.0382
4JPT-9Bkirp / llm2jevadapter · 9BBase model: Qwen3.5-9Bsourcedetails70.0569.9288.1185.7450.5499/139 · 71.2%295/361 · 81.7%USD 0.0445
5Visual-Jev 4B Answer-SFTadapter · 4BBase model: Qwen/Qwen3-VL-4B-Instructsourcedetails69.7562.1084.0987.9655.6099/139 · 71.2%265/361 · 73.4%USD 0.0302
6Jev-Omnifine-tune · 12BBase model: Gemma 4 12B ITsourcedetails69.7155.9687.3289.6759.1771/139 · 51.1%276/361 · 76.5%USD 0.0230
7JPT-4Bkirp / llm2jevadapter · 4BBase model: Qwen3.5-4Bsourcedetails67.9660.3283.0986.9453.3487/139 · 62.6%273/361 · 75.6%USD 0.0359
8imajev 2Badapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails66.7051.8686.5888.2456.1785/139 · 61.2%243/361 · 67.3%USD 0.0289
9Reflex 4Breleased stable configurationadapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails65.7559.2282.5286.5049.90103/139 · 74.1%249/361 · 69.0%USD 0.0468
10Glancefrozen Qwen3-VL-4B4BBase model: Qwen3-VL-4Bsourcedetails65.0455.3270.4388.6355.7499/139 · 71.2%239/361 · 66.2%USD 0.0299
11CUA-S1 4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails64.2752.9183.3285.8950.7991/139 · 65.5%245/361 · 67.9%USD 0.0437
12AutoJev-27Bfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsourcedetails63.9662.2779.8086.6449.1789/139 · 64.0%278/361 · 77.0%USD 0.0495
13imajev 9Badapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails63.9669.8785.8984.7948.33111/139 · 79.9%280/361 · 77.6%USD 0.0528
14Jevify Gemma 4 26B-A4Badapter · 26B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails63.2166.9084.2786.2348.32105/139 · 75.5%276/361 · 76.5%USD 0.0528
15OmniJev 4Btinnel123adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails63.0750.4980.6287.0550.6984/139 · 60.4%239/361 · 66.2%USD 0.0440
16Surogate Rune 26B-A4B v326B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails62.2970.9383.8185.9047.77107/139 · 77.0%289/361 · 80.1%USD 0.0551
17Standard One 8Badapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16sourcedetails61.9049.6083.8485.5250.4595/139 · 68.3%222/361 · 61.5%USD 0.0448
18shisa-de-126B-A4BBase model: Gemma 4 26B-A4Bsourcedetails61.8868.3690.4685.8347.5299/139 · 71.2%289/361 · 80.1%USD 0.0562
19JevAny-27B SFTadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details60.2656.9388.7686.1048.0390/139 · 64.7%276/361 · 76.5%USD 0.0540
20JevAny-27B RLCRadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details59.9757.7188.1286.0547.8991/139 · 65.5%278/361 · 77.0%USD 0.0546
21Jev-Vision 8BSeanLiuadapter · 8BBase model: Qwen/Qwen3-VL-8B-Instructsourcedetails59.8657.8153.9386.4751.4997/139 · 69.8%251/361 · 69.5%USD 0.0414
22Visual-Jev genericzero-shot Qwen3.5-4B baseline4BBase model: undiscloseddetails59.1055.0681.4584.5648.2499/139 · 71.2%238/361 · 65.9%USD 0.0531
23Mapika decider-2b-vision BF16fine-tune · 2BBase model: Qwen/Qwen3.5-2B-Basesourcedetails54.8346.2978.5988.5759.0975/139 · 54.0%234/361 · 64.8%USD 0.0231
24OmniJev 2Btinnel123adapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails52.8646.0378.4688.5054.4492/139 · 66.2%212/361 · 58.7%USD 0.0330
25diffusiongemma-26b djev v10 step160snowicarusfine-tune · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails46.1550.2267.6882.8044.6671/139 · 51.1%254/361 · 70.4%USD 0.0699
26Autoloops – Gemma 4 31B ITAPIClosed APIBase model: undiscloseddetails45.9975.1076.4975.0442.62107/139 · 77.0%305/361 · 84.5%USD 0.0818
27Jevify Qwen3-VL-2B T2adapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails44.2541.8782.8389.8161.2788/139 · 63.3%201/361 · 55.7%USD 0.0195
28Winnow-12BQ8_0 + F16 vision projectormerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsourcedetails42.8359.4080.3382.4941.8189/139 · 64.0%267/361 · 74.0%USD 0.0870
29jev-spatialFr0zencr4nEadapterBase model: allenai/Molmo2-ERsourcedetails41.7441.8066.0090.4959.3274/139 · 53.2%218/361 · 60.4%USD 0.0227
30OmniJev-Qwen3.5-9B-v4tzcflyadapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails38.9740.9360.7390.3459.4864/139 · 46.0%227/361 · 62.9%USD 0.0224
31djev-distill-v4tarsur385distilled · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails36.0343.8878.3283.0045.1660/139 · 43.2%245/361 · 67.9%USD 0.0673
32djev-spark NVFP426B-A4B · NVFP4Base model: google/diffusiongemma-26B-A4B-itsource 1source 2details32.1641.1180.2683.5545.8167/139 · 48.2%224/361 · 62.0%USD 0.0640
33vjev-visionyah01adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails30.7936.1079.4390.7060.9766/139 · 47.5%206/361 · 57.1%USD 0.0200
34djev-dev BF1626B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails29.9841.3672.4183.0444.5571/139 · 51.1%220/361 · 60.9%USD 0.0705
35Standard One 3Badapter · 3BBase model: mistralai/Ministral-3-3B-Instruct-2512-BF16sourcedetails24.6833.3481.2186.9954.8572/139 · 51.8%188/361 · 52.1%USD 0.0320
36Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjLLM decoderBase model: undisclosedsourcedetails18.9568.8687.6774.1629.48103/139 · 74.1%286/361 · 79.2%USD 0.2243
37JPT-0.8Bkirp / llm2jevadapter · 0.8BBase model: Qwen3.5-0.8Bsourcedetails18.2429.3276.0089.3758.8549/139 · 35.3%201/361 · 55.7%USD 0.0235
38Jevify Gemma 4 E4Badapter · 4BBase model: google/gemma-4-E4B-itsourcedetails17.7428.8975.5290.7161.0259/139 · 42.4%187/361 · 51.8%USD 0.0199
39GPT-6 Lunalow reasoning effort

Setting: OpenRouter reasoning.effort=low

Cost receipts cover 90.6% of calls

APIClosed APIBase model: undiscloseddetails		17.1076.6284.8266.9228.33114/139 · 82.0%310/361 · 85.9%USD 0.1955
40OmniJev 0.8Btinnel123adapter · 0.8BBase model: Qwen/Qwen3.5-0.8Bsourcedetails16.6728.5173.4288.9655.3874/139 · 53.2%167/361 · 46.3%USD 0.0307
41Gevva E2B multimodal2BBase model: Gemma 4 E2Bsourcedetails15.9328.2676.2686.4749.9553/139 · 38.1%192/361 · 53.2%USD 0.0466
42Gevva E4Btext checkpoint, image input4BBase model: Gemma 4 E4Bsourcedetails15.6829.2772.2885.5847.9761/139 · 43.9%186/361 · 51.5%USD 0.0542
43Qevi-2BMeerDevelopmentadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails15.0727.7554.9589.8060.8783/139 · 59.7%153/361 · 42.4%USD 0.0201
44JEVisiondivyanshx11, visual route0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails11.8927.8237.2585.6047.8550/139 · 36.0%194/361 · 53.7%USD 0.0548
45GPT-5.6 Luna

Cost receipts cover 99.6% of calls

APIClosed APIBase model: undiscloseddetails		10.9089.5894.0063.0823.40122/139 · 87.8%342/361 · 94.7%USD 0.3550
46Gemini 3.1 Flash LiteAPIClosed APIBase model: undiscloseddetails9.0679.4185.5665.1721.96105/139 · 75.5%324/361 · 89.8%USD 0.3994
47PlayJev 0.8Bgated by Intelligence (below the Jev-class floor)fine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails0.043.0048.3590.1260.9932/139 · 23.0%121/361 · 33.5%USD 0.0200
48Gemini 3.8 Flashgated by Cost (USD 2.24 per 1,000 decisions)APIClosed APIBase model: undiscloseddetails0.0085.1173.6759.410.00113/139 · 81.3%336/361 · 93.1%USD 2.2351
49OpenJev 4B NLI v2official image-premise pathgated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.0060.060.0080.2041.25104/139 · 74.8%251/361 · 69.5%USD 0.0909
50OpenJev 4B NLI v5gated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.0061.390.0080.5741.5196/139 · 69.1%266/361 · 73.7%USD 0.0890
Core composite bars
  1. 1. Wity-1APIClosed APIBase model: undisclosed · Base model: undisclosed (author request).details78.63
  2. 2. Imajev-4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails75.49
  3. 3. NeoHorse Jev 4B4BBase model: NeoHorse-1-4Bsourcedetails70.85
  4. 4. JPT-9Bkirp / llm2jevadapter · 9BBase model: Qwen3.5-9Bsourcedetails70.05
  5. 5. Visual-Jev 4B Answer-SFTadapter · 4BBase model: Qwen/Qwen3-VL-4B-Instructsourcedetails69.75
  6. 6. Jev-Omnifine-tune · 12BBase model: Gemma 4 12B ITsourcedetails69.71
  7. 7. JPT-4Bkirp / llm2jevadapter · 4BBase model: Qwen3.5-4Bsourcedetails67.96
  8. 8. imajev 2Badapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails66.70
  9. 9. Reflex 4Breleased stable configurationadapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails65.75
  10. 10. Glancefrozen Qwen3-VL-4B4BBase model: Qwen3-VL-4Bsourcedetails65.04
  11. 11. CUA-S1 4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails64.27
  12. 12. AutoJev-27Bfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsourcedetails63.96
  13. 13. imajev 9Badapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails63.96
  14. 14. Jevify Gemma 4 26B-A4Badapter · 26B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails63.21
  15. 15. OmniJev 4Btinnel123adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails63.07
  16. 16. Surogate Rune 26B-A4B v326B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails62.29
  17. 17. Standard One 8Badapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16sourcedetails61.90
  18. 18. shisa-de-126B-A4BBase model: Gemma 4 26B-A4Bsourcedetails61.88
  19. 19. JevAny-27B SFTadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details60.26
  20. 20. JevAny-27B RLCRadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details59.97
  21. 21. Jev-Vision 8BSeanLiuadapter · 8BBase model: Qwen/Qwen3-VL-8B-Instructsourcedetails59.86
  22. 22. Visual-Jev genericzero-shot Qwen3.5-4B baseline4BBase model: undiscloseddetails59.10
  23. 23. Mapika decider-2b-vision BF16fine-tune · 2BBase model: Qwen/Qwen3.5-2B-Basesourcedetails54.83
  24. 24. OmniJev 2Btinnel123adapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails52.86
  25. 25. diffusiongemma-26b djev v10 step160snowicarusfine-tune · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails46.15
  26. 26. Autoloops – Gemma 4 31B ITAPIClosed APIBase model: undiscloseddetails45.99
  27. 27. Jevify Qwen3-VL-2B T2adapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails44.25
  28. 28. Winnow-12BQ8_0 + F16 vision projectormerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsourcedetails42.83
  29. 29. jev-spatialFr0zencr4nEadapterBase model: allenai/Molmo2-ERsourcedetails41.74
  30. 30. OmniJev-Qwen3.5-9B-v4tzcflyadapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails38.97
  31. 31. djev-distill-v4tarsur385distilled · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails36.03
  32. 32. djev-spark NVFP426B-A4B · NVFP4Base model: google/diffusiongemma-26B-A4B-itsource 1source 2details32.16
  33. 33. vjev-visionyah01adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails30.79
  34. 34. djev-dev BF1626B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails29.98
  35. 35. Standard One 3Badapter · 3BBase model: mistralai/Ministral-3-3B-Instruct-2512-BF16sourcedetails24.68
  36. 36. Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjLLM decoderBase model: undisclosedsourcedetails18.95
  37. 37. JPT-0.8Bkirp / llm2jevadapter · 0.8BBase model: Qwen3.5-0.8Bsourcedetails18.24
  38. 38. Jevify Gemma 4 E4Badapter · 4BBase model: google/gemma-4-E4B-itsourcedetails17.74
  39. 39. GPT-6 Lunalow reasoning effortAPIClosed APIBase model: undiscloseddetails17.10
  40. 40. OmniJev 0.8Btinnel123adapter · 0.8BBase model: Qwen/Qwen3.5-0.8Bsourcedetails16.67
  41. 41. Gevva E2B multimodal2BBase model: Gemma 4 E2Bsourcedetails15.93
  42. 42. Gevva E4Btext checkpoint, image input4BBase model: Gemma 4 E4Bsourcedetails15.68
  43. 43. Qevi-2BMeerDevelopmentadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails15.07
  44. 44. JEVisiondivyanshx11, visual route0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails11.89
  45. 45. GPT-5.6 LunaAPIClosed APIBase model: undiscloseddetails10.90
  46. 46. Gemini 3.1 Flash LiteAPIClosed APIBase model: undiscloseddetails9.06
  47. 47. PlayJev 0.8Bgated by Intelligence (below the Jev-class floor)fine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails0.04
  48. 48. Gemini 3.8 Flashgated by Cost (USD 2.24 per 1,000 decisions)APIClosed APIBase model: undiscloseddetails0.00
  49. 49. OpenJev 4B NLI v2official image-premise pathgated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.00
  50. 50. OpenJev 4B NLI v5gated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.00

Everyday photo decisions · 184 synthetic items

89 public images from the promo set; 95 sealed: 61 variants from the reviewed v0.1 candidate pool across 17 matched situations and 34 fresh pool photos. Ambiguous labels were dropped after visual, two-model and gold-blind human checks. No brands and no focused faces.

#SystemCompositeIntelligenceCalibrationSpeedCostPublic accuracySealed accuracyUSD / 1,000
1Wity-1

Under author review: server build ID was not recorded; this score may change after verification.

Setting: Wity SystemOne, reasoning=auto

APIClosed APIBase model: undisclosed · Base model: undisclosed (author request).details		84.1689.9290.1380.1077.9786/89 · 96.6%89/95 · 93.7%USD 0.0054
2OmniJev-Qwen3.5-9B-v4tzcflyadapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails80.4891.7690.9690.4059.7185/89 · 95.5%91/95 · 95.8%USD 0.0220
3jev-spatialFr0zencr4nEadapterBase model: allenai/Molmo2-ERsourcedetails80.2589.1690.3890.5760.5285/89 · 95.5%89/95 · 93.7%USD 0.0207
4Jev-Omnifine-tune · 12BBase model: Gemma 4 12B ITsourcedetails79.8690.7787.7489.7460.5082/89 · 92.1%92/95 · 96.8%USD 0.0207
5Imajev-4B

Setting: PyTorch; one rotation; --fast; --merge-lora; calibration.json; server 8501f5c3; adapter c9e5f132.

adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails		77.9087.4193.8486.5656.5081/89 · 91.0%90/95 · 94.7%USD 0.0282
6AutoJev-27Bfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsourcedetails74.2494.3686.2186.9049.9385/89 · 95.5%93/95 · 97.9%USD 0.0467
7vjev-visionyah01adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails73.3869.0980.6090.5560.2874/89 · 83.1%80/95 · 84.2%USD 0.0211
8Mapika decider-2b-vision BF16fine-tune · 2BBase model: Qwen/Qwen3.5-2B-Basesourcedetails73.2377.1185.1587.2554.2076/89 · 85.4%85/95 · 89.5%USD 0.0336
9Jevify Qwen3-VL-2B T2adapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails72.8869.3189.9888.8155.3076/89 · 85.4%79/95 · 83.2%USD 0.0309
10imajev 2Badapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails72.6779.3991.7986.9149.9879/89 · 88.8%85/95 · 89.5%USD 0.0465
11Glancefrozen Qwen3-VL-4B4BBase model: Qwen3-VL-4Bsourcedetails72.2576.0378.3388.7155.0278/89 · 87.6%83/95 · 87.4%USD 0.0316
12OmniJev 4Btinnel123adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails71.8380.6983.2487.1050.5179/89 · 88.8%86/95 · 90.5%USD 0.0446
13NeoHorse Jev 4B4BBase model: NeoHorse-1-4Bsourcedetails71.1384.8192.3286.6549.2181/89 · 91.0%88/95 · 92.6%USD 0.0493
14Jevify Gemma 4 E4Badapter · 4BBase model: google/gemma-4-E4B-itsourcedetails69.2960.8472.8990.6260.4670/89 · 78.7%76/95 · 80.0%USD 0.0208
15Jevify Gemma 4 26B-A4Badapter · 26B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails69.1389.7090.2086.2848.5084/89 · 94.4%90/95 · 94.7%USD 0.0521
16OmniJev 2Btinnel123adapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails68.8565.5076.2788.5854.2771/89 · 79.8%79/95 · 83.2%USD 0.0335
17JevAny-27B SFTadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details68.0595.8886.6086.1548.0987/89 · 97.8%93/95 · 97.9%USD 0.0537
18Visual-Jev 4B Answer-SFTadapter · 4BBase model: Qwen/Qwen3-VL-4B-Instructsourcedetails67.6979.7181.4486.4949.0276/89 · 85.4%87/95 · 91.6%USD 0.0501
19JevAny-27B RLCRadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details67.6195.8887.2186.1147.9387/89 · 97.8%93/95 · 97.9%USD 0.0544
20JPT-0.8Bkirp / llm2jevadapter · 0.8BBase model: Qwen3.5-0.8Bsourcedetails67.4959.0079.0088.5654.4371/89 · 79.8%74/95 · 77.9%USD 0.0330
21Surogate Rune 26B-A4B v326B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails66.6989.7087.3185.9947.9284/89 · 94.4%90/95 · 94.7%USD 0.0544
22OmniJev 0.8Btinnel123adapter · 0.8BBase model: Qwen/Qwen3.5-0.8Bsourcedetails66.3957.3973.8588.9655.4374/89 · 83.1%71/95 · 74.7%USD 0.0306
23shisa-de-126B-A4BBase model: Gemma 4 26B-A4Bsourcedetails66.1686.3389.7285.9347.8183/89 · 93.3%88/95 · 92.6%USD 0.0549
24Reflex 4Breleased stable configurationadapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails64.1079.6180.7386.0647.9681/89 · 91.0%84/95 · 88.4%USD 0.0543
25JPT-4Bkirp / llm2jevadapter · 4BBase model: Qwen3.5-4Bsourcedetails62.8489.1693.7285.4146.5285/89 · 95.5%89/95 · 93.7%USD 0.0606
26Gevva E4Btext checkpoint, image input4BBase model: Gemma 4 E4Bsourcedetails62.6860.3059.6287.4252.5771/89 · 79.8%75/95 · 78.9%USD 0.0381
27Jev-Vision 8BSeanLiuadapter · 8BBase model: Qwen/Qwen3-VL-8B-Instructsourcedetails62.1788.4087.6885.3346.6084/89 · 94.4%89/95 · 93.7%USD 0.0602
28djev-spark NVFP426B-A4B · NVFP4Base model: google/diffusiongemma-26B-A4B-itsource 1source 2details59.7276.7991.3883.4746.3479/89 · 88.8%83/95 · 87.4%USD 0.0615
29djev-dev BF1626B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails55.5277.7787.8682.9745.0582/89 · 92.1%82/95 · 86.3%USD 0.0679
30djev-distill-v4tarsur385distilled · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails55.0177.7786.1482.9244.9582/89 · 92.1%82/95 · 86.3%USD 0.0684
31diffusiongemma-26b djev v10 step160snowicarusfine-tune · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails54.3279.6186.4682.7944.6181/89 · 91.0%84/95 · 88.4%USD 0.0702
32JPT-9Bkirp / llm2jevadapter · 9BBase model: Qwen3.5-9Bsourcedetails53.7395.3490.6083.9943.5288/89 · 98.9%92/95 · 96.8%USD 0.0763
33Visual-Jev genericzero-shot Qwen3.5-4B baseline4BBase model: undiscloseddetails53.1181.9986.3683.5744.0479/89 · 88.8%87/95 · 91.6%USD 0.0733
34CUA-S1 4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails52.2083.2980.5684.2143.9079/89 · 88.8%88/95 · 92.6%USD 0.0741
35JEVisiondivyanshx11, visual route0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails51.3560.5363.8085.1045.9373/89 · 82.0%74/95 · 77.9%USD 0.0635
36Gevva E2B multimodal2BBase model: Gemma 4 E2Bsourcedetails50.3945.6668.4988.2154.5162/89 · 69.7%69/95 · 72.6%USD 0.0328
37Autoloops – Gemma 4 31B ITAPIClosed APIBase model: undiscloseddetails49.4493.8287.9173.2742.7386/89 · 96.6%92/95 · 96.8%USD 0.0811
38Standard One 8Badapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16sourcedetails49.1067.0380.4883.6543.6973/89 · 82.0%79/95 · 83.2%USD 0.0754
39imajev 9Badapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails47.4893.8293.3482.9241.3586/89 · 96.6%92/95 · 96.8%USD 0.0902
40Standard One 3Badapter · 3BBase model: mistralai/Ministral-3-3B-Instruct-2512-BF16sourcedetails44.7945.8879.1385.2747.3064/89 · 71.9%68/95 · 71.6%USD 0.0571
41Winnow-12BQ8_0 + F16 vision projectormerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsourcedetails44.4995.3496.4481.8840.1588/89 · 98.9%92/95 · 96.8%USD 0.0988
42Qevi-2BMeerDevelopmentadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails36.7241.0052.5788.5253.9961/89 · 68.5%66/95 · 69.5%USD 0.0342
43Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjLLM decoderBase model: undisclosedsourcedetails19.8996.6493.5272.3829.2988/89 · 98.9%93/95 · 97.9%USD 0.2275
44GPT-6 Lunalow reasoning effort

Setting: OpenRouter reasoning.effort=low

Cost receipts cover 95.1% of calls

APIClosed APIBase model: undiscloseddetails		13.6487.0092.7961.9525.6889/89 · 100.0%85/95 · 89.5%USD 0.2712
45GPT-5.6 Luna

Cost receipts cover 99.5% of calls

APIClosed APIBase model: undiscloseddetails		11.4398.7096.5461.7623.7389/89 · 100.0%94/95 · 98.9%USD 0.3451
46Gemini 3.1 Flash LiteAPIClosed APIBase model: undiscloseddetails10.89100.0092.9361.7923.3189/89 · 100.0%95/95 · 100.0%USD 0.3601
47PlayJev 0.8Bgated by Intelligence (below the Jev-class floor)fine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails0.779.0035.2389.0955.7348/89 · 53.9%49/95 · 51.6%USD 0.0299
48Gemini 3.8 Flashgated by Cost (USD 1.58 per 1,000 decisions)APIClosed APIBase model: undiscloseddetails0.0998.7070.4157.604.0189/89 · 100.0%94/95 · 98.9%USD 1.5839
49OpenJev 4B NLI v2official image-premise pathgated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.0075.930.0079.5339.9883/89 · 93.3%80/95 · 84.2%USD 0.1002
50OpenJev 4B NLI v5gated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.0088.400.0079.4540.0084/89 · 94.4%89/95 · 93.7%USD 0.1000
Everyday photo composite bars
  1. 1. Wity-1APIClosed APIBase model: undisclosed · Base model: undisclosed (author request).details84.16
  2. 2. OmniJev-Qwen3.5-9B-v4tzcflyadapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails80.48
  3. 3. jev-spatialFr0zencr4nEadapterBase model: allenai/Molmo2-ERsourcedetails80.25
  4. 4. Jev-Omnifine-tune · 12BBase model: Gemma 4 12B ITsourcedetails79.86
  5. 5. Imajev-4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails77.90
  6. 6. AutoJev-27Bfine-tune · 27BBase model: Qwen/Qwen3.8-27Bsourcedetails74.24
  7. 7. vjev-visionyah01adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails73.38
  8. 8. Mapika decider-2b-vision BF16fine-tune · 2BBase model: Qwen/Qwen3.5-2B-Basesourcedetails73.23
  9. 9. Jevify Qwen3-VL-2B T2adapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails72.88
  10. 10. imajev 2Badapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails72.67
  11. 11. Glancefrozen Qwen3-VL-4B4BBase model: Qwen3-VL-4Bsourcedetails72.25
  12. 12. OmniJev 4Btinnel123adapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails71.83
  13. 13. NeoHorse Jev 4B4BBase model: NeoHorse-1-4Bsourcedetails71.13
  14. 14. Jevify Gemma 4 E4Badapter · 4BBase model: google/gemma-4-E4B-itsourcedetails69.29
  15. 15. Jevify Gemma 4 26B-A4Badapter · 26B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails69.13
  16. 16. OmniJev 2Btinnel123adapter · 2BBase model: Qwen/Qwen3.5-2Bsourcedetails68.85
  17. 17. JevAny-27B SFTadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details68.05
  18. 18. Visual-Jev 4B Answer-SFTadapter · 4BBase model: Qwen/Qwen3-VL-4B-Instructsourcedetails67.69
  19. 19. JevAny-27B RLCRadapter · 27BBase model: Qwen/Qwen3.8-27Bsource 1source 2details67.61
  20. 20. JPT-0.8Bkirp / llm2jevadapter · 0.8BBase model: Qwen3.5-0.8Bsourcedetails67.49
  21. 21. Surogate Rune 26B-A4B v326B-A4BBase model: google/gemma-4-26B-A4B-itsourcedetails66.69
  22. 22. OmniJev 0.8Btinnel123adapter · 0.8BBase model: Qwen/Qwen3.5-0.8Bsourcedetails66.39
  23. 23. shisa-de-126B-A4BBase model: Gemma 4 26B-A4Bsourcedetails66.16
  24. 24. Reflex 4Breleased stable configurationadapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails64.10
  25. 25. JPT-4Bkirp / llm2jevadapter · 4BBase model: Qwen3.5-4Bsourcedetails62.84
  26. 26. Gevva E4Btext checkpoint, image input4BBase model: Gemma 4 E4Bsourcedetails62.68
  27. 27. Jev-Vision 8BSeanLiuadapter · 8BBase model: Qwen/Qwen3-VL-8B-Instructsourcedetails62.17
  28. 28. djev-spark NVFP426B-A4B · NVFP4Base model: google/diffusiongemma-26B-A4B-itsource 1source 2details59.72
  29. 29. djev-dev BF1626B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails55.52
  30. 30. djev-distill-v4tarsur385distilled · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails55.01
  31. 31. diffusiongemma-26b djev v10 step160snowicarusfine-tune · 26B-A4BBase model: google/diffusiongemma-26B-A4B-itsourcedetails54.32
  32. 32. JPT-9Bkirp / llm2jevadapter · 9BBase model: Qwen3.5-9Bsourcedetails53.73
  33. 33. Visual-Jev genericzero-shot Qwen3.5-4B baseline4BBase model: undiscloseddetails53.11
  34. 34. CUA-S1 4Badapter · 4BBase model: Qwen/Qwen3.5-4Bsourcedetails52.20
  35. 35. JEVisiondivyanshx11, visual route0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails51.35
  36. 36. Gevva E2B multimodal2BBase model: Gemma 4 E2Bsourcedetails50.39
  37. 37. Autoloops – Gemma 4 31B ITAPIClosed APIBase model: undiscloseddetails49.44
  38. 38. Standard One 8Badapter · 8BBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16sourcedetails49.10
  39. 39. imajev 9Badapter · 9BBase model: Qwen/Qwen3.5-9Bsourcedetails47.48
  40. 40. Standard One 3Badapter · 3BBase model: mistralai/Ministral-3-3B-Instruct-2512-BF16sourcedetails44.79
  41. 41. Winnow-12BQ8_0 + F16 vision projectormerge · 12B · GGUF Q8_0Base model: google/gemma-4-12B-itsourcedetails44.49
  42. 42. Qevi-2BMeerDevelopmentadapter · 2BBase model: Qwen/Qwen3-VL-2B-Instructsourcedetails36.72
  43. 43. Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProjLLM decoderBase model: undisclosedsourcedetails19.89
  44. 44. GPT-6 Lunalow reasoning effortAPIClosed APIBase model: undiscloseddetails13.64
  45. 45. GPT-5.6 LunaAPIClosed APIBase model: undiscloseddetails11.43
  46. 46. Gemini 3.1 Flash LiteAPIClosed APIBase model: undiscloseddetails10.89
  47. 47. PlayJev 0.8Bgated by Intelligence (below the Jev-class floor)fine-tune · 0.8BBase model: Qwen/Qwen3.5-0.8B-Basesourcedetails0.77
  48. 48. Gemini 3.8 Flashgated by Cost (USD 1.58 per 1,000 decisions)APIClosed APIBase model: undiscloseddetails0.09
  49. 49. OpenJev 4B NLI v2official image-premise pathgated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.00
  50. 50. OpenJev 4B NLI v5gated by Calibration (no probabilities reported)4BBase model: Qwen/Qwen3.5-4Bsourcedetails0.00

djev-spark sealed photo result: 83/95 sealed decisions · 87.4%. It saw the public promo images in an earlier inference-only video run, with no training; this sealed score is the independent measurement for it.

Split

The split is 228 public / 456 sealed (33.3% / 66.7%). The public part is unchanged. The sealed part is 123 never-exposed v0.1 items plus 333 fresh items from our private synthetic rotation pool, and all 50 systems were re-run on the fresh items with their original settings. 93 legacy items are retired and not scored. Split counts and hashes were frozen and posted before any system saw a fresh item.

Public / sealed split

228 / 456

33.3% public · 66.7% sealed · was 228 / 216

Licensed real-source items

201 items

139 public · 62 sealed · 93 retired

Fresh sealed items

333 items

Our own synthetic renders and photos · 299 core · 34 everyday photos

Synthetic share

70.6%

483/684 scored items are our own synthetic content

Public means an item was already exposed anywhere. That includes all 79 Mind2Web-derived items because Kev's training data overlaps Mind2Web; the 89 promo photos shown in videos; the 8 example cards on this page and in the status video; and 61 source rows in the public site repository since the 21 Sep preview. The split also counts 1 image asset already present in that repository. Every scored item that has never been exposed is sealed.

Sealed is 123 never-exposed v0.1 items plus 333 fresh items drawn from our private synthetic rotation pool (documents, charts, inventory and safety scenes, and everyday photos). The draw was stratified by family and difficulty with a seeded draw, and the split counts and hashes were frozen before any system saw a fresh item. Computer Use and Browser Use pool items were excluded because those tracks stay separate. Existing item-level outputs are reused for the public and kept sealed items; every system was run on the fresh items with the same code, pinned revisions, prompts and settings as its original run.

Retired. 93 items (ScreenSpot 45, ScreenSpot-Pro 31, Android-in-the-Wild 17) were moved from public to sealed on 24 Sep while their inputs and every system's predictions sat outside the sealed store, so they cannot count as an unseen holdout. They are not scored and not relabelled.

FamilyPublicSealedRetired
Android-in-the-Wild (AITW_Single mirror)01117
Everyday photo89950
FinQA2000
Geometry3K1900
Multimodal-Mind2Web7900
Pool: charts0790
Pool: documents01020
Pool: inventory0610
Pool: safety inspection0570
ScreenSpot202945
ScreenSpot-Pro12231
Total22845693

Computer Use and Browser Use tracks (preview)

Two planned tracks extend the image benchmark to computer and browser interactions.

Computer Use

400 items

147 public · 253 sealed

Public decision types

  • target location
  • element type
  • next action class

Sealed decision types

  • task completion
  • action success
  • dialog safety
  • blocker status

Sealed origin: original synthetic local fixtures

Browser Use

400 items

240 public · 160 sealed

Public decision types

  • target location
  • element type
  • next action class

Sealed decision types

  • task completion
  • action success
  • dialog safety
  • blocker status

Sealed origin: original synthetic local fixtures

Cross-track sealing rule. A public-origin Computer Use or Browser Use row that shares a source row or screenshot with a sealed core item is sealed too.

Kev / Mind2Web flag. 133 Mind2Web-derived Browser Use rows are public-only and reported separately for Kev, whose training data overlaps Mind2Web.

Not measured yet — no scores. Neither track has a reviewed system or reusable output. Computer Use has no reviewed candidate in the current queue. Browser Use has one candidate requested, not yet evaluated: kev-0.6b-browser-use. Its public, ungated HF adapter is pinned at 08414b0001f6eba32bc69372abfeac5b74718ac0 (model card license: Apache-2.0), with base jaredpalmer/kev-0.6b at dece6dba8d43f0f7ded45e9f5b9df12474d90843. The model takes textual DOM state and uses a pointer head; its serving path and safe head loader are not independently reviewed. Its Mind2Web-derived examples remain public-only. Both tracks still lack a ready matched-family split and measurements, so they remain separate, requested, and unscored outside the core ranking.

Method and limitations

System types (colours)

  • Jev — reference (TypeSafe, closed) — The closed system JevBench is named after, shown as the reference.
  • Closed API (weights not public) — Available through a hosted API; the weights cannot be downloaded.
  • Open weights · LLM decoder — An autoregressive language model with public weights, including fine-tunes, merges and Jev rebuilds.
  • Open weights · diffusion LM — A language model that generates by iterative denoising instead of token by token.
  • Open weights · encoder / classifier — BERT-style encoders, NLI zero-shot classifiers and GLiNER-type models.
  • Open weights · reranker — A cross-encoder or LLM reranker that scores options against the input.
  • Base model control (no decision fine-tune, raw logits) — An official open checkpoint without any decision fine-tune, read out from raw logits, used as a floor.
  • System (router / cascade / ensemble) — Several models combined at inference time.

Colours show architecture only; they never change a score or rank. Each class is assigned from cited evidence (config.json, model card, provider docs, or our own run record for hosted APIs).

Intelligence. Accuracy counts missing, invalid and unparseable answers as wrong. Each part is chance-corrected against its own average chance rate, then combined as 35% public and 65% sealed. Calibration uses the same weights.

Matched-family overfit penalty. The gap is public accuracy minus sealed accuracy within families that have at least 10 items on both sides: ScreenSpot and Everyday photo. If that matched gap is above 15 percentage points, Intelligence is multiplied by max(0, 1 − (gap − 15)/100). The same rule applies to every system. The raw overall gap is shown in the data but does not affect the score.

Calibration. Ten-bin top-label ECE is scaled by valid probability coverage. Label-only output receives zero calibration. OpenJev's NLI entailment values select an answer but are not treated as categorical probabilities.

Speed and cost. Speed uses whole-call p50 and p95 latency; local latency uses the v1.4 2× plus 0.15-second adjustment. Hosted unit cost uses returned per-call usage receipts; missing receipts are not zero-filled, and the Cost axis is scaled by receipt coverage. Retry costs are tracked separately. Local cost uses measured GPU seconds at the recorded per-system GPU-hour rate and excludes loading, downloads, build, and idle time. For self-hosted systems, the original run's timings are combined with the fresh-item run on the same GPU type.

Capability eligibility. Jev itself cannot read images, so there is no Jev row to anchor on. We use the same absolute budget as JevBench: at most 2× Jev 1.13.0's JevBench v1.5.4 cost (≤ USD 0.0646 per 1,000 decisions) and at most 2× its median latency (≤ 1.23 s, adjusted p50), frozen in this release. A decision model in this class should fit the same budget whether the input is text or an image. 38 of 50 systems qualify, a larger share than on JevBench, so we keep 2×. Loosening to 3× would add 7 open-weight fine-tunes that sit just above the cost line; the general-purpose hosted LLMs stay outside even at 3×.

API pricing (1 Oct 2026). API models whose base model we know are ranked at the developer's stated API price. A striped second bar shows the score and would-be rank at the base-model reference price we use for self-served open weights of the same base. Wity-1 uses USD 0.042 per million input tokens, with output free, on its measured 120,542 input and zero output tokens. Its base model is undisclosed at the author's request; the striped bar applies the matching base-model reference price. The API does not quantify image or thinking work, so token pricing may understate full image-compute cost. Pricing provenance.

Presets and What-If. Equal 25/25/25/25 weights reproduce the official composite ranking. Capability-only, accuracy, speed and cost presets and the weight sliders explore the same measured axes; the low Intelligence, Speed and Cost gates still apply, including at zero weight. Under custom weights every gated row carries a “gate ×…” tag naming the axis and factor, and the weights panel lists the gates that fire. The striped price alternative re-scores under the same selected weights; its rank is hypothetical. Capability Score and its official frozen eligibility budget stay fixed; the cost and latency cap sliders on the Capability ranking give an unofficial, shareable view with other caps.

Composite and gates. These rules are unchanged. The four axes use an equal-weight harmonic mean, followed by the Jev-class Intelligence, Speed, and Cost gates below 50. Gemini 3.8 Flash's high raw accuracy but near-zero composite reflects its measured cost and the Cost gate; label-only systems have zero Calibration under the inherited convention.

Difficulty balance. The split follows exposure, not a stratified draw, so the parts differ in family mix: browser actions (Mind2Web), chart questions (FinQA) and geometry are public-only, while ScreenSpot-Pro, Android-in-the-Wild and the fresh pool families are sealed-only. This is why the overfit penalty compares only matched families (ScreenSpot and Everyday photo).

Fresh-item difficulty. The 333 fresh sealed items are our own synthetic images and renders. They are easier for frontier API models than the older real-source items, so sealed accuracy is higher than on the earlier split for most hosted systems. Scores on this split are therefore not comparable with the earlier 228/216 preview.

Re-evaluation policy. Every release re-evaluates the current top 10 on the composite score. Models ranked #11 and below are re-evaluated on a slower cadence — at least monthly, or with every third scheduled refresh release, whichever comes first — and their score is shown as last measured on its release. A material method change re-evaluates every model. Paid fast-lane runs are evaluated within 48 hours of payment, and new submissions are evaluated in the order received.

Exposure. GPT-6 Luna and Gemini 3.8 Flash previously saw public promo-photo candidates and sealed-photo candidates in stateless label-check calls, including candidates later dropped. The checks showed no gold; human gold-blind adjudication decided inclusion. GPT-6 Luna is the saved low-reasoning-effort setting. Four OpenRouter systems (GPT-6 Luna low, GPT-5.6 Luna, Gemini 3.1 Flash Lite and Gemini 3.8 Flash) received sealed images and questions through the requested no-retention route with provider fallback disabled. Gemma 4 31B used the Autoloops endpoint. Wity-1 was run on 29 Sep through the hosted Wity SystemOne endpoint with bearer authentication; no no-retention claim is made for Wity or Autoloops. These six hosted rows carry the API flag. Autoloops – Gemma 4 31B IT: results on the original items are reused from the completed Autoloops measurement; the 333 fresh sealed items were run on 25 Sep with the original Autoloops runner. Returned per-request token usage is costed at the published Gemma 4 31B rates. djev-spark saw the 100 public promo photos in the earlier inference-only video run, with no training. Its score on the 95 sealed photo items is the independent measure. Local systems ran without network, credentials, or gold maps. The fresh pool items were authored with Claude and reviewed by OpenAI Codex as a blind critic, and the pool photos were generated with an OpenAI image model. This is disclosed for the two OpenAI rows. Self-hosted systems ran the fresh items on one rented GPU pod in offline containers with read-only inputs.

Requested and excluded candidates (69)

The ranking covers 50 measured configurations. Wity-1 completed a full 684-decision API run. Its Cost axis uses Wity’s stated API tariff by Florian’s 1 Oct price decision; the exact deployed server build remains under author review. Candidates below have no score unless listed in the ranking. Requested rows remain visible with the exact access or review blocker; exclusions describe the reviewed interface, license or duplicate status.

CandidateSource revisionAccessStatusReason
CUA-S1-4B-0.2 multimodal adapterHF 16818868bPublic, ungated; pinned revision resolvesincluded in v0.1.5 ranking (#23 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Visual Jev 4B Answer-SFTHF 7a3f1bb0dPublic, ungated; pinned revision resolvesincluded in v0.1.5 ranking (#5 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
NeoHorse-Jev-4BHF 56c36ae62Public, ungated; pinned revision resolvesincluded in v0.1.5 ranking (#4 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Jevify Qwen3-VL-2B Tier 2HF 46e8e72c2Public, ungated; pinned revision resolvesincluded in v0.1.5 ranking (#25 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Standard One 3BHF c0d23877ePublic, ungated; pinned revision resolvesincluded in v0.1.5 ranking (#37 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Standard One 8BHF 5d1285dd7Public, ungated; pinned revision resolvesincluded in v0.1.5 ranking (#24 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Sage1 / Levanto SageHosted imageAPI requires SAGE_API_KEY; no configured key or free grantrequested, not yet evaluatedNo-cost access is unavailable. No paid quota was purchased.
djev-distill-v426B-class chNo checkpoint stagedincluded in v0.1.5 ranking (#28 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
yah01/vjev-visionHF 2fa8b58e4Public, ungatedincluded in v0.1.5 ranking (#31 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
WIlfLin/JEV-Qwen3.8-Flash-Next-Linear-RuntimeHF 1847787ffPublic model metadata; upstream weights about 184 GBrequested, not yet evaluatedLicense permission and a viable reviewed multi-GPU runtime remain unresolved; no weights were downloaded.
Visual-Jev generic Qwen3.5-4B scorerGitHub 2d68cPublic repositoryincluded in v0.1.5 ranking (#27 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
divyanshx11/JEVisionHF 324698caaPublic, ungatedincluded in v0.1.5 ranking (#41 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
SeanLiu/Jev-Vision 8BExact infereCheckpoint page is publicincluded in v0.1.5 ranking (#20 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Fr0zencr4nE/jev-spatialHF 5727eade6Metadata public; previous anonymous weight-file inventory returned HTTP 401included in v0.1.5 ranking (#11 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Sarashina 2.2 vision JEV mmproj 3BHF 09ce275e3Public, ungatedrequested, not yet evaluatedThe vision-language projection has no reviewed general typed-choice adapter/runtime.
OhtaMan Gemma 4 E2B IT choice-64HF 3d403b990Public, ungatedrequested, not yet evaluatedImage-text metadata is present, but its actual choice interface and inference code are unreviewed.
MetaSK-Jev 4B policy mixHF ea20fe85bPublic, ungatedrequested, not yet evaluatedImage-text metadata is present, but its actual image path and typed-choice runtime are unreviewed.
Jevify Gemma 4 E4BHF a6b5a716fPublic, ungatedincluded in v0.1.5 ranking (#36 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Jevify Gemma 4 26B-A4BHF d4c0d1d45Public, ungatedincluded in v0.1.5 ranking (#16 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
JevAny-27B-SFTHF ad7b48b70Public, ungatedincluded in v0.1.5 ranking (#18 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
JevAny-27B-RLCRHF 078883b2ePublic, ungatedincluded in v0.1.5 ranking (#19 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
LFM2.5-VL-3B-Decision-NVFP4HF 1ff8bf556Public, ungatedrequested, not yet evaluatedImage/video input is reported, but license terms, decision-head behavior and local runtime are unresolved.
yeyan00/Jev-DecisionGitHub eac9dPublic repository; exact checkpoint access unknownrequested, not yet evaluatedThe checkpoint, image interface and inference runtime are not pinned or reviewed.
Bonsai-Llama-Jev submission (measured as Bonsai-2-27B v2)kyr0/Bonsai-Public pinned submission and model artifacts; completed item-level outputs reused from the Bonsai measurement jobincluded in v0.1.5 ranking (#42 of 50)Bonsai-Llama-Jev is retained from the live ImageJevBench roster; see its ranking row above.
AutoJev-27BGitHub denisPublic, ungated; Apache-2.0 weights / MIT codeincluded in v0.1.5 ranking (#9 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
CLM-8BContrastive-Public checkpoint; text encoderexcluded from image rankingThe reviewed Engine.answer(state, {decision: question}) adapter and frozen Qwen3-8B encoder have no image argument or image processor.
Eikos 4B / 27BReviewed LetPublic checkpoint; source reviewedexcluded from image rankingLetterAdapter.dist(state_text, question, options) receives text only; conditional-generation architecture selection is not image input.
Main Jev / TypeSafeOfficial docOfficial API/interface; no native image fieldexcluded from image rankingThe documented state/decision interface accepts text, objects and arrays, with no supported image input. A wrapper field is insufficient.
OpenJev-VisionGitHub b83bePublic repositoryexcluded from core rankingThe published fixed classifier/head does not expose the general typed dynamic-option interface used by the benchmark.
hr98w/jev-visualGitHub 4382bPublic repositoryexcluded as a duplicate wrapperA request/runtime wrapper, not a distinct checkpoint or decision head.
Jev-Omni reuploads and quantizationsCandidate swPublic reuploadsexcluded as duplicatesNo distinct model family/runtime is established beyond the already measured Jev-Omni row.
joyfox/Qwen3.5-0.8B-JEVCandidate swPublic checkpointexcluded from image rankingThe decision artifact removes the vision tower; text-only.
Hanno-Labs/bosun-v3.1 0.6B / 1.7BCandidate swPublic checkpointexcluded from image rankingQwen3 text decision models; no image path documented in the reviewed release.
autotrust/JEVCandidate swPublic sourceexcluded from image rankingTyped text/JSON decisions only; no image input documented in the reviewed interface.
Programalyst/realtime-vision-decision-agentCandidate swPublic repositoryexcluded as an application wrapperCombines vision detection with Jev but is not a distinct model checkpoint.
JPT-0.8BPinned measuPublic checkpoint; non-commercial termsincluded in v0.1.5 ranking (#35 of 50)Included under Florian's 27 Sep decision despite the stated non-commercial license; see the measured row above.
JPT-4BPinned measuPublic checkpoint; non-commercial termsincluded in v0.1.5 ranking (#6 of 50)Included under Florian's 27 Sep decision despite the stated non-commercial license; see the measured row above.
JPT-9BPinned measuPublic checkpoint; non-commercial termsincluded in v0.1.5 ranking (#12 of 50)Included under Florian's 27 Sep decision despite the stated non-commercial license; see the measured row above.
Qevi-2BPinned measuPublic checkpoint; non-commercial termsincluded in v0.1.5 ranking (#43 of 50)Included under Florian's 27 Sep decision despite the stated non-commercial license; see the measured row above.
OmniJev 0.8BPinned measuPublic aggregate rowincluded in v0.1.5 ranking (#39 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
OmniJev 2BPinned measuPublic aggregate rowincluded in v0.1.5 ranking (#22 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
OmniJev 4BPinned measuPublic aggregate rowincluded in v0.1.5 ranking (#13 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Imajev-4BImajev-4B faPublic request modelincluded in v0.1.5 ranking (#2 of 50)Remeasured on the unchanged v0.1 method and split with the pinned server and adapter, one rotation, --fast, and --merge-lora. Only aggregate results are published.
Glance frozen Qwen3-VL-4ByoheinakajimFrozen checkpoint; requested by Yoheiincluded in v0.1.5 ranking (#8 of 50)Retained from the reviewed ImageJev aggregate. The hold applies only to the separate Qwen3-VL-2B CUDA speedlab path.
Laya Visionthaitea/layaCustomer-requested evaluationheld at author's requestNot ranked or published while the author explores other options.
Glance speedlab Qwen3-VL-2Bglance.yoheiApple-only MLX path; measured CUDA port differsheld pending author discussionNot ranked or published until Yohei is asked about the separate CUDA port.
Winnow-12BEldanRing/wiReviewed source pin; independently scored ImageJev handoffincluded in v0.1.5 ranking (#34 of 50)Independent recomputation matched 21/21 metrics with zero delta. The pinned source and author build.py/runtime lock were used, but this run used an isolated server image rather than the separate multi-model load-test Docker context. Medium dependency reproducibility: apt packages resolved at build time and pip wheels lack hashes. All per-file hashes and image membership passed; a separate image-tree digest was not compared with the aggregate receipt (low assurance limitation).
Jev-OmniFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#3 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
imajev 2BFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#7 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Mapika decider-2b-vision BF16Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#10 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Reflex 4B (released stable configuration)Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#14 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
OmniJev-Qwen3.5-9B-v4 (tzcfly)Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#15 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Surogate Rune 26B-A4B v3Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#17 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
shisa-de-1Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#21 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
imajev 9BFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#26 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
djev-spark NVFP4Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#29 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
diffusiongemma-26b djev v10 step160 (snowicarus)Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#30 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Autoloops – Gemma 4 31B ITFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#32 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
djev-dev BF16Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#33 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Gevva E4B (text checkpoint, image input)Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#38 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Gevva E2B multimodalFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#40 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
GPT-6 Luna (low reasoning effort)Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#44 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
GPT-5.6 LunaFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#45 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Gemini 3.1 Flash LiteFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#46 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
PlayJev 0.8BFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#47 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Gemini 3.8 FlashFrozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#48 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
OpenJev 4B NLI v2 (official image-premise path)Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#49 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
OpenJev 4B NLI v5Frozen v0.1.Public aggregate rowincluded in v0.1.5 ranking (#50 of 50)Measured aggregate included in the ranking under the frozen ImageJevBench v0.1 method; see its row above.
Wity-1Wity productHosted Wity SystemOne API; Cost uses the stated Wity tariffincluded in v0.1.5 ranking (#1 of 50)Full 684-decision run with valid probabilities and complete input/output usage; the stated API tariff was selected by the owner on 1 Oct. Server build remains under author review.

Revision history

These are composite rankings at each revision, read from its frozen aggregate artifact. The Capability Score became the headline in v0.1.5. Earlier split scores are not directly comparable with the current split.

v0.1 · 25 Sep 2026 clean-split revision

The original preview established the scoring method; the 25 Sep clean-split revision corrected exposure and added fresh sealed items. No frozen v0.1 ranking copy is retained in this repository, so its top three and full composite ranking cannot be reconstructed here.

v0.1.1 · Expanded the clean public/sealed split to 48 measured systems; kept the frozen scoring method.Top three: 1. Jev-Omni (73.10) · 2. NeoHorse Jev 4B (71.94) · 3. Visual-Jev 4B Answer-SFT (69.81)
#SystemComposite
1Jev-Omni73.10
2NeoHorse Jev 4B71.94
3Visual-Jev 4B Answer-SFT69.81
4JPT-4B (kirp / llm2jev)69.55
5imajev 2B68.72
6Glance (frozen Qwen3-VL-4B)66.86
7AutoJev-27B66.85
8Mapika decider-2b-vision BF1666.77
9jev-spatial (Fr0zencr4nE)65.87
10JPT-9B (kirp / llm2jev)65.82
11OmniJev 4B (tinnel123)65.65
12Reflex 4B (released stable configuration)65.38
13OmniJev-Qwen3.5-9B-v4 (tzcfly)65.14
14Jevify Gemma 4 26B-A4B65.00
15Surogate Rune 26B-A4B v363.82
16JevAny-27B SFT63.79
17JevAny-27B RLCR63.46
18Jev-Vision 8B (SeanLiu)63.39
19shisa-de-163.20
20OmniJev 2B (tinnel123)62.84
21CUA-S1 4B62.63
22Standard One 8B59.80
23Jevify Qwen3-VL-2B T259.02
24imajev 9B58.33
25Visual-Jev generic (zero-shot Qwen3.5-4B baseline)57.73
26djev-distill-v4 (tarsur385)50.04
27djev-spark NVFP449.38
28diffusiongemma-26b djev v10 step160 (snowicarus)48.76
29vjev-vision (yah01)47.67
30Autoloops – Gemma 4 31B IT47.14
31djev-dev BF1646.64
32Winnow-12B (Q8_0 + F16 vision projector)43.56
33JPT-0.8B (kirp / llm2jev)30.51
34Jevify Gemma 4 E4B29.76
35Standard One 3B29.01
36Gevva E4B (text checkpoint, image input)26.83
37OmniJev 0.8B (tinnel123)26.81
38Gevva E2B multimodal21.85
39JEVision (divyanshx11, visual route)21.50
40Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj19.26
41Qevi-2B (MeerDevelopment)17.44
42GPT-6 Luna (low reasoning effort)16.06
43GPT-5.6 Luna11.05
44Gemini 3.1 Flash Lite9.58
45PlayJev 0.8B0.11
46Gemini 3.8 Flash0.00
47OpenJev 4B NLI v2 (official image-premise path)0.00
48OpenJev 4B NLI v50.00
v0.1.2 · Added Imajev-4B (49 systems), initially #11 in the composite ranking.Top three: 1. Jev-Omni (73.10) · 2. NeoHorse Jev 4B (71.94) · 3. Visual-Jev 4B Answer-SFT (69.81)
#SystemComposite
1Jev-Omni73.10
2NeoHorse Jev 4B71.94
3Visual-Jev 4B Answer-SFT69.81
4JPT-4B (kirp / llm2jev)69.55
5imajev 2B68.72
6Glance (frozen Qwen3-VL-4B)66.86
7AutoJev-27B66.85
8Mapika decider-2b-vision BF1666.77
9jev-spatial (Fr0zencr4nE)65.87
10JPT-9B (kirp / llm2jev)65.82
11Imajev-4B65.72
12OmniJev 4B (tinnel123)65.65
13Reflex 4B (released stable configuration)65.38
14OmniJev-Qwen3.5-9B-v4 (tzcfly)65.14
15Jevify Gemma 4 26B-A4B65.00
16Surogate Rune 26B-A4B v363.82
17JevAny-27B SFT63.79
18JevAny-27B RLCR63.46
19Jev-Vision 8B (SeanLiu)63.39
20shisa-de-163.20
21OmniJev 2B (tinnel123)62.84
22CUA-S1 4B62.63
23Standard One 8B59.80
24Jevify Qwen3-VL-2B T259.02
25imajev 9B58.33
26Visual-Jev generic (zero-shot Qwen3.5-4B baseline)57.73
27djev-distill-v4 (tarsur385)50.04
28djev-spark NVFP449.38
29diffusiongemma-26b djev v10 step160 (snowicarus)48.76
30vjev-vision (yah01)47.67
31Autoloops – Gemma 4 31B IT47.14
32djev-dev BF1646.64
33Winnow-12B (Q8_0 + F16 vision projector)43.56
34JPT-0.8B (kirp / llm2jev)30.51
35Jevify Gemma 4 E4B29.76
36Standard One 3B29.01
37Gevva E4B (text checkpoint, image input)26.83
38OmniJev 0.8B (tinnel123)26.81
39Gevva E2B multimodal21.85
40JEVision (divyanshx11, visual route)21.50
41Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj19.26
42Qevi-2B (MeerDevelopment)17.44
43GPT-6 Luna (low reasoning effort)16.06
44GPT-5.6 Luna11.05
45Gemini 3.1 Flash Lite9.58
46PlayJev 0.8B0.11
47Gemini 3.8 Flash0.00
48OpenJev 4B NLI v2 (official image-premise path)0.00
49OpenJev 4B NLI v50.00
v0.1.3 · Re-measured Imajev-4B with fast serving; it moved to #1. Other systems and the split stayed frozen.Top three: 1. Imajev-4B (76.39) · 2. Jev-Omni (73.10) · 3. NeoHorse Jev 4B (71.94)
#SystemComposite
1Imajev-4B76.39
2Jev-Omni73.10
3NeoHorse Jev 4B71.94
4Visual-Jev 4B Answer-SFT69.81
5JPT-4B (kirp / llm2jev)69.55
6imajev 2B68.72
7Glance (frozen Qwen3-VL-4B)66.86
8AutoJev-27B66.85
9Mapika decider-2b-vision BF1666.77
10jev-spatial (Fr0zencr4nE)65.87
11JPT-9B (kirp / llm2jev)65.82
12OmniJev 4B (tinnel123)65.65
13Reflex 4B (released stable configuration)65.38
14OmniJev-Qwen3.5-9B-v4 (tzcfly)65.14
15Jevify Gemma 4 26B-A4B65.00
16Surogate Rune 26B-A4B v363.82
17JevAny-27B SFT63.79
18JevAny-27B RLCR63.46
19Jev-Vision 8B (SeanLiu)63.39
20shisa-de-163.20
21OmniJev 2B (tinnel123)62.84
22CUA-S1 4B62.63
23Standard One 8B59.80
24Jevify Qwen3-VL-2B T259.02
25imajev 9B58.33
26Visual-Jev generic (zero-shot Qwen3.5-4B baseline)57.73
27djev-distill-v4 (tarsur385)50.04
28djev-spark NVFP449.38
29diffusiongemma-26b djev v10 step160 (snowicarus)48.76
30vjev-vision (yah01)47.67
31Autoloops – Gemma 4 31B IT47.14
32djev-dev BF1646.64
33Winnow-12B (Q8_0 + F16 vision projector)43.56
34JPT-0.8B (kirp / llm2jev)30.51
35Jevify Gemma 4 E4B29.76
36Standard One 3B29.01
37Gevva E4B (text checkpoint, image input)26.83
38OmniJev 0.8B (tinnel123)26.81
39Gevva E2B multimodal21.85
40JEVision (divyanshx11, visual route)21.50
41Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj19.26
42Qevi-2B (MeerDevelopment)17.44
43GPT-6 Luna (low reasoning effort)16.06
44GPT-5.6 Luna11.05
45Gemini 3.1 Flash Lite9.58
46PlayJev 0.8B0.11
47Gemini 3.8 Flash0.00
48OpenJev 4B NLI v2 (official image-premise path)0.00
49OpenJev 4B NLI v50.00
v0.1.4 · Added Wity-1 (50 systems), priced at a base-model reference (base undisclosed at the author’s request); its build remains under author review.Top three: 1. Imajev-4B (76.39) · 2. Wity-1 (74.36) · 3. Jev-Omni (73.10)
#SystemComposite
1Imajev-4B76.39
2Wity-174.36
3Jev-Omni73.10
4NeoHorse Jev 4B71.94
5Visual-Jev 4B Answer-SFT69.81
6JPT-4B (kirp / llm2jev)69.55
7imajev 2B68.72
8Glance (frozen Qwen3-VL-4B)66.86
9AutoJev-27B66.85
10Mapika decider-2b-vision BF1666.77
11jev-spatial (Fr0zencr4nE)65.87
12JPT-9B (kirp / llm2jev)65.82
13OmniJev 4B (tinnel123)65.65
14Reflex 4B (released stable configuration)65.38
15OmniJev-Qwen3.5-9B-v4 (tzcfly)65.14
16Jevify Gemma 4 26B-A4B65.00
17Surogate Rune 26B-A4B v363.82
18JevAny-27B SFT63.79
19JevAny-27B RLCR63.46
20Jev-Vision 8B (SeanLiu)63.39
21shisa-de-163.20
22OmniJev 2B (tinnel123)62.84
23CUA-S1 4B62.63
24Standard One 8B59.80
25Jevify Qwen3-VL-2B T259.02
26imajev 9B58.33
27Visual-Jev generic (zero-shot Qwen3.5-4B baseline)57.73
28djev-distill-v4 (tarsur385)50.04
29djev-spark NVFP449.38
30diffusiongemma-26b djev v10 step160 (snowicarus)48.76
31vjev-vision (yah01)47.67
32Autoloops – Gemma 4 31B IT47.14
33djev-dev BF1646.64
34Winnow-12B (Q8_0 + F16 vision projector)43.56
35JPT-0.8B (kirp / llm2jev)30.51
36Jevify Gemma 4 E4B29.76
37Standard One 3B29.01
38Gevva E4B (text checkpoint, image input)26.83
39OmniJev 0.8B (tinnel123)26.81
40Gevva E2B multimodal21.85
41JEVision (divyanshx11, visual route)21.50
42Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj19.26
43Qevi-2B (MeerDevelopment)17.44
44GPT-6 Luna (low reasoning effort)16.06
45GPT-5.6 Luna11.05
46Gemini 3.1 Flash Lite9.58
47PlayJev 0.8B0.11
48Gemini 3.8 Flash0.00
49OpenJev 4B NLI v2 (official image-premise path)0.00
50OpenJev 4B NLI v50.00
v0.1.5 · Ranked Wity-1 at its stated API tariff, kept a base-model price alternative, and froze the JevBench cost/latency envelope for the Capability Score.Top three: 1. Wity-1 (80.20) · 2. Imajev-4B (76.39) · 3. Jev-Omni (73.10)
#SystemComposite
1Wity-180.20
2Imajev-4B76.39
3Jev-Omni73.10
4NeoHorse Jev 4B71.94
5Visual-Jev 4B Answer-SFT69.81
6JPT-4B (kirp / llm2jev)69.55
7imajev 2B68.72
8Glance (frozen Qwen3-VL-4B)66.86
9AutoJev-27B66.85
10Mapika decider-2b-vision BF1666.77
11jev-spatial (Fr0zencr4nE)65.87
12JPT-9B (kirp / llm2jev)65.82
13OmniJev 4B (tinnel123)65.65
14Reflex 4B (released stable configuration)65.38
15OmniJev-Qwen3.5-9B-v4 (tzcfly)65.14
16Jevify Gemma 4 26B-A4B65.00
17Surogate Rune 26B-A4B v363.82
18JevAny-27B SFT63.79
19JevAny-27B RLCR63.46
20Jev-Vision 8B (SeanLiu)63.39
21shisa-de-163.20
22OmniJev 2B (tinnel123)62.84
23CUA-S1 4B62.63
24Standard One 8B59.80
25Jevify Qwen3-VL-2B T259.02
26imajev 9B58.33
27Visual-Jev generic (zero-shot Qwen3.5-4B baseline)57.73
28djev-distill-v4 (tarsur385)50.04
29djev-spark NVFP449.38
30diffusiongemma-26b djev v10 step160 (snowicarus)48.76
31vjev-vision (yah01)47.67
32Autoloops – Gemma 4 31B IT47.14
33djev-dev BF1646.64
34Winnow-12B (Q8_0 + F16 vision projector)43.56
35JPT-0.8B (kirp / llm2jev)30.51
36Jevify Gemma 4 E4B29.76
37Standard One 3B29.01
38Gevva E4B (text checkpoint, image input)26.83
39OmniJev 0.8B (tinnel123)26.81
40Gevva E2B multimodal21.85
41JEVision (divyanshx11, visual route)21.50
42Bonsai-2-27B v2 PQ2_0 + Q8_0 MMProj19.26
43Qevi-2B (MeerDevelopment)17.44
44GPT-6 Luna (low reasoning effort)16.06
45GPT-5.6 Luna11.05
46Gemini 3.1 Flash Lite9.58
47PlayJev 0.8B0.11
48Gemini 3.8 Flash0.00
49OpenJev 4B NLI v2 (official image-premise path)0.00
50OpenJev 4B NLI v50.00

Image JevBench is a separate benchmark from the text-only JevBench Score. Submit a model for evaluation → Sealed item-level content remains private. Public/sealed item counts, accuracy, track and score breakdowns are aggregates. Results describe these exact tested configurations and do not establish absence from model training data.