Autoloops – Gemma 4 31B IT
Data: JevBench v1.6.1, published 2026-10-06
Capability 72.7 · API offering, ranked on the API leaderboard · outside the Jev-class caps (cost 3.0× Jev (v1.5 reference)), below the Capability ranking
Measured 2026-10-06 on A4 ∪ P, 600 items, equated (+2.57 I / +3.35 C); round score-a4-2.
Gemma terms of use (Google) for the base weights; hosted Autoloops service
intelligence
58.1
calibration
87.3
speed
84.5
cost
40.5
Composite (secondary)
40.2 · Composite rank #13 of 27 on the API leaderboard
Cost and measurement conditions
$0.096 per 1,000 decisions (public tariff)
measured: token usage of the 6 Oct 2026 A4 u P run x tariff USD 0.2/M input, 0.45/M output (spec kushal-gemma4-31b-it-autoloops.json), per 1,000 decisions on the v1.6.1 common cost basis (591 of 600 items; the 9 A4 u P items outside the Jev reference's accepted input range are excluded, cost-basis-a4.json)
p50 latency: 0.6 s
none (API)
Breakdowns (not part of the score)
Chance-corrected competence (0 = chance, 100 = perfect) on A4 ∪ P (600 items: 300 open + 300 sealed; fewer sealed items than the full set, so wider uncertainty). Raw values, not equated. Category cells use A4+P+L1+L2+L3; per-type/tier cells use the original basis above. Radars against Jev.
| Request type | Open | Sealed |
|---|---|---|
| choice | 75.1 | 80.3 |
| noul | 50.0 | 39.9 |
| score | 47.1 | 40.7 |
Capability by subject topic
| Rules, policy & law | 57.1 | n=1133 |
|---|---|---|
| Coding & software | 70.8 | n=202 |
| Math & numbers | 11.3 | n=269 |
| Finance & commerce | 53.2 | n=178 |
| Support & operations | 69.1 | n=109 |
| Everyday language | 85.5 | n=130 |
| Safety & security | 69.2 | n=84 |
Use cases (TypeSafe categories)
| Model routing | 77.2 | n=267 |
|---|---|---|
| Legal & compliance | 52.2 | n=269 |
| Customer support | 60.7 | n=156 |
| E-commerce | 46.4 | n=88 |
| Insurance claims | 49.8 | n=92 |
| Risk assessment | 42.6 | n=88 |
| Financial crime | 36.9 | n=76 |
| Feature extraction | 37.5 | n=90 |
| Lead generation | 60.2 | n=86 |
| Recruiting | 35.2 | n=88 |
| Knowledge graphs | 60.9 | n=82 |
| LLM guardrails | 90.0 | n=82 |
| Moderation | 85.8 | n=69 |
| Code linting | 64.4 | n=81 |
| Search & retrieval | 90.5 | n=84 |
| Science | 39.9 | n=82 |
| Advertising | 53.3 | n=85 |
| Gaming | 22.8 | n=79 |
| Demand forecasting | 0.0 | n=72 |
Cells need at least 15 answered items; † fewer than 30 (indicative only, listed below the radar in the compare view).
Previous measurement (v1.5.x, not comparable)
Capability 81.3 · composite 40.5 · cost $0.103 per 1,000 decisions (not published)
Last measured 2026-09-28, measurement release v1.5.0. Older method and scale; the current figures above replace it.