Jev vs decisio v0.8.0 on gemma-4-31B-it: published benchmark comparison

Data: JevBench v1.6.1, published 2026-10-06

Rank #2 on the open-weights board in v1.6.1.

Compare Jev-compatible models by separate measures and their measurement conditions. A historical measurement is not a current rank.

Jev and decisio v0.8.0 on gemma-4-31B-it: published axes and category radars

Pick any two. Radars for the score axes, capability by subject topic and use cases, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.

System A

System B

A: decisio v0.8.0 on gemma-4-31B-it — Open weights · LLM decoder · Score 71.7 (#2)
B: Jev 1.13.0REFERENCE · TypeSafe — Jev — reference (TypeSafe, closed) · Score 71.5 (ranked, not ranked)

The four score axes

0–100, the values in the table. An axis a system has no published value for is left as a gap (it counts as 0 in the composite).

Capability by subject topic

decisio v0.8.0 on gemma-4-31B-it: Language and category cells use the unchanged original S+P observations and all 1,505 newly completed frozen supplemental items (354 L1, 33 L2, 1,118 L3). Each supplement also carries the same original P300 answers by stable item identity; they count once in the union. The original S+P measurement has 1,477 native answers and 23 HTTP 500 errors, which remain operational non-answers and are not imputed. Headline, rank, Composite and original pricing are unchanged. L3 items were reviewed by an OpenAI model, so cells based on L3 carry that exposure; headline scores do not use L3. Decisio is the original locally served Gemma offering, not an OpenAI API offering.

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

What the decision is about — math and dates, coding, rules, money, support work, everyday messages, safety. Chance-corrected competence per category (0 = at or below chance; negative averages are reported as 0, 100 = perfect), decisio v0.8.0 on gemma-4-31B-it: S+P+L1+L2+L3; Jev 1.13.0: S+P+L1+L2+L3 items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.

What each category means · items per category

Rules, policy & law — applying written rules: company policies, contracts, regulations, eligibility. 1616 items (151 open / 1465 sealed)
Coding & software — code, SQL, repositories, developer tools and IT systems. 382 items (62 open / 320 sealed)
Math & numbers — a calculation decides the answer: arithmetic, word problems, probability, dates, units. 370 items (37 open / 333 sealed)
Finance & commerce — money: payments, refunds, invoices, orders, expenses, insurance payouts. 223 items (12 open / 211 sealed)
Support & operations — support tickets, incidents, logistics, scheduling desks and routing work to a team. 160 items (21 open / 139 sealed)
Everyday language — short everyday messages: intents, assistant requests, reading a detail out of a text. 145 items (11 open / 134 sealed)
Safety & security — untrusted or injected instructions, fraud, moderation, access and security triage. 109 items (6 open / 103 sealed)

Use cases (TypeSafe categories)

decisio v0.8.0 on gemma-4-31B-it: Language and category cells use the unchanged original S+P observations and all 1,505 newly completed frozen supplemental items (354 L1, 33 L2, 1,118 L3). Each supplement also carries the same original P300 answers by stable item identity; they count once in the union. The original S+P measurement has 1,477 native answers and 23 HTTP 500 errors, which remain operational non-answers and are not imputed. Headline, rank, Composite and original pricing are unchanged. L3 items were reviewed by an OpenAI model, so cells based on L3 carry that exposure; headline scores do not use L3. Decisio is the original locally served Gemma offering, not an OpenAI API offering.

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

The real-world application area of each decision, after the TypeSafe use-case map; an item can count in two. Chance-corrected competence per category (0 = at or below chance; negative averages are reported as 0, 100 = perfect), decisio v0.8.0 on gemma-4-31B-it: S+P+L1+L2+L3; Jev 1.13.0: S+P+L1+L2+L3 items pooled; only categories with at least 30 items are spokes, smaller ones are listed below. Hover a category for its definition and item count.A point on the bold 0 ring printed 0.0 is a measured value, not a gap: these published cells clip below-chance results to 0, so 0 means at or below chance. Their score markers cannot fall inside that ring.

What each category means · items per category

Model routing — deciding which LLM or model tier should handle a prompt; classifying a request's intent, domain, difficulty or risk to pick a model. 524 items (86 open / 438 sealed)
Legal and compliance — contracts, policies, regulations, eligibility rules, checking documents against written requirements. 515 items (74 open / 441 sealed)
Customer support — classifying tickets and customer intents, urgency, refunds, routing cases to a team, checking support replies. 240 items (37 open / 203 sealed)
Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 174 items (24 open / 150 sealed)
E-commerce marketplaces — product listings, product attributes, orders and deliveries, seller catalog policy. 119 items (9 open / 110 sealed)
Insurance claims — first-notice-of-loss reports, claim complexity, missing information, claim triage. 119 items (9 open / 110 sealed)
Risk assessment — estimating probabilities and severity of risks, incidents, vendor or underwriting risk. 118 items (8 open / 110 sealed)
Financial crime — suspicious transactions, KYC, fraud alerts, entity matching for investigations. 101 items (9 open / 92 sealed)
Feature extraction for predictive modeling — turning natural-language data into probabilistic features or estimates for a downstream prediction. 99 items (5 open / 94 sealed)
Lead generation — matching companies or inbound messages to an ideal customer profile, buyer intent, prioritising and routing sales leads. 95 items (4 open / 91 sealed)
Recruiting — resumes, applications, interview feedback, matching candidates to roles. 94 items (1 open / 93 sealed)
Graphs and knowledge graphs — entity types, relationships between records, contradictions between facts, multi-step lookups across linked facts. 94 items (3 open / 91 sealed)
LLM guardrails — checking LLM inputs, outputs and tool calls: prompt injection, jailbreaks, policy violations, tool-call errors, answer quality. 93 items (5 open / 88 sealed)
Moderation and trust and safety — user content moderation: toxicity, harassment, spam, unsafe advice, personal data exposure. 93 items (7 open / 86 sealed)
Semantic code linting — checking code or writing against conventions and guidelines, as in CI review of code. 90 items (3 open / 87 sealed)
Search and retrieval — scoring query-to-candidate relevance, reranking results, selecting useful context for RAG. 90 items (5 open / 85 sealed)
Scientific discovery — screening papers, labelling research passages or survey answers, checking that citations support claims, methodology checks. 90 items (2 open / 88 sealed)
Advertising — ad creatives, campaign copy, brand safety, prohibited claims in ads. 88 items (3 open / 85 sealed)
Gaming — player reports, in-game chat, game support. 86 items (3 open / 83 sealed)
Demand forecasting — purchase intent, product interest, demand and supply signals for forecasting. 83 items (3 open / 80 sealed)

Competence per request type, open / sealed

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Chance-corrected competence for Choice, Noul and Score on the 300 open and 1200 sealed decisions.

Competence per tier — open set

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence, the three request types pooled by their published decision counts.

Competence per tier — sealed set

Linear signed scale: −100 at the centre, 0 on the bold middle ring, 100 at the rim.

Per-tier competence on the sealed decisions, types pooled the same way; item text stays private.

How the categories were made

O1S chance-corrected competence: equal mean across the request types present, clipped to 0–100. Zero means the equal-weight mean across present request types was at or below baseline; negative means are clipped. It does not mean every type scored zero. Choice uses a random-option baseline; Noul uses a fixed 50% accuracy baseline (abstentions count as wrong); Score uses the midpoint-guess error (random-guess fallback when every gold is at the midpoint). Compare models within a category; row pools and supported types can differ.

Categories use ruled subject-topic labels and authoring use-case precedence across the answered pool union, with stable item identities counted once.

Official O1S; recorded failures count where supported; no missing response is imputed; stable identity dedupe parent first.
S-based category cells pool recorded supported responses from S+P and the answered L1/L2/L3 supplements. API overlay category cells keep their separately recorded A4/A5+P union. Headline and per-type/tier scores are unchanged.

All values as a table

Spoke	A: decisio v0.8.0 on gemma-4-31B-it	B: Jev 1.13.0
The four score axes
Intelligence	70.4	63.6
Calibration	88.7	90.6
Speed	91.2	91.5
Cost	51.7	54.7
Capability by subject topic
Rules, policy & law	68.9	49.4
Coding & software	80.8	64.3
Math & numbers	33.3	10.4
Finance & commerce	62.2	40.9
Support & operations	66.2	48.8
Everyday language	85.1	77.5
Safety & security	73.0	52.2
Use cases (TypeSafe categories)
Model routing	82.4	73.0
Legal & compliance	76.0	68.6
Customer support	67.8	49.6
Other	49.8	40.1
E-commerce	58.4	40.5
Insurance claims	57.0	46.1
Risk assessment	54.8	32.8
Financial crime	45.6	35.8
Feature extraction	57.7	28.3
Lead generation	73.9	49.2
Recruiting	39.6	0.0
Knowledge graphs	68.0	28.8
LLM guardrails	89.9	68.0
Moderation	75.7	51.6
Code linting	75.5	29.1
Search & retrieval	96.8	80.9
Science	52.3	21.2
Advertising	44.9	16.4
Gaming	31.5	5.1
Demand forecasting	7.5	0.0
Competence per request type, open / sealed
Choice · open	70.8	79.0
Choice · sealed	83.1	75.8
Noul · open	73.1	54.5
Noul · sealed	76.0	46.1
Score · open	58.8	63.3
Score · sealed	60.6	63.0
Competence per tier — open set
Easy	93.0	93.9
Standard	65.2	63.2
Judge	61.2	67.7
Hard	69.0	65.6
Competence per tier — sealed set
Easy	85.9	78.4
Standard	81.2	68.7
Judge	80.0	66.9
Hard	67.7	58.9

Published values

Measure	Jev 1.13.0 (TypeSafe AI)	decisio v0.8.0 on gemma-4-31B-it (frozen, FP8 on load, one prefill per state, self-hosted)
Capability Score	77.1	79.6
Rank (open-weights board)	Reference, not ranked	#2
Composite (secondary)	71.5	71.7
intelligence	63.6	70.4
calibration	90.6	88.7
speed	91.5	91.2
cost	54.7	51.7
USD per 1,000 decisions	$0.032 per 1,000 decisions	$0.041 per 1,000 decisions
p50 latency (seconds)	0.2	0.047 measured on-GPU, no network · 0.24 est. incl. network (2x + 150 ms)
Last measured	2026-10-05	2026-10-06

* Self-hosted rows: the first value is the p50 we measured on our own hardware over loopback (on-GPU, or on-CPU where marked, no network). It is not directly comparable with API latency, which we time end to end from our measuring host, network included. “est.” is the est. incl. network (2x + 150 ms) that the ranking and the latency cap use for every self-hosted row. Model authors can ask us to time a hosted endpoint instead (on request).

Jev 1.13.0 (TypeSafe AI)

proprietary API

Published source

Measured on: the operator's hosted API
Cost basis (not published): TypeSafe list tariff USD 0.042 per 1M input tokens, output free; measured on the v1.6.1 common cost basis (all items except those outside the Jev reference input range) from this run's token usage
Latency adjustment: none (API)

decisio v0.8.0 on gemma-4-31B-it (frozen, FP8 on load, one prefill per state, self-hosted)

Apache-2.0 software; Gemma model terms

Published source

Measured on: measured 2026-10-06 on the live v1.6.0/v1.6.1 pool. github.com/aminry/decisio tag v0.8.0 (5b42101a191d222062a795194b0aedb5003bf6ea) - the same tag whose other two entries run 54 measured - installed with their own `uv sync --extra serve --frozen` so their lockfile fixes every version, and served with their own `python -m decisio.serve.vllm_engine --base gemma-4-31b`. --base carries google/gemma-4-31B-it@842da379, the official weights, FROZEN: no fine-tuning, no adapter, no task registered; vLLM 0.30.0 quantizes it to FP8 when it loads (30.6 GiB). That profile's measured settings: temperature 4.672 for choice questions and 5.252 for the rest, the 12B entry's prompt unchanged, a system turn, the chat template's own answer position, every single-token form of each letter summed, no tokens generated. New in 0.8.0 on this base and the reason it is worth its own row: several questions of a state are scored in ONE pass after a single prefill. The author's own limit is a 32,768-token prompt, so the pool's 23 items at ~80,000 tokens are a documented capacity limit, not a defect. THE AUTHOR STATES ONE 96 GB CARD; this is one H100 80 GB (Lium) and no flag of theirs was changed to make it fit. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. add-requests run 55, GitHub #169
Cost basis (estimated): ESTIMATE (base-model reference; self-hosted open weights, no public tariff) and CLEAN: unlike run 54's FP8 sibling, the frozen pricing snapshot resolves this EXACT base - google/gemma-4-31B-it - to USD 0.09 per 1M input and USD 0.34 per 1M output. Read before the pod and asserted in make_registry_r55.py. x this row's own measured tokens, the deck31b method. Nothing is provisional and no release-lane line is open on it. Estimated, not charged
Latency adjustment: x2 + 0.15 s (assumption, not measured)

Frequently asked questions

What does JevBench show for Jev and decisio v0.8.0 on gemma-4-31B-it?: Jev 1.13.0 has Capability 77.1 and is the unranked reference (the open-weights board ranks 92 open-weights systems; hosted API offerings are ranked on the API leaderboard). decisio v0.8.0 on gemma-4-31B-it (frozen, FP8 on load, one prefill per state, self-hosted) has Capability 79.6. Rank #2 on the open-weights board in v1.6.1. Historical scores use their original method and are not ranked against the current release.
How does Capability differ from Composite?: Capability is the mean of Intelligence and Calibration; headline ranks require both official cost and median-latency caps. Composite additionally scores Speed and Cost and is secondary.
Can I compare the prices as actual bills?: Jev 1.13.0 (TypeSafe AI): not published, $0.032 per 1,000 decisions. decisio v0.8.0 on gemma-4-31B-it (frozen, FP8 on load, one prefill per state, self-hosted): estimated, $0.041 per 1,000 decisions. Estimates and self-reported vendor prices are not measured charges.
Is decisio v0.8.0 on gemma-4-31B-it open source?: The published licence note for decisio v0.8.0 on gemma-4-31B-it (frozen, FP8 on load, one prefill per state, self-hosted) is Apache-2.0 software; Gemma model terms. Jev 1.13.0 is listed as proprietary API. Check the linked sources for terms.