Hosting = where inference runs; company = where the provider or lab is registered.
EU hosting means the route's inference runs inside the EU: an EU region, AWS Bedrock's EU cross-region (geo) profiles, Azure's Europe Data Zone, or a provider whose entire public fleet is documented as EU-hosted, each checked per model against the provider's documentation. Global deployments do not count, and neither does an EU billing region, an EU company or an EU control plane on its own. One disclosed company-policy exception stays in, marked “EU equivalent”.
Evidence requirements (benchmark evidence, priced provider, measured task tokens) sit above the table they apply to.
“Hide deprecated” removes retired models (every Artificial Analysis configuration retired, on no current DesignArena board) and older retired configurations of current models. Applies to model tables, charts, price views & model offers; benchmark evidence stays unfiltered.
JevBench by Benchmark Heaven · released v1.6.1
No. Jev 1.13.0 is TypeSafe AI’s proprietary API; its weights are not published. But 105 open-weight Jev-class models were measured on the same benchmark, and one of them scores above Jev on the Capability Score.
Data: JevBench v1.6.1, published 2026-10-06
Ranks are open-weights board ranks in v1.6.1. Jev 1.13.0 is the unranked reference (Capability Score 77.1); hosted API offerings are ranked on the API leaderboard.
| Rank | System | Capability | Intelligence | Cost per 1,000 | License | Measured on |
|---|---|---|---|---|---|---|
| 1 | Quyet-1.0-Large (Chinh Nguyen, Gemma-4-31B decoder, option-letter logits) | 81.7 | 73.4 | $0.045 (estimated) | apache-2.0 | evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order 3643732b) |
| 4 | René-1 31B FP8 (salfatigroup, Gemma 4 31B full fine-tune, one-pass option readout) | 76.0 | 61.7 | $0.063 (estimated) | apache-2.0 | Candidate re-measured on 2026-10-07, current v1.6.0/v1.6.1 1500-item pool (sha256901983ae...). Immutable salfatigroup/rene-1-31b-fp8@bd634489957f8da63ccce858ee33fd6c2929d286, Apache-2.0. Same unmodified reviewed contributor source and run-39 transport shim. Author noul/yesno type mapping only; two-decimal probability rounding retained. One evaluator-owned RTX PRO6000 Blackwell 96GB pod, loopback-only network namespace, offline credential-free inference; no external hosted API, no retry or batching. Unmodified run_v16/typesafe adapter, serial one request at a time, selfhosted x2+0.15s latency adjustment. Missing ATTRIBUTIONS.md and NOTICE disclosed: training-data overlap cannot be checked. Florian GO in DECISIONS.md (go gor it, Apache2.0 confirmed) resolves prior attribution hold; publication remains release lane responsibility. COMPLETE: 1500/1500 distinct input IDs, 1477 HTTP200 native answers,23 HTTP422 refusals under the author's own32768-token state+question /65536-token request budget; every refusal counted wrong. Smoke passed all8 checks. All42 release-manifest files and6 prior source files hash-verified. Raw sha256 a28a8f0068f38b548f064a6e678bf63b13304f9add0a67b448dcc8003c9f38f1. Stack torch2.14.0, transformers5.17.0, compressed-tensors0.19.0, accelerate1.15.0. Provider pod91916684-9980-419c-98b7-c7a714a3912c removed and verified absent; approxUSD0.17. Fresh latest feed places this candidate inside both official caps; material open-board top-five change requires release-lane preview disposition before publication. |
| 5 | H2O-Lightning-4B v1.1 (H2O.ai, Qwen3.5-4B fine-tune, stock vLLM + open shim) | 75.0 | 60.0 | $0.021 (estimated) | apache-2.0 | evaluator-owned Lium GPU pod, 1x NVIDIA RTX 5090, official vLLM 0.30.0 image + the author's open standard-library shim (h2o_lightning_shim.py) as published on the model card, offline (all credentials unset), driven over loopback only by the JevBench client; label-free inputs; pod destroyed afterwards; JevBench add-request (GitHub #181), measured 4 Oct 2026 |
| 6 | Bobcat Flash 1.2 (Gemma-4-26B-A4B-it + two merged rank-64 LoRA adapters, typed-decision readout at the first answer position, FP8, self-hosted) | 73.5 | 65.8 | $0.048 (estimated) | apache-2.0 | measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights sanghwa-na/bobcat-flash-1.2@58d15456937ce58625b13a5e3d982846554a3aef, Apache-2.0, 51.68 GB BF16 - a merge of two rank-64 LoRA adapters on google/gemma-4-26B-A4B-it@4d7ae4984b7db7de8f8457170b3f1a419ee76d52 (0.6*A + 0.4*B, B being the Flash 1.1 adapter), file-by-file verified against the Hub's own manifest before the engine was started. Code github.com/foxl-ai/bobcat@43387a09bbc0310edcf57e2b21fa59ea81ac4166, Apache-2.0. NO MAPPING DECISION OF OURS EXISTS FOR THIS ROW, which is why it is worth saying so explicitly. The author implements TypeSafe's published System One contract, and their own src/bobcat/protocol.py parse_request accepts the JevBench canonical record EXACTLY as the UNCHANGED jevbench.adapters.typesafe already emits it: choice -> a dict of 1-255 named criteria, so the item's own labels are the keys and its own criteria[label] strings travel verbatim as the descriptions (their protocol renders {name, description} per option); score -> the item's ordered array of levels, which their own parse_request labels '0'..'n-1' - the same keys JevBench uses - with the item's own level text as each description, so the ordinal order is carried; noul -> criteria restricted to {true,false}, and the item's OWN true/false description strings are delivered (unlike run 56's APUS rows, whose runtime required canonical criteria there), with the answer returned as noul = P(yes). Their answer() refuses to reply unless the probabilities sum to one. Nothing was rewritten, no option was dropped, no item type was recorded as a miss, and no shim of ours sits anywhere in the path. STACK: the author's own published install line, verbatim from their model card's Quickstart - a uv-managed Python 3.12 (their note: it ships the headers Triton compiles against) and vllm==0.30.0 LETTING IT RESOLVE ITS OWN STACK, which gave torch 2.13.0+cu130, transformers 5.19.0, tokenizers 0.23.2. Served by THEIR OWN SERVER, bobcat.flash_server, with THEIR OWN published command verbatim: --engine vllm --quantization fp8 (their online dynamic FP8 of these BF16 weights, the served configuration) --temperature choice=0.594604,noul=0.353553,score=0.529732 --max-num-seqs 64 --max-model-len 98368 --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN, with VLLM_USE_FLASHINFER_SAMPLER=0 as their command sets it. --local only because the server is on loopback on our own pod. One H100 PCIe 80 GB (Lium) - the second of the two cards the author measured. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. No file of theirs was edited and no prompt of ours exists. CALIBRATION IS THEIR PUBLISHED DEFAULT AND WAS NOT CHOSEN BY US. Their card ships two calibrations: the per-type triple used here, which is the Quickstart's - i.e. the published serve command of this release - and a single product temperature 0.9232 fitted by log-loss on their six product tasks. The triple was bound in PREREG-R57 s3.3 before any item was sent and the alternative was deliberately NOT probed on our items, because the author has ALREADY PUBLISHED both configurations' numbers on JevBench's public 231 (per-type Capability 68.2 = 57.5/78.9; product 68.3 = 54.3/82.3), so there was nothing to learn from our pool and choosing between them from our own measurement would have been exactly the mistake run 56's probe was designed to avoid. The author states the triple was fitted on a held-out split of 8,906 JevBench-FORMAT decisions (generated and public) for the highest JevBench-style Capability, and separately that no JevBench item was used for training, calibration, selection or prompt design, with generated rows audited against JevBench's public items at 0 exact and 0 8-gram overlaps. THE 23 LONG ITEMS: the server's limit is --max-model-len 98368 (their compiler gets max_model_len - 64) against a pool whose longest item is 128,111 BYTES of canonical record, and their RequestLimitError answers 422 with 'Inputs are never truncated' rather than truncating. This row therefore reads every item in full. Serial, one decision per request, no retry, no batching, offline, credential-free. add-requests run 57, benchmarkheaven.com/submit ref 7A714784 (bh-submit e9d576d7-e1f1-49d8-8151-05cdbbdd7ed4) |
| 7 | Surogate Rune 26B-A4B v3 (v1.6 pool) | 73.5 | 56.1 | $0.048 (estimated) | apache-2.0 | evaluator-owned Lium GPU pod, 1x NVIDIA RTX PRO 6000 Blackwell 96GB, a thin transport shim around the author's engine in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 |
| 8 | decider-12b v2 (Mapika) | 72.5 | 63.0 | $0.026 (estimated) | apache-2.0 | Current frozen 1500-item pool measured 2026-10-04 by coordinator plan P83 on one evaluator-owned Lium RTX PRO6000. Offline contributor image, serial loopback-only requests, no batching or retry; standard selfhosted x2+0.15 s latency adjustment. Current stage receipt verifies the same model pin used by the prior add-request run. Cost is the carried documented estimate recorded in current registry. |
| 9 | Xor 26B-A4B (Juspay, Gemma-4-26B-A4B, bf16) | 72.4 | 58.7 | $0.044 (estimated) | apache-2.0 | evaluator-owned Lium GPU pod, 1x NVIDIA RTX PRO 6000 Blackwell 96GB (the author's card class), the author's own serving bundle (SGLang image pinned by digest + their compatibility server, single GPU via their documented switch), offline (all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 |
| 10 | decider-12b v1, stock Gemma-4-12B-it (Mapika) | 72.2 | 60.6 | $0.026 (estimated) | apache-2.0 | Current frozen 1500-item pool measured 2026-10-04 by coordinator plan P83 on one evaluator-owned Lium RTX PRO6000. Offline contributor image, serial loopback-only requests, no batching or retry; standard selfhosted x2+0.15 s latency adjustment. Current stage receipt verifies the same model pin used by the prior add-request run. Cost is the carried documented estimate recorded in current registry. |
| 11 | torchcast-decision-12b (Torchcast AI, Gemma-4-12B fine-tune, option-letter logprob readout) | 71.7 | 60.5 | $0.028 (estimated) | cc-by-nc-4.0 | evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order b113eac1) |
| 12 | Jev-Omni (akhilaaa3, Gemma-4-12B merged) | 71.3 | 55.5 | $0.029 (estimated) | Apache-2.0, following Gemma 4; dataset rights stated separately by the author | evaluator-owned Lium GPU pod (RTX6000), offline read-only container |
Jev was measured through its production API from a server in Germany, network included, at not published $0.032 per 1,000 decisions. Most open models were measured on GPUs we rented or on their authors’ endpoints; rows on our own hardware carry an assumed speed adjustment, and their cost is usually an estimate from a comparable hosted price. Self-hosting keeps decisions on your own hardware, which the API cannot.
“Open” here means the published row lists public code or weights. Terms differ: some rows are Apache-2.0 or MIT, others inherit a base model’s use policy, and a few are non-commercial research licenses. Each row keeps its own license text.
Head-to-head: Jev vs decider-4b v2 · Jev vs Cygnet · Jev vs Hopper · Jev vs Laya