Best Jev alternatives (October 2026)

Top 15 by Capability Score

Hosted API alternatives are ranked on the API leaderboard (Sage 1.3.0, Mercury Decide, wity-1, SPX-CD Flash, SPX-CD Pro, Fastino GLiNER-2.5-Decide): JevBench API leaderboard.

Open-weight Jev-class alternatives

Only current-release measurements are shown; Jev 1.13.0 is the reference row. Rows outside the caps have no Capability rank. Read the licence and recorded setup before self-hosting.

• Quyet-1.0-Large (Chinh Nguyen, Gemma-4-31B decoder, option-letter logits): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order 3643732b) · Published source or weights
• René-1 31B FP8 (salfatigroup, Gemma 4 31B full fine-tune, one-pass option readout): apache-2.0 · Candidate re-measured on 2026-10-07, current v1.6.0/v1.6.1 1500-item pool (sha256901983ae...). Immutable salfatigroup/rene-1-31b-fp8@bd634489957f8da63ccce858ee33fd6c2929d286, Apache-2.0. Same unmodified reviewed contributor source and run-39 transport shim. Author noul/yesno type mapping only; two-decimal probability rounding retained. One evaluator-owned RTX PRO6000 Blackwell 96GB pod, loopback-only network namespace, offline credential-free inference; no external hosted API, no retry or batching. Unmodified run_v16/typesafe adapter, serial one request at a time, selfhosted x2+0.15s latency adjustment. Missing ATTRIBUTIONS.md and NOTICE disclosed: training-data overlap cannot be checked. Florian GO in DECISIONS.md (go gor it, Apache2.0 confirmed) resolves prior attribution hold; publication remains release lane responsibility. COMPLETE: 1500/1500 distinct input IDs, 1477 HTTP200 native answers,23 HTTP422 refusals under the author's own32768-token state+question /65536-token request budget; every refusal counted wrong. Smoke passed all8 checks. All42 release-manifest files and6 prior source files hash-verified. Raw sha256 a28a8f0068f38b548f064a6e678bf63b13304f9add0a67b448dcc8003c9f38f1. Stack torch2.14.0, transformers5.17.0, compressed-tensors0.19.0, accelerate1.15.0. Provider pod91916684-9980-419c-98b7-c7a714a3912c removed and verified absent; approxUSD0.17. Fresh latest feed places this candidate inside both official caps; material open-board top-five change requires release-lane preview disposition before publication. · Published source or weights
• H2O-Lightning-4B v1.1 (H2O.ai, Qwen3.5-4B fine-tune, stock vLLM + open shim): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA RTX 5090, official vLLM 0.30.0 image + the author's open standard-library shim (h2o_lightning_shim.py) as published on the model card, offline (all credentials unset), driven over loopback only by the JevBench client; label-free inputs; pod destroyed afterwards; JevBench add-request (GitHub #181), measured 4 Oct 2026 · Published source or weights
• Bobcat Flash 1.2 (Gemma-4-26B-A4B-it + two merged rank-64 LoRA adapters, typed-decision readout at the first answer position, FP8, self-hosted): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights sanghwa-na/bobcat-flash-1.2@58d15456937ce58625b13a5e3d982846554a3aef, Apache-2.0, 51.68 GB BF16 - a merge of two rank-64 LoRA adapters on google/gemma-4-26B-A4B-it@4d7ae4984b7db7de8f8457170b3f1a419ee76d52 (0.6*A + 0.4*B, B being the Flash 1.1 adapter), file-by-file verified against the Hub's own manifest before the engine was started. Code github.com/foxl-ai/bobcat@43387a09bbc0310edcf57e2b21fa59ea81ac4166, Apache-2.0. NO MAPPING DECISION OF OURS EXISTS FOR THIS ROW, which is why it is worth saying so explicitly. The author implements TypeSafe's published System One contract, and their own src/bobcat/protocol.py parse_request accepts the JevBench canonical record EXACTLY as the UNCHANGED jevbench.adapters.typesafe already emits it: choice -> a dict of 1-255 named criteria, so the item's own labels are the keys and its own criteria[label] strings travel verbatim as the descriptions (their protocol renders {name, description} per option); score -> the item's ordered array of levels, which their own parse_request labels '0'..'n-1' - the same keys JevBench uses - with the item's own level text as each description, so the ordinal order is carried; noul -> criteria restricted to {true,false}, and the item's OWN true/false description strings are delivered (unlike run 56's APUS rows, whose runtime required canonical criteria there), with the answer returned as noul = P(yes). Their answer() refuses to reply unless the probabilities sum to one. Nothing was rewritten, no option was dropped, no item type was recorded as a miss, and no shim of ours sits anywhere in the path. STACK: the author's own published install line, verbatim from their model card's Quickstart - a uv-managed Python 3.12 (their note: it ships the headers Triton compiles against) and vllm==0.30.0 LETTING IT RESOLVE ITS OWN STACK, which gave torch 2.13.0+cu130, transformers 5.19.0, tokenizers 0.23.2. Served by THEIR OWN SERVER, bobcat.flash_server, with THEIR OWN published command verbatim: --engine vllm --quantization fp8 (their online dynamic FP8 of these BF16 weights, the served configuration) --temperature choice=0.594604,noul=0.353553,score=0.529732 --max-num-seqs 64 --max-model-len 98368 --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN, with VLLM_USE_FLASHINFER_SAMPLER=0 as their command sets it. --local only because the server is on loopback on our own pod. One H100 PCIe 80 GB (Lium) - the second of the two cards the author measured. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. No file of theirs was edited and no prompt of ours exists. CALIBRATION IS THEIR PUBLISHED DEFAULT AND WAS NOT CHOSEN BY US. Their card ships two calibrations: the per-type triple used here, which is the Quickstart's - i.e. the published serve command of this release - and a single product temperature 0.9232 fitted by log-loss on their six product tasks. The triple was bound in PREREG-R57 s3.3 before any item was sent and the alternative was deliberately NOT probed on our items, because the author has ALREADY PUBLISHED both configurations' numbers on JevBench's public 231 (per-type Capability 68.2 = 57.5/78.9; product 68.3 = 54.3/82.3), so there was nothing to learn from our pool and choosing between them from our own measurement would have been exactly the mistake run 56's probe was designed to avoid. The author states the triple was fitted on a held-out split of 8,906 JevBench-FORMAT decisions (generated and public) for the highest JevBench-style Capability, and separately that no JevBench item was used for training, calibration, selection or prompt design, with generated rows audited against JevBench's public items at 0 exact and 0 8-gram overlaps. THE 23 LONG ITEMS: the server's limit is --max-model-len 98368 (their compiler gets max_model_len - 64) against a pool whose longest item is 128,111 BYTES of canonical record, and their RequestLimitError answers 422 with 'Inputs are never truncated' rather than truncating. This row therefore reads every item in full. Serial, one decision per request, no retry, no batching, offline, credential-free. add-requests run 57, benchmarkheaven.com/submit ref 7A714784 (bh-submit e9d576d7-e1f1-49d8-8151-05cdbbdd7ed4) · Published source or weights
• Surogate Rune 26B-A4B v3 (v1.6 pool): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA RTX PRO 6000 Blackwell 96GB, a thin transport shim around the author's engine in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• decider-12b v2 (Mapika): apache-2.0 · Current frozen 1500-item pool measured 2026-10-04 by coordinator plan P83 on one evaluator-owned Lium RTX PRO6000. Offline contributor image, serial loopback-only requests, no batching or retry; standard selfhosted x2+0.15 s latency adjustment. Current stage receipt verifies the same model pin used by the prior add-request run. Cost is the carried documented estimate recorded in current registry. · Published source or weights
• Xor 26B-A4B (Juspay, Gemma-4-26B-A4B, bf16): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA RTX PRO 6000 Blackwell 96GB (the author's card class), the author's own serving bundle (SGLang image pinned by digest + their compatibility server, single GPU via their documented switch), offline (all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• decider-12b v1, stock Gemma-4-12B-it (Mapika): apache-2.0 · Current frozen 1500-item pool measured 2026-10-04 by coordinator plan P83 on one evaluator-owned Lium RTX PRO6000. Offline contributor image, serial loopback-only requests, no batching or retry; standard selfhosted x2+0.15 s latency adjustment. Current stage receipt verifies the same model pin used by the prior add-request run. Cost is the carried documented estimate recorded in current registry. · Published source or weights
• torchcast-decision-12b (Torchcast AI, Gemma-4-12B fine-tune, option-letter logprob readout): cc-by-nc-4.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order b113eac1) · Published source or weights
• Jev-Omni (akhilaaa3, Gemma-4-12B merged): Apache-2.0, following Gemma 4; dataset rights stated separately by the author · evaluator-owned Lium GPU pod (RTX6000), offline read-only container · Published source or weights
• Winnow-12B Q8: Apache-2.0, including the applicable Gemma 4 base/derivative licence terms · evaluator-owned Lium GPU pod (RTX6000), offline read-only container · Published source or weights
• Cygnet (blockbrain, frozen Gemma-4-12B-it): shim MIT; weights Apache-2.0 with Google's Gemma Prohibited Use Policy · evaluator-owned Lium GPU pod (RTXPRO6000), offline read-only container · Published source or weights
• decider chat on Gemma-4-31B-it (Mapika, frozen base, inference technique): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA RTX PRO 6000 Blackwell 96GB, the author's own server in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• GEV-26B-Decide (AutoTrust, Gemma-4-26B-A4B + LoRA + head): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, a thin transport shim around the author's engine in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• Hopper 12B trained (gemma-4-12B-it frozen + LoRA r32 unmerged, one forward pass, self-hosted): apache-2.0 · measured 2026-10-06 on the live v1.6.0/v1.6.1 pool. Serving code github.com/hopit-ai/hopper tag 12b-1.1.1, installed with the author's own one-line pip install; adapter HopitAI/hopper@4b605f25eb903c2cf4d121040ffb097240b3bc37 (branch hopper-12b), LoRA r32 alpha 64, loaded UNMERGED because the author states a bf16 merge changes answers; base google/gemma-4-12B-it@707f0a3b, frozen; readout map maps/hopper-12b-trained-readout.json as shipped in their package. 1.1.1 is their own fix for corrupted option scores at prompt lengths one above a multiple of 32. Their flags left at their defaults: --shortlist tournament (their measured default) for menus over 26 options, --cuda-graphs OFF. The author asked on 6 Oct 2026 that the TRAINED version replace the untrained Hopper 12B request in GitHub #164; this is that row. NOT the published `hopper` row (HopitAI, rank 25), a different system. One RTX 6000 Ada 48 GB (Lium, Toronto); the author ran theirs on an L40S. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. add-requests run 54, GitHub #164 · Published source or weights
• SPX-CD-Omni (SurdAI, google/gemma-4-12B-it + LoRA r32, one forward pass, self-hosted): apache-2.0 · measured 2026-10-06 on the live v1.6.0 S u P (1,500 rows, 1,477 ok; the 23 refusals are exactly the 23 long-context items, through the author's own "Prompt exceeds context limit; no automatic truncation" guard at their shipped --max-context 16384, although the base model itself declares 262,144 positions). Author's own code unmodified (infer.py / decision.py, sha256 1c5db5de.../43389f44...) with their shipped --backend transformers on the unquantized base and the flags their own EVALUATION.md prescribes FOR JEVBENCH: --prompt-format open-format --effort 1 --temperature 1.0. Their requirements-transformers.txt is missing torchvision, which transformers 5.16.1 needs for the Gemma 4 unified processor; adding it was the only deviation and it changes no numerics. One RTX PRO 6000 Blackwell (Lium, Brasov, Romania), torch 2.12.0+cu130, transformers 5.16.1, peft 0.20.0. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment; raw p50/p95 are in the run 53 receipts; add-requests run 53, benchmarkheaven.com/submit ref 75D5EF0C · Published source or weights
• Aplomb 1 (5.3B decision model on Qwen3.5-4B, trained readout head, self-hosted): EmpirioLabs Model License 1.0 · measured 2026-10-06 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights empiriolabsai/aplomb-1@f8e2dc81f0505db21a3b7a933ea479658e4ebb8f, bf16, 5.3 B, base Qwen/Qwen3.5-4B plus the shipped 1.3 GB audio encoder and the trained DecisionHead; click-through gate accepted under the EmpirioLabs Model License 1.0 section 2.1, which grants evaluation and the publication of benchmark results without asking. Served behind the author's own TypeSafe wire format by a TRANSPORT SHIM ONLY: run_aplomb.main()'s loading block verbatim, every request handed to THEIR run_aplomb.decide() with THEIR edm_config.json temperatures and abstain_bias and debias=True - the configuration the author names as the reference in #195 and in the model card. Their own aplomb/schema.py documents the request schema as wire-compatible with TypeSafe /v1/systemone and their aplomb/readout.py already emits the answer dicts the unchanged typesafe adapter parses, so no answer field is rewritten and no contributor file is edited. STACK: the author's own documented install line, with transformers pinned to 5.17.0 - the version their own audio sidecar was saved with; their loader compares the saved config to the live one and 5.19.0 fails it on the transformers_version string alone. SPEED CAVEAT: their documented install line does not include causal_conv1d, so transformers falls back to a reference PyTorch implementation it itself calls much slower; the Speed axis is measured on the stack the author documents. One H100 80 GB (Lium). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. add-requests run 55, GitHub #195 · Published source or weights
• Bespoke Nimble 9B v3 (Qwen3.5-9B LoRA): cc-by-nc-4.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, a thin transport shim around the author's engine in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• decider-4b v2 (Mapika): Apache-2.0 (package and weights) · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• Clef-Flash (Cloudflare, Qwen3.5-9B post-train with a joint schema head, multimodal, measured on text): apache-2.0 · evaluator-owned Lium GPU pod (H100 80GB), offline (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE), credential-free, server bound to 127.0.0.1, harness on the same pod over loopback HTTP; Lium pod 08a1ff4a-248a-4ff2-948e-ea558f99e808 (huid cosmic-hawk-4f, node calm-fox-65), 1x NVIDIA H100 80GB HBM3, United States, USD 1.30/h · Published source or weights
• janus 4B (Icarus AI / cmxu, Qwen3.5-4B + LoRA r64 and pointer decision head): apache-2.0 · Current frozen 1500-item pool measured 2026-10-04 by coordinator plan P83 on one evaluator-owned Lium RTX PRO6000. Offline contributor image, serial loopback-only requests, no batching or retry; standard selfhosted x2+0.15 s latency adjustment. Current stage receipt verifies the same model pin used by the prior add-request run. Cost is the carried documented estimate recorded in current registry. · Published source or weights
• JevK5 v0.2.0: Apache-2.0 (code/adapter); Apache-2.0 (Qwen3.5-4B base) · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• Hopper: Component-specific terms recorded in RESULT.md; submitted adapter release and Qwen base retain their respective terms · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• jqv (Qwen3-32B zero-shot): Apache-2.0 (Qwen3-32B weights); serving code public · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• metask-jev-4b: Apache-2.0 · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• Kahn1 4B (Okura66, Qwen3.5-4B LoRA merge, author's sysone engine): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA RTX 5090, the author's own sysone engine (github.com/Okura66/kahn1) on vLLM 0.30.0 with the author's documented max-model-len 4096 and calibration.json, offline (all credentials unset), driven over loopback only by the JevBench client through a thin pre-registered adapter for the author's /v1/evaluate/jev endpoint; label-free inputs; pod destroyed afterwards; JevBench add-request, measured 7 Oct 2026 · Published source or weights
• lev (Interfaze AI, Qwen3.5-4B + LoRA r32, label-token readout + candidate-path head, per-bucket temperatures): apache-2.0 · evaluator-owned Lium GPU pod (H100 80GB), offline (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE), credential-free, server bound to 127.0.0.1, harness on the same pod over loopback HTTP; Lium pod 08a1ff4a-248a-4ff2-948e-ea558f99e808 (huid cosmic-hawk-4f, node calm-fox-65), 1x NVIDIA H100 80GB HBM3, United States, USD 1.30/h · Published source or weights
• Malkuth-4B (newfull5, Kev post-train): CC-BY-NC-4.0, research use only (XNLI and RACE in the training mix) · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• Quyet-1.0-Medium (Chinh Nguyen, Qwen3.5-4B decoder, option-letter logits): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order 3643732b) · Published source or weights
• Decision-4B (Eval Engine / Chromia): apache-2.0 · Current frozen 1500-item pool measured 2026-10-04 by coordinator plan P88 on one evaluator-owned Lium H100. Current staged log verifies evalengine/decision-4b-gguf@08859d1d8307f7b7ea452549e303b7805c35e350; stored GGUF SHA256 f2519bcdc945291974c81a37ff92417b22df27154c9df2564420d08576b907ce. Serial offline loopback-only inference, standard selfhosted x2+0.15 s latency adjustment. Carried published estimate retained. · Published source or weights
• Manchego v2.1: Apache-2.0 (code and weights, author-declared) · evaluator-owned offline RTX6000 GPU · Published source or weights
• Messier One v0.2 (Qwen3.5-4B fine-tune, one prefill, self-hosted): cc-by-nc-4.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (sha256 901983ae...), 1,500 of 1,500 answered, 0 invalid. REPLACES the v0.1 request (GitHub #194, measured by run 54); the submission names it so. Weights agentmessier/messier-one tag v0.2 = 8bbddb84140e8fc732b35713f998ed628fc70306 (CC BY-NC 4.0, NON-COMMERCIAL), front end github.com/agentmessier-ai/messier-one tag v0.2 = 1b8c3b80 (Apache-2.0). README install line (vllm==0.30.0, transformers==5.18.0; torch 2.13.0+cu130 resolved by vllm) and the author's two serve commands verbatim; --gate off, their default and the configuration of their own reported single-read run. The author discloses that part of the training material was written in the families of JevBench's hard tier, with no JevBench item used. Their Prompter caps the state at 8,000 tokens (max_total 8,600) and splices long states around a readable marker, so the 23 long items are answered truncated (r54 s on disclosed truncation). UNCHANGED typesafe adapter; the smoke gate refused the first attempt at usage-input-tokens and passed with its existing pre-registered opt-in. One RTX 5090 32 GB (Lium, France). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. Serial, one decision per request, no retry, no batching, offline, credential-free. add-requests run 59, benchmarkheaven.com/submit ref 455A3944 · Published source or weights
• Diffusion Jev (DiffusionGemma 26B-A4B on patched SGLang, 48-step diffusion readout, self-hosted): apache-2.0 (model card metadata); linked Gemma 4 model terms · measured 2026-10-06 on the live v1.6.0/v1.6.1 pool. github.com/Hangzhi/diffusion-jev-sglang@d61f0a9c0c9b6d78631aa250125dc4b20e9ea0a2 serving google/diffusiongemma-26B-A4B-it@f7f5b7f5fa82ffc52addd066915886d497f5517b on SGLang@ef90bf677e4e6b280bf796f1c5f4ef0f05dda27f (their open PR 34061) with their own version-checked readout and context-admission patch. Built from THEIR OWN tested container recipe (deployment/modal_app.py + deployment/gemma-requirements.txt), which their doc names as the recipe to use; their `engine` extra and top-level Dockerfile target the older LLaDA deployment and their doc says plainly not to use them as a DiffusionGemma installer, so they were not used. SGLang is a pure-python install imported through PYTHONPATH, no source compile. Served with their own scripts/launch_diffusiongemma.py: native Gemma4Renoise, a 256-position canvas, up to 48 adaptive denoising steps, seed 42, thinking disabled, 8,192-token context, Triton attention, no CUDA graphs, no prefix cache, T=1 softmax over the options offered - their configuration, unchanged, on one H100 80 GB (Lium) against the A100 80 GB they tested. THE AUTHOR'S OWN STATEMENT, CARRIED UNSOFTENED: these are self-conditioned denoiser scores, NOT independent masked-token likelihoods and NOT calibrated probabilities of correctness; this is not a one-forward-pass or zero-generation method. There are no new trained weights - it is an inference configuration over Google's unchanged checkpoint. A format mismatch returns their own HTTP 503 and is recorded invalid, the same way for everyone. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. add-requests run 55, GitHub #30 · Published source or weights
• jev-local (Qwen3.5-9B): no licence stated in the repository (public code); Apache-2.0 base weights · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• JEV Qwen3.5-9B Base NVFP4: Apache-2.0 entrant and upstream checkpoint · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• CoCo-Decision-4B (corners-ai, LoRA on Qwen3.5-4B, option-label logits of one pass, served by oh-my-jev): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (sha256 901983ae...), 1,477 of 1,500 answered. LoRA adapter corners-ai/CoCo-Decision-4B-Ko tag v1.0.0 = commit b580ede993493690212a07ed0e6cb972868cc808 (the submission's 'Version 1.0.0', the card's own pin; Apache-2.0) on Qwen/Qwen3.5-4B @ 851bf6e8 as its own adapter_config.json names it. Toolkit github.com/iamupd/oh-my-jev @ f0b8c786 (Apache-2.0) installed with its README's `uv sync` plus the semif extra the README names for --adapter (torch 2.9.1+cu128, transformers 5.17.0, peft 0.21.0). Served by the author's own `omj serve --backend semif --adapter <adapter dir>`: the README documents that --adapter runs locally on semif and loads the adapter's base, in 4-bit only when it would not fit; on the 32 GB card omj itself chose bf16 ('Using Qwen/Qwen3.5-4B (the adapter's base model, bf16)'), the card's headline configuration. NO config.toml was authored and no setting of ours was chosen, which removes run 58's blocker. UNCHANGED typesafe adapter; all eight checks of the unchanged smoke gate passed on synthetic requests before any sealed item was sent. THE 23 LONG ITEMS ARE REFUSED with the documented 422 context_length_exceeded at omj serve's default 4,096-token limit (README 'Known limitations'); they are the system's own disclosed refusal, counted wrong, and no other item failed. One RTX 5090 32 GB (Lium, France). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. Serial, one decision per request, no retry, no batching, credential-free. add-requests run 59, benchmarkheaven.com/submit ref 619F3346 (bh-submit 0195e1e7) · Published source or weights
• APUS-OpenJev-v1-9B (merged bf16 checkpoint-3000 on Qwen3.5-9B, their own native runtime, full depth, self-hosted): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights apus-ailab/APUS-OpenJev-v1-9B@82c9c56cfa9de8d36704ed91948d4726ef111635, Apache-2.0, 17.55 GiB bf16 merged weights, checkpoint-3000, declared base Qwen/Qwen3.5-9B, full depth 32 layers. STACK: the authors' own requirements.txt, unmodified - torch==2.8.0 and transformers==5.16.1. The transformers pin is THEIRS, not a workaround: OpenJet.__init__ raises 'Layer execution is audited for transformers==5.16.1' on any other version. Served behind a TRANSPORT SHIM ONLY (r56/serve/apus_systemone_r56.py): loaded with THEIR OpenJet.from_pretrained(device=cuda:0, dtype=bfloat16) and every request handed to THEIR decide(), returning THEIR format_response() output. No contributor file edited. effort='high', the full-depth path, their decide() default and the configuration their own headline Frozen80 number is quoted on; effort='low' is not measured and no routing between the two is invented. One H100 PCIe 80 GB (Lium). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. MAPPING (run 56's, not the authors'; pre-registered in PREREG-R56 s2.2 before any item was sent and repeated here because it is part of the number): their runtime accepts primitive in {choice, noul, score_level} with 2-16 candidates labelled A-P. choice -> their 'choice' primitive, candidates = the item's own labels as id and its own criteria[label] as description, in labels order; their format_response returns probabilities keyed by candidate id, i.e. by the JevBench label already, so nothing is rewritten. All 750 choice items carry 3-9 options, inside their 2-16 window. noul -> their 'noul' primitive; their contract REQUIRES the canonical yes/no criteria there, so the item's own true/false description strings are NOT delivered - the proposition itself travels verbatim in instructions, which is what their noul is built to judge. score -> their 'choice' primitive over the ordinal levels, because their own RUNTIME.md rules score_level out as an ordinal Score API ('score_level is the judgement of a single proposition, it cannot be treated as a complete ordinal Score API'); the 4-5 ordered levels become candidates with id = the level index and description = the item's own level text, which carries the order. Recording all 375 score items as misses instead would understate a system whose published interface answers them. CALIBRATION IS MEASURED AS THE AUTHORS SHIP IT AND THEY DISCLOSED IT IN ADVANCE: their response carries calibrated: False and both their card and RUNTIME.md state the probabilities are an uncalibrated relative softmax over the offered candidates. OUTPUT TOKENS ARE 0 BY CONSTRUCTION, NOT BY ESTIMATE: decide() runs one forward pass with logits_to_keep=1 and reads candidate logits; nothing is generated anywhere in that path. Input tokens are their own prompt_tokens field. THE 23 LONG ITEMS: their compile() raises ValueError('input exceeds runtime limit; no truncation permitted') above 8,192 tokens and their RUNTIME.md states it in advance. Recorded invalid. This is the first row in the series whose refusal on those items is guaranteed in the author's own code with an explicit no-truncation promise. add-requests run 56, GitHub #106, benchmarkheaven.com/submit ref 2033065A · Published source or weights
• typecastlm (Mikhail Gribov, Qwen3.5-4B computed head): Apache-2.0 (package and weights) · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• TypeCastLM 1.4.0 (Mikhail Gribov, Qwen3.5-4B computed head): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (sha256 901983ae...), 1,500 of 1,500 answered, 0 invalid. A FOLLOW-UP OF THE PUBLISHED typecastlm ROW: the author asks to 'measure 1.4.0 to replace it' and to list it as 'TypeCastLM'; measured here as a CANDIDATE - replace-or-keep-both is the release lane's decision. Weights mihailgribov/typecastlm-qwen3.5-3.8b tag v1.4.0 = commit 0984c3e832f7c28495134bc1d22c66bb0e0a555d (GGUF files excluded), package typecastlm[server]==1.2.2 (latest on PyPI) + flash-linear-attention + fla-core, the model card's own install lines, in an isolated venv (torch 2.14.1+cu130, transformers 5.19.0). Served with the author's own `typecastlm-serve --model <local v1.4.0 snapshot> --host 127.0.0.1 --port 8000 --device cuda`; the UNCHANGED typesafe adapter with the published row's own wire model; all eight checks of the unchanged smoke gate passed on synthetic requests before any sealed item was sent. THE 23 LONG ITEMS ARE ANSWERED, folded in the middle to the server's own 32,768-token state context (its documented behaviour; max input 32,953 tokens observed). One RTX 5090 32 GB (Lium, France). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. Serial, one decision per request, no retry, no batching, credential-free. add-requests run 59, benchmarkheaven.com/submit ref 34CA4AED · Published source or weights
• SemIf, formerly OpenJev (Qwen3.5-4B, TheoLeeCJ): MIT (code); Qwen3.5 weights Apache-2.0 · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• local-jev Qwen3.5-4B: MIT code; Apache-2.0 Qwen weights · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• Decision 2B (FlyMy.AI, v59): Apache-2.0 notices on the included code and the pinned base; the weights are an evaluation preview under EVALUATION-PERMISSION.md, not a cleared commercial release · evaluator-owned Lium GPU pod (RTX6000), offline read-only container · Published source or weights
• spark-s1-4b-v6 (Open Spark Jev, abhishek085): Apache-2.0 (code and weights); base Qwen/Qwen3.5-4B Apache-2.0 · evaluator-owned Lium GPU pod (RTX6000), offline read-only container · Published source or weights
• ClassOne Qwen 3.5 9B (Qwen3.5-9B backbone with ClassOne decision heads, schema-constrained single forward pass, self-hosted): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (sha256 901983ae...). Weights devops-thiago/classone-qwen3.5-9b (sha a45d16f1f5d1840a420f8ede59fa5cca3f079961), Apache-2.0, public and ungated; code github.com/devops-thiago/class-one, Apache-2.0. ONE DOCUMENTED MAPPING OF OURS, AND IT IS THE ONLY ONE. The author's server keys SCORE probabilities '1'..'n' while JevBench's canonical keys are '0'..'n-1'. The UNAMENDED smoke gate REFUSED the first attempt at check score-keys-are-indices, before any sealed item was sent, and the server was torn down with the card verified free. The 1-based convention - which their own schemas.py documents only as 'Probability distribution across each ordered rubric level', without stating the convention - was then established on the AUTHOR'S OWN SERVER with SYNTHETIC requests across every rubric size their own validator allows (2..10): exactly the keys '1'..'n' in order every time, never '0' and never a key above n. It is corroborated independently by their own `score` field being a 1-based probability-weighted mean of the same distribution (1.0185 on a 2-level rubric, 1.6147 on a 10-level one, 3.1155 where the mass sits on key 3; a 0-based convention would have given 0.0185), and a reversed rubric moved the mass to a different key, so 'k' means the k-th level of the criteria array AS SENT - which is what JevBench's 'k-1' means. classone_reindex_r58.ClassOneReindexAdapter therefore relabels score keys k -> k-1 and NOTHING else: the request body, the answer mapping, the noul range check and the choice-must-be-one-of-the-item's-labels rule are the UNCHANGED vendored typesafe adapter's, reused by calling its own run(). An item whose score key set is not exactly {'1'..'n'} is REFUSED and recorded invalid rather than rescored on a guess; none was. choice and noul are untouched. The gate then ran AMENDED in exactly that one check, requiring the key set to be exactly {'1'..'n'} - stricter than the original, because it also pins the count - with the other seven byte-identical, and passed. PREREG-R58-AMENDMENT-1.md (sha256 in PREREG-R58-AMENDMENT-1.md.sha256), receipt PROBE-classone-e2b.json. No file of the author's was edited and no prompt of ours exists. THE SUBMIT FORM CARRIED NO DESCRIPTION FOR EITHER CLASSONE ROW, so everything about how it is run comes from the author's own repository: install line README Quickstart step 1 (`pip install -e ".[dev]"`, in an isolated venv - torch 2.14.1+cu130, transformers 5.19.0, fastapi 0.142.2), serve line step 3 (`uvicorn classone.server.app:app`), and the checkpoint selected the way their own src/classone/server/app.py reads it (CLASSONE_BASE_MODEL). THEIR SERVER REGISTERS POST /v1/systemone ITSELF (app.py:115, next to /v1/decide and /v1/classone) and their DecisionRequest/DecisionResponse in src/classone/schemas.py are the TypeSafe System One shapes, so no transport shim exists anywhere in the path. One RTX 5090 32 GB (Lium, Paris FR). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. output_tokens is 0 by their own schema comment. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. add-requests run 58, benchmarkheaven.com/submit ref 957CEE0B (bh-submit b45cdc66-df44-4d77-a616-e7c67c83706d) · Published source or weights
• open-alternative-jev (Qwen3.5-4B, IkerMoel): Apache-2.0 (code and weights) · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• OpenSourceJev (Qwen3.5-4B Q4_K_M, native llama.cpp): MIT (repository code); Apache-2.0 (Qwen/Qwen3.5-4B base and unsloth/Qwen3.5-4B-GGUF Q4_K_M conversion) · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• EXAONE-4.0-1.2B-JEV v0.3 (unofficial fine-tune of LGAI-EXAONE/EXAONE-4.0-1.2B, native option-label softmax, two reads averaged, self-hosted): EXAONE AI Model License 1.2 (non-commercial) · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), 1,477 of 1,500 answered. NEW REVISION of the row run 48 measured: weights carrtesy/EXAONE-4.0-1.2B-JEV at TAG v0.3, server github.com/carrtesy/EXAONE-JEV at tag v0.3 (commit recorded in receipts/POD-R58.json), both the artefacts the author's own submit-form entry names. EXAONE AI Model License 1.2 - NC, NON-COMMERCIAL, and an unofficial personal project - the author states it is 'not an LG AI Research release'; both belong on the row. NO MAPPING DECISION OF OURS: their serve/systemone_server.py speaks the TypeSafe Jev POST /v1/systemone contract, so the UNCHANGED jevbench.adapters.typesafe emits exactly what it parses, and all eight checks of the unamended smoke gate passed on synthetic requests before any sealed item was sent. Read count (permute 2: options in the given and the reversed order, distributions averaged) and the per-type temperatures come from the weights' own jev_config.json, which the author's doc states needs no flags - their shipped default, not a setting of ours. STACK: the author's own docs/REPRODUCE.md install line in an isolated venv, vllm==0.30.0 letting it resolve its own stack (torch 2.13.0+cu130, transformers 5.19.0), served with their own published command verbatim (`python serve/systemone_server.py --model weights --port 8000 --gpu-util 0.9`) and VLLM_USE_FLASHINFER_SAMPLER=0 as their doc sets it. One RTX 5090 32 GB (Lium, Paris FR). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM, at the limit the author DECLARES: their docs/REPRODUCE.md section 3 states a 65,536-token prompt limit answering 400, and all 23 came back as 'HTTP 400: prompt of 84996 tokens is longer than the maximum context length 65536'. The 23 invalid are therefore the system's own disclosed refusal, counted wrong, and no other item failed. Serial, one decision per request, no retry, no batching, offline, credential-free. add-requests run 58, benchmarkheaven.com/submit ref CA53D793 (bh-submit 5631d258-1eaa-4e2d-a658-f6417b85b351); previously GitHub issue #176 · Published source or weights
• Malkuth-2B (newfull5, Kev post-train): CC-BY-NC-4.0, research use only (XNLI and RACE in the training mix) · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• Nemotron Diffusion 8B (pst2154, optimized vLLM): MIT (application code); modified vLLM runtime Apache-2.0; nvidia/Nemotron-Labs-Diffusion-8B keeps its upstream licence · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• decider-2b (Mapika): Apache-2.0 · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• kev 4B (research preview): Apache-2.0 · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• lev-350m (Franck Verrot, LFM2.5-350M): Apache-2.0 (code); weights under LiquidAI's LFM1.0 licence, following the LFM2.5-350M base · evaluator-owned Lium GPU pod (RTX6000), offline read-only container · Published source or weights
• Certo v1 (AltSlate Labs): MIT · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• smalljev semantic-v9: Apache-2.0 · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• jul fast (usejul/jul-decision-minicpm5-2b: MiniCPM5-2B fine-tune with a pointer head, option probabilities from one pass, self-hosted via `jul serve`): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (sha256 901983ae...), 1,498 of 1,500 answered. Library github.com/usejul/jul tag v0.5.1, PyPI jul==0.5.1, Apache-2.0; weights usejul/jul-decision-minicpm5-2b at f556a6936089693ea68bfed3080c637e4853a3ff, Apache-2.0; base openbmb/MiniCPM5-2B at 12a3808a. NO MAPPING DECISION OF OURS: `jul serve` speaks the TypeSafe /v1/systemone wire format, so the UNCHANGED jevbench.adapters.typesafe applies, and all eight checks of the unamended smoke gate passed on synthetic requests before any sealed item was sent. STACK: the author's own published install line from their request, `pip install "jul[torch]==0.5.1"` in an isolated venv (torch 2.14.1+cu130, transformers 5.19.0), served with their own published command verbatim (`jul serve --model jul-decision-minicpm5-2b --host 127.0.0.1 --port 8577`). One RTX 5090 32 GB (Lium, Paris FR). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW IS THE FIRST THAT NEITHER REFUSES NOR UNIFORMLY TRUNCATES THEM, AND THE SPLIT IS BY QUESTION TYPE. The author declares a 2,048-token state budget with no refusal - 'the beginning of the text is kept and the end is cut, without an error'. Their server's own usage.input_tokens shows that budget applied to 17 of the 23 (counts 2,096-2,135) and NOT to the other 6, which were read at 107,331-107,997 tokens; those 6 are exactly the 6 score questions among the 23, while all 17 truncated ones are choice or noul. Two of the six full reads then failed with the server's own CUDA OutOfMemoryError on a 32 GB card ('Tried to allocate 5.59 GiB' / '5.41 GiB'), which is this row's only 2 invalid and which a larger card would most likely have answered: the cause is the score path not applying the author's own documented truncation, and the author's note that '24 GB is enough' holds only under that truncation. The 2 invalid are counted wrong, the conservative direction, and are 0.13 % of the pool. PROBABILITIES ARE RETURNED ROUNDED TO FOUR DECIMALS: 426 of 1,498 answered distributions sum to 1 +/- 2e-4 (observed range 0.9998-1.0002); none is off by more than that, so it is the server's rounding and not a partial readout. Serial, one decision per request, no retry, no batching, credential-free. add-requests run 58, GitHub fstandhartinger/jevbench#198 · Published source or weights
• watt-flash-0.1 (Zaitgeist Labs, 140M mmBERT-small encoder with an option-marker scorer, one forward pass per request, self-hosted): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights zaitlabs/watt-flash-0.1 @ d49a3a880b0201b325f2d2299d4c36ffc8735477 (Apache-2.0, public and ungated), model.safetensors sha256 099e611bd27c9542732b89be47bcace54964c00d9c7161ebccb061648c9780bf = THE SUBMITTER'S OWN DECLARED HASH, loaded from a local snapshot with the author's own --model flag (no network during the run). Code github.com/zaitgeist-charly/zaitlabs-watt-flash-0.1 @ e9863f06230ad874a5a2cfca3af922e45161a4d7 (Apache-2.0, = HEAD of main), installed with the author's own line `uv sync` against their own uv.lock in an isolated venv (torch 2.14.0, transformers 5.17.0, safetensors 0.8.0) and served with their own documented command `python -m watt_flash.serve --port 8942`. No file of theirs was edited. Their server answers POST /v1/systemone with Jev's shapes (noul -> {noul: P(true)}; choice probabilities keyed by the criteria keys with choice = argmax; score probabilities keyed '0'..'n-1'), so the UNCHANGED typesafe adapter applies and no mapping is ours; all eight checks of the unchanged smoke gate passed on synthetic requests before any sealed item was sent. THE AUTHOR'S FREE HOSTED ENDPOINT api.inference.zaitlabs.com WAS NEVER USED - the submission states it may train on the data, so no sealed item reached it. THE LONG ITEMS ARE REFUSED, NOT TRUNCATED, by the author's own code: watt_flash/layout.py raises Overflow past its documented 8,192-token window and serve.py answers that with HTTP 422, which their own commit e9863f06 says is deliberate ('answer an over-long request with 422, which JevBench scores as a refusal, not an outage'); those items are counted wrong and no other item failed. A RESUBMISSION of GitHub #113 via the form, which the author asks to dedupe with it. RUN ON SANDY'S SHARED CPU (12-core Zen 2, AVX2, fp32 - the package itself calls model.float() off-accelerator, 8 threads, nice), server and driver inside a loopback-only network namespace (unshare -rn) with no credential in the environment; load average logged every 60 s, so the Speed axis is a DECLARED LOWER BOUND. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. Serial, one decision per request, no retry, no batching, offline. add-requests run 61, benchmarkheaven.com/submit ref 0874B655 (bh-submit 4cf0bdfb-b6d7-4777-a8ac-350d29c5ac3f) · Published source or weights
• Decision Fast (FlyMy.AI, v53a): Apache-2.0 notices on the included code and the pinned base; the weights are an evaluation preview under EVALUATION-PERMISSION.md, not a cleared commercial release · evaluator-owned Lium GPU pod (RTX6000), offline read-only container · Published source or weights
• openJev Verdict 1.4: Apache-2.0 · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• Quyet-1.0-Small-EN (Chinh Nguyen, ModernBERT-base encoder with fixed heads): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order 3643732b) · Published source or weights
• verdict-small (Manavarya09, multilingual-e5-small 118M): Apache-2.0 (verdictml code and the Manav2op/verdict-small checkpoint over intfloat/multilingual-e5-small) · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• kev 0.6B (research preview): Apache-2.0 · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• WaterSheep (Samrat Dutta, ModernBERT-base encoder with calibrated heads, served by the author's own `watersheep --serve`): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (sha256 901983ae...), 1,500 of 1,500 answered, 0 invalid. RE-MEASUREMENT on v1.6 of the row run 48 measured on the retired v1.5 basis, same weights: samratduttaofficial/WaterSheep @ ebf3705d2d9dc648d5cc6b4cbca7646c6ab48998 (Apache-2.0), package github.com/SamratDuttaOfficial/WaterSheep @ 3ccf02b3 via the author's own install line in an isolated venv (torch 2.14.1+cpu, transformers 5.19.0), served with the author's own command `watersheep --model samratduttaofficial/WaterSheep@ebf3705d... --serve --device cpu`. Their server speaks POST /v1/systemone with Jev's shapes, so the UNCHANGED typesafe adapter applies; all eight checks of the unchanged smoke gate passed on synthetic requests before any sealed item was sent. Their own max_len 512 (watersheep.json) truncates every longer state, the 23 long items included, without refusing; not overridden. RUN ON SANDY'S SHARED CPU (12-core Zen 2, 8 threads) with server and driver inside a loopback-only network namespace and no credential in the environment; load average 8.6-15.4 (median 13.8) logged every 60 s, so the Speed axis is a DECLARED LOWER BOUND. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. Serial, one decision per request, no retry, no batching, offline. add-requests run 59, benchmarkheaven.com/submit ref 6F1561FC; previously GitHub #179 · Published source or weights
• kev 0.5B: Apache-2.0 · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• Quyet-1.0-Small (Chinh Nguyen, SEA-LION-ModernBERT-300M encoder with fixed heads): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order 3643732b) · Published source or weights
• Quyet-1.0-Tiny (Chinh Nguyen, mmBERT-small encoder (16 layers kept) with fixed heads): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, weights fetched in a separate model-only container, scored run inside docker --network none (loopback only, no secrets, label-free inputs); pod destroyed afterwards; paid public fast-lane evaluation, 4 Oct 2026 (order 3643732b) · Published source or weights
• Open Jev JSON Canvas (JoshuaSP): MIT code; Apache-2.0 DiffusionGemma weights · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• openJev Verdict (heman10x, ModernBERT-base 151M): Apache-2.0 · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• Tacet Sonata (CodePawl, 144M packed encoder on mmBERT-small, option-marker softmax in one pass, in-process via the author's tacet package): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), 1,500 of 1,500 answered, 0 invalid. Weights codepawl/tacet-sonata @ 26f447c35881a80020fe557c95ceb8c61bc3fd12, Apache-2.0, public and ungated, loaded from a local snapshot (no network during the run). Runtime tacet==0.3.0 from PyPI, the author's own install line (`pip install "tacet>=0.3"`) in an isolated venv (torch 2.14.1+cpu, transformers 5.19.0). ADAPTER: the author's own PR fstandhartinger/jevbench#197 file tacet_local.py at head 76eb75da, byte-identical (sha256 c63a4fe0...), reviewed before running: it maps answers only (noul P(true) -> yes/no; choice and score probabilities used as returned, score keys '0'..'n-1') and reaches nothing but the Hub. It ran through the UNCHANGED v1.6 driver run_v16.py; the only addition, in a wrapper, registers the file as jevbench.adapters.tacet_local and lists it in IN_PROCESS_MODULES as laya_local is listed - lane verification and the runtime egress guard unchanged. Synthetic smoke gate passed before any sealed item was sent (PREREG-R59.md, receipts/SMOKE-tacet-sonata.json). THE 23 LONG ITEMS ARE TRUNCATED, NOT REFUSED, by the package's own code at its default and documented max_length 4,096 (tacet/packing.py keeps the questions and cuts the state; it refuses only states over 200,000 characters, which no item here reaches): exactly 23 records report input_tokens = 4,096 and they are answered and scored. RUN ON SANDY'S SHARED CPU (12-core Zen 2, AVX2, fp32 as the package runs off-accelerator, 8 threads), load average 6.6-24.2 (median ~14) logged every 60 s: the Speed axis is a DECLARED LOWER BOUND, and the author reports raw p50 0.042 s on an RTX 3060 against this run's 0.265 s. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. Serial, one decision per call, no retry, no batching, offline, credential-free. add-requests run 59, benchmarkheaven.com/submit ref 716D1957 (bh-submit 07d384eb-fc4e-4536-ba61-888567efdf34) · Published source or weights
• CLM-8B (Contrastive-LM, clm-latest): Apache-2.0 (code and CLM-v0.1-8B head); Qwen3-8B encoder Apache-2.0 · evaluator-owned Lium GPU pod (RTXPRO6000), offline read-only container · Published source or weights
• Deem 0.8B v1: Not independently recorded · evaluator-owned H100 GPU · Published source or weights
• Laya multilingual: apache-2.0, not gated · evaluator-owned CPU, Sandy (AMD Ryzen 5 3600), 4 of 12 threads, nice -n 5, shared host, offline (HF_HUB_OFFLINE=1) · Published source or weights
• ClassOne Gemma 4 E2B (Gemma-4-E2B backbone with ClassOne decision heads, schema-constrained single forward pass, self-hosted): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (sha256 901983ae...), 1,477 of 1,500 answered. Weights devops-thiago/classone-gemma4-e2b (sha cd502aa33d288e67f16937c2e125f2297ab7b4ad), Apache-2.0, public and ungated; code github.com/devops-thiago/class-one, Apache-2.0. ONE DOCUMENTED MAPPING OF OURS, AND IT IS THE ONLY ONE. The author's server keys SCORE probabilities '1'..'n' while JevBench's canonical keys are '0'..'n-1'. The UNAMENDED smoke gate REFUSED the first attempt at check score-keys-are-indices, before any sealed item was sent, and the server was torn down with the card verified free. The 1-based convention - which their own schemas.py documents only as 'Probability distribution across each ordered rubric level', without stating the convention - was then established on the AUTHOR'S OWN SERVER with SYNTHETIC requests across every rubric size their own validator allows (2..10): exactly the keys '1'..'n' in order every time, never '0' and never a key above n. It is corroborated independently by their own `score` field being a 1-based probability-weighted mean of the same distribution (1.0185 on a 2-level rubric, 1.6147 on a 10-level one, 3.1155 where the mass sits on key 3; a 0-based convention would have given 0.0185), and a reversed rubric moved the mass to a different key, so 'k' means the k-th level of the criteria array AS SENT - which is what JevBench's 'k-1' means. classone_reindex_r58.ClassOneReindexAdapter therefore relabels score keys k -> k-1 and NOTHING else: the request body, the answer mapping, the noul range check and the choice-must-be-one-of-the-item's-labels rule are the UNCHANGED vendored typesafe adapter's, reused by calling its own run(). An item whose score key set is not exactly {'1'..'n'} is REFUSED and recorded invalid rather than rescored on a guess; none was. choice and noul are untouched. The gate then ran AMENDED in exactly that one check, requiring the key set to be exactly {'1'..'n'} - stricter than the original, because it also pins the count - with the other seven byte-identical, and passed. PREREG-R58-AMENDMENT-1.md (sha256 in PREREG-R58-AMENDMENT-1.md.sha256), receipt PROBE-classone-e2b.json. No file of the author's was edited and no prompt of ours exists. THE SUBMIT FORM CARRIED NO DESCRIPTION FOR EITHER CLASSONE ROW, so everything about how it is run comes from the author's own repository: install line README Quickstart step 1 (`pip install -e ".[dev]"`, in an isolated venv - torch 2.14.1+cu130, transformers 5.19.0, fastapi 0.142.2), serve line step 3 (`uvicorn classone.server.app:app`), and the checkpoint selected the way their own src/classone/server/app.py reads it (CLASSONE_BASE_MODEL). THEIR SERVER REGISTERS POST /v1/systemone ITSELF (app.py:115, next to /v1/decide and /v1/classone) and their DecisionRequest/DecisionResponse in src/classone/schemas.py are the TypeSafe System One shapes, so no transport shim exists anywhere in the path. One RTX 5090 32 GB (Lium, Paris FR). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. output_tokens is 0 by their own schema comment ('In System 1 models this is 0 because there is no autoregressive decode'). THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM at a limit far below them: all 23 came back as 'HTTP 400: Prompt length (81338 tokens) exceeds maximum limit (8192)', an 8,192-token ceiling the author's code enforces with an explicit error rather than truncating. The 23 invalid are the system's own refusal, counted wrong, and no other item failed. Serial, one decision per request, no retry, no batching, credential-free. add-requests run 58, benchmarkheaven.com/submit ref C4D7AACD (bh-submit 4567a065-e912-40df-9bc1-68fb37cc7bba) · Published source or weights
• APUS-OpenJev-v1-35B-A3B (merged bf16 checkpoint-5949 MoE on Qwen3.5-35B-A3B, their own native runtime, full depth, self-hosted): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights apus-ailab/APUS-OpenJev-v1-35B-A3B@090388abe2237117cc3b4df6f4250e81cbb36864, Apache-2.0, 66.99 GiB bf16 merged weights, checkpoint-5949, Qwen3_5MoeForConditionalGeneration (40 layers, 256 experts, 8 active), declared base Qwen/Qwen3.5-35B-A3B, full depth 40 layers (their depth_config declares exit_depth 20 / full_depth 40). accelerate>=1.10 additionally, per this repo's own requirements.txt. STACK: the authors' own requirements.txt, unmodified - torch==2.8.0 and transformers==5.16.1. The transformers pin is THEIRS, not a workaround: OpenJet.__init__ raises 'Layer execution is audited for transformers==5.16.1' on any other version. Served behind a TRANSPORT SHIM ONLY (r56/serve/apus_systemone_r56.py): loaded with THEIR OpenJet.from_pretrained(device=cuda:0, dtype=bfloat16) and every request handed to THEIR decide(), returning THEIR format_response() output. No contributor file edited. effort='high', the full-depth path, their decide() default and the configuration their own headline Frozen80 number is quoted on; effort='low' is not measured and no routing between the two is invented. One H100 PCIe 80 GB (Lium). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. MAPPING (run 56's, not the authors'; pre-registered in PREREG-R56 s2.2 before any item was sent and repeated here because it is part of the number): their runtime accepts primitive in {choice, noul, score_level} with 2-16 candidates labelled A-P. choice -> their 'choice' primitive, candidates = the item's own labels as id and its own criteria[label] as description, in labels order; their format_response returns probabilities keyed by candidate id, i.e. by the JevBench label already, so nothing is rewritten. All 750 choice items carry 3-9 options, inside their 2-16 window. noul -> their 'noul' primitive; their contract REQUIRES the canonical yes/no criteria there, so the item's own true/false description strings are NOT delivered - the proposition itself travels verbatim in instructions, which is what their noul is built to judge. score -> their 'choice' primitive over the ordinal levels, because their own RUNTIME.md rules score_level out as an ordinal Score API ('score_level is the judgement of a single proposition, it cannot be treated as a complete ordinal Score API'); the 4-5 ordered levels become candidates with id = the level index and description = the item's own level text, which carries the order. Recording all 375 score items as misses instead would understate a system whose published interface answers them. CALIBRATION IS MEASURED AS THE AUTHORS SHIP IT AND THEY DISCLOSED IT IN ADVANCE: their response carries calibrated: False and both their card and RUNTIME.md state the probabilities are an uncalibrated relative softmax over the offered candidates. OUTPUT TOKENS ARE 0 BY CONSTRUCTION, NOT BY ESTIMATE: decide() runs one forward pass with logits_to_keep=1 and reads candidate logits; nothing is generated anywhere in that path. Input tokens are their own prompt_tokens field. THE 23 LONG ITEMS: their compile() raises ValueError('input exceeds runtime limit; no truncation permitted') above 8,192 tokens and their RUNTIME.md states it in advance. Recorded invalid. This is the first row in the series whose refusal on those items is guaranteed in the author's own code with an explicit no-truncation promise. add-requests run 56, GitHub #105, benchmarkheaven.com/submit ref AE709BBB · Published source or weights
• Bespoke Nimble 9B (Bespoke Labs): Apache-2.0 (weights); repository without a licence file as of 19 Sep · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• Bev / Bonsai 27B: Not independently recorded · evaluator-owned H100 GPU · Published source or weights
• Clef (Cloudflare, Qwen3.8-27B post-train with a joint schema head, multimodal, measured on text): apache-2.0 · evaluator-owned Lium GPU pod (H100 80GB), offline (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE), credential-free, server bound to 127.0.0.1, harness on the same pod over loopback HTTP; Lium pod 08a1ff4a-248a-4ff2-948e-ea558f99e808 (huid cosmic-hawk-4f, node calm-fox-65), 1x NVIDIA H100 80GB HBM3, United States, USD 1.30/h · Published source or weights
• decider-35b-a3b (Mapika): Apache-2.0 · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• Decision 2.0 Vega 27B (vLLM-SR, Qwen3.8-27B): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, a thin transport shim around the author's engine in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• JADE (Qwen3.8-27B LoRA): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, a thin transport shim around the author's engine in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• Jebadiah 27B (Frontier Infra, Qwen3.8-27B LoRA): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, the author's own server in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• JEV-27B (AutoTrust, Qwen3.8-27B): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, a thin transport shim around the author's engine in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• JevOne: Component-specific JevOne/Qwen/SGLang terms recorded in RESULT.md · evaluator-owned Lium GPU pod (RTXPRO6000), offline read-only container · Published source or weights
• JPT-35B-A3B (Qwen3.5-35B-A3B fine-tune): cc-by-nc-4.0 · evaluator-owned Lium GPU pod, 1x NVIDIA RTX PRO 6000 Blackwell 96GB, the author's own server in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• Kev 27B (Kev 1.0, Qwen3.8-27B): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, the author's own server in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• kev 8B (research preview): Apache-2.0 · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• Laya (Convai Innovations, ModernBERT-large 421M): Apache-2.0 · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• Laya typed-decisions: apache-2.0, not gated · evaluator-owned CPU, Sandy (AMD Ryzen 5 3600), 4 of 12 threads, nice -n 5, shared host, offline (HF_HUB_OFFLINE=1) · Published source or weights
• LitJev (Qwen3.8-27B): Apache-2.0 (code); Apache-2.0 base weights · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• Mirror: Apache-2.0 wrapper and Mirror release; upstream DeBERTa/model assets retain their own terms · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• Open-Jev 27B v1.1 (Zefan Cai, LoRA + decision head on Qwen3.8-27B): Apache-2.0 · Self-hosted author loader, Transformers reference path on RTX PRO 6000 Blackwell; speed is a LOWER BOUND: flash-linear-attention and causal-conv1d kernels were not installed. The author documents the kernel path as answer-identical. Each candidate is encoded independently, producing high input-token usage. · Published source or weights
• Open-Jev 2B (Zefan Cai): MIT (loader); Apache-2.0 (adapter and pinned Qwen base); CC0-1.0 public training projection · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• Open-Jev 9B (Zefan Cai): MIT (loader); Apache-2.0 (adapter and pinned Qwen base); CC0-1.0 public training projection · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• open-jev-deberta-v3-large (local CPU): Apache-2.0 (model card); DeBERTa-v3 keeps its own terms · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• OpenJev (thinking, BF16): Apache-2.0 · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• Perplexity Decider v1.1 27B (Qwen3.8-27B, noncausal decision readout): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, the author's own server in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• Qwen3.5-0.8B Decision Model (Mourad Ghafiri): MIT code and training data; Apache-2.0 model weights · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• Qwen3.5-9B Jev-like data-mix v2: Apache-2.0 (adapter code and weights; Qwen3.5-9B base is Apache-2.0) · evaluator-owned Lium GPU pod (A6000), offline read-only container · Published source or weights
• reflex 4B (kshetrajna12): MIT (code, adapter); Apache-2.0 (base) · evaluator-owned Lium GPU pod (H100), offline read-only container · Published source or weights
• Seb-9B (Qwen3.5-9B-shaped multimodal decision model, one forward pass, stock vLLM, self-hosted): apache-2.0 · measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights ironbcc/seb-9b@ac57b36bbe978762032d0ad7197778b4bc4a0209, Apache-2.0, 18.0 GiB bf16. Served with the AUTHOR'S OWN PUBLISHED vLLM COMMAND, verbatim from their model card, on vLLM 0.30 - the version their card states it is tested with - at the 8,192 --max-model-len their own flag note names for long documents (the larger of the two values they give). r56/serve/seb_systemone_r56.py supplies TRANSPORT ONLY: it builds their own documented chat request per decision (their system string verbatim, max_tokens 1, temperature 0, logprobs true, top_logprobs = the option count, structured_outputs.choice = the option keys, enable_thinking false) inside their own user-message envelope {task,state,question,type,options}, and reads the probabilities exactly where their card says they are - exp of choices[0].logprobs.content[0].top_logprobs - renormalised over the option keys. Their own key convention is used unchanged (noul Y/N, choice A..T, score 0..4) and every criterion description travels verbatim; no prompt wording of theirs is altered and no file of theirs is edited. THE 23 LONG ITEMS are rejected server-side by vLLM because the author's own published command sets --max-model-len 8192; recorded invalid and recorded as a CONFIGURATION limit under their own published flags, NOT as a model refusal - their config declares 262,144 positions and raising the flag would be our change, not theirs. One H100 PCIe 80 GB (Lium). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. add-requests run 56, benchmarkheaven.com/submit ref C566D021 · Published source or weights
• SimpleJev (Qwen3.5-0.8B, CPU): Apache-2.0 server; Apache-2.0 Qwen3.5-0.8B checkpoint · evaluator-owned CPU container, offline read-only, pinned cores · Published source or weights
• SimpleJev Qwen3.8-27B (self-hosted, v1.6 pool): apache-2.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, the author's own server in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights
• Standard One 8B (Standard Thinking): Apache-2.0 (jev-adapter server, LoRA and merged weights; base Ministral 3 8B Apache-2.0) · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• swanOne (blockbrain, Qwen3.8-Flash-Next NVFP4): patches Apache-2.0, shim MIT; weights under the Qwen licence (LICENSE-NOTICE.md) · evaluator-owned Lium GPU pod (RTXPRO6000), offline read-only container · Published source or weights
• system-one (Qwen3-8B, Sean Goedecke): no licence file in the repository as of 19 Sep; Qwen3 weights Apache-2.0 · evaluator-owned Lium GPU pod (RTX5090), offline read-only container · Published source or weights
• torchcast-decision-27b (Torchcast AI, Qwen3.8-27B fine-tune): cc-by-nc-4.0 · evaluator-owned Lium GPU pod, 1x NVIDIA H100 80GB, the author's own server in its own pinned environment, offline (Hub offline mode, all credentials unset), driven over loopback only by the JevBench client with its loopback guard; label-free inputs; pod destroyed afterwards; HF Jev Decision Index intake, 6-7 Oct 2026 · Published source or weights

Frequently asked questions

What does JevBench compare?

JevBench v1.6.1 measures Jev-class decision models across Intelligence, Calibration, Speed and Cost. Capability is the mean of Intelligence and Calibration within the official cost and median-latency caps; Composite is secondary.

Which open-weight Jev alternative scores highest?

Quyet-1.0-Large (Chinh Nguyen, Gemma-4-31B decoder, option-letter logits) has the highest eligible Capability Score among open-weight Jev-class alternatives: 81.7, rank #1 on the JevBench open-weights board.

What counts as an open-weight alternative?

The published row must have public code or weights, a licence and a source link. Unknown evidence remains unclassified. Terms can differ between code, adapters and weights.

Are prices and latency directly measured?

Each row labels measured, estimated or self-reported announced pricing. Latency comes from the published benchmark conditions and may include a disclosed adjustment. Neither is a guarantee for a different workload.