About & data sources

Benchmark Heaven brings together pricing and benchmark data for open-source and frontier LLMs into one comparable view. Prices are normalized to USD per 1M tokens (input and output) unless a platform prices differently (GitHub Copilot's current token/AI-Credit rates and legacy request billing are shown on a separate product axis).

Open source & hobby project

Benchmark Heaven is open source under the MIT licence and a hobby project — built to give the world better tools to decide which LLM to choose for the job. The code lives at github.com/fstandhartinger/model-market-comparison. The licence covers this code only: the benchmark results, prices and other data collected here are third-party data under their own terms (see Sources below — Artificial Analysis requires attribution, Epoch AI is CC BY, DesignArena is shown under a documented risk decision), and nothing collected here is relicensed by the MIT licence. The site is still in beta: data and features change daily.

Data sources & removal on request

Benchmark Heaven is a free, open-source, non-commercial hobby project. Benchmark results are shown with attribution and links to their original publishers (Artificial Analysis, Epoch AI, DesignArena, OpenRouter, …). If you publish benchmark data and would like your numbers removed or shown differently, email info@productivity-boost.com and we will act promptly.

Support this project

If Benchmark Heaven helps you choose a model, you can Support this project. Payments go to productivity-boost.com Betriebs UG (haftungsbeschränkt) & Co. KG, the one-person company behind these projects.

For agents

Two ways in. Headless or server-side: the public HTTP API — no key, CORS open, documented in API.md. In a browser tab: in browsers with WebMCP (navigator.modelContext), every page of this site registers three read-only tools — search_benchmarks, get_benchmark_results and get_model_benchmark_summary. They work only while a benchmarkheaven.com tab is open and the browser supports WebMCP; they are not a remote MCP server. They return published results only: the value in the benchmark's own unit, its basis (measured, self-reported, derived, preliminary), benchmark version and the source with its retrieval date. A missing result is null, never an estimate. At most 25 benchmarks, 50 results or 10 models per call, with cursors for more. The tools send no cookies, store nothing and cannot change anything; other sites cannot use them from an embedded frame.

Sources

• OpenRouter — model catalog and per-provider endpoint pricing (live API).
• ArtificialAnalysis — Intelligence & Coding indices plus sub-benchmarks (LiveCodeBench, SciCode, Terminal-Bench Hard, τ²-Bench, GPQA, MMLU-Pro) via the v2 API.
• AA Coding Agent Index — Composite retains v1.4, last collected 2026-09-09. AA now publishes v1.5 with different benchmark components. We collect it separately and do not mix the versions.
• Some AA licensing and model identifiers are retained from earlier source publications. Their original dates are recorded per field in the downloadable dataset.
• DesignArena (Arcada Labs) — agentic Web Apps & Full-Stack Elo leaderboards, read from the site's own public leaderboard endpoint. DesignArena publishes no API documentation or data licence, so this is not an official feed and we do not present it as one. We show these two boards and collect nothing further from this source; that scope is recorded in the repository and would only change with the source's documented permission.
• LisanBench — word-chain benchmark by Lisan al Gaib (@scaling01), read from the files lisanbench.com itself loads. Its repository publishes usage terms rather than an open-source licence: results are shared with credit to the creator and a link to the repository.
• Benchmark aggregators (Lumina Bench, BenchLM, LLM Stats, Vellum, CodeSOTA and others) — used only to discover benchmarks and to check where a published result came from. No number on this site is taken from these aggregators: each comes from the organisation that ran the benchmark, or from the model's developer and is then labelled self-reported, and one result is never counted twice through two routes.
• AWS Bedrock — on-demand token pricing, European regions (eu-central-1 where available).
• Azure AI Foundry — retail token meters plus model-card serving-region checks. A billing/resource region alone is not treated as proof that inference stays in the EU. By company policy, only Azure Direct Global DeepSeek V4 Pro and Kimi K2.7 Code are additionally eligible as EU-hosted equivalents; they remain marked Global because inference may occur outside the EU.
• Google Vertex AI — pay-as-you-go token pricing for Gemini and Model Garden partner models (Claude, Llama, Mistral, DeepSeek, Qwen); offers are marked EU-hosted only where the model supports a documented European serving location.
• TensorX, Inceptron, Scaleway, IONOS, Mistral, Nebius, OVHcloud, STACKIT & T-Systems — direct European serverless/managed catalogs. TensorX is the broadest in-EU host for GLM/Kimi/DeepSeek/MiniMax; Inceptron serves GLM/Kimi/MiniMax from Finland; Scaleway and OVHcloud serve from France; STACKIT and T-Systems publish German/EU catalogs. Nebius is EU-capable but mixes EU, US and UK serving regions, so every model offer is checked separately. See the EU & Sovereign tab.
• GitHub Copilot — current 32-model AI-Credit/token-price catalog plus the 18-model legacy premium-request multiplier table. The UI's separate per-request field applies only to eligible legacy annual Pro/Pro+ plans.
• Anthropic / Claude Code — all 14 currently callable first-party API models, their cache/batch/list prices, active promotions and Claude Code Enterprise terms.

What are “Featured” models?

★ Featured is the shortlist the recommendation views start from. It is not hand-picked: it is the top 20 model families of the Artificial Analysis Intelligence Index, ranked by the best score any of their reasoning-effort variants reaches, with models their vendor has deprecated left out. Because the set is derived from the chart on every data refresh, it follows new releases by itself instead of waiting for someone to edit a list.

1. 1.Claude Opus 5.557.6
2. 2.Claude Sonnet 5.556.0
3. 3.Muse Wonder Full55.8
4. 4.Claude Fable 5.153.4
5. 5.GPT 6 Astra52.7
6. 6.Gemini 4 Argon52.6
7. 7.Muse Wonder52.4
8. 8.GPT 6.1 Sol51.8
9. 9.Muse Spark 1.348.1
10. 10.Grok 4.746.4
11. 11.MiMo V2.6 Pro46.3
12. 12.Qwen3.8 Max 090245.4
13. 13.GLM 5.344.8
14. 14.Step 5 Preview43.7
15. 15.Kimi K343.6
16. 16.Claude Haiku 5.543.4
17. 17.GPT 5.6 Terra42.1
18. 18.GLM 5.3 Flash41.8
19. 19.Ling 3.1 Flash41.1
20. 20.Gemini 3.8 Flash40.9
21. 23.DeepSeek V4.1 Flash39.5pinned

Turn the “Featured” filter off on any page to explore all 897 tracked models. Featured status belongs to the family, so every reasoning variant of a featured family is included. One family can be pinned into the set on request — today that is DeepSeek V4.1 Flash — and pinned entries are labelled above. Ranking by index means a family can leave the list when a newer model outranks it; the ranks shown here are the ones in the current dataset, dated 2026-10-14.

Snapshot

How we calculate

Two numbers on this site are ours rather than a source's: the adjusted cost of a task and the composite score of a model. Everything else is reproduced as published. This section explains both in full, so any figure on the site can be traced back to its inputs.

Adjusted cost per task

A price per million tokens does not tell you what a model costs you, because two models need very different numbers of tokens for the same job, and because what you actually pay depends on which provider serves you and how much of your input that provider can read from cache. The adjusted cost folds all of that into one number — modelled USD for one task:

• Which provider. Of all the routes that survive your filters, we cost the cheapest one, using that exact endpoint's published prices — not a platform average.
• Its cache behaviour. Cached input is billed at that endpoint's cache-read price and uncached input at its normal input price; where the route publishes a cache-write price above its input price (explicit caching, such as Anthropic's 1.25×), the uncached input is assumed to be written to the cache once at that price. The cache-hit rate is the measured one for that exact endpoint where OpenRouter publishes it.
• The model's token efficiency. How many output tokens this specific model variant needs per task, from Artificial Analysis' measurements (Data: Artificial Analysis). A model that reasons at length pays for every one of those tokens.
• One common workload. Input tokens per task = the model's output tokens × one input:output ratio that is the same for every model: the documented global ratio from public LLM usage statistics (Chutes, last 7 days). Two models with the same prices and the same tokens per task therefore cost the same. Under Options → Task workload you can switch toAs used on OpenRouter, which takes each model's own traffic mix instead — that says who uses a model (long agent sessions push the ratio from 3:1 up to 115:1), not what a task costs, and can make a model look several times pricier.

USD/task = [input × (1 − hit) × input price + input × hit × cache-read price + input × (1 − hit) × (write price − input price) + output × output price] ÷ 1,000,000, the write term only where a write price above the input price is published and a cache-hit rate is applied. Where an input is unknown we say so instead of hiding it: missing task-token measurements assume 1,000 output tokens/task; a route that publishes a cache-read price but has no cache-hit measurement of its own — on OpenRouter or direct from a provider — gets the typical rate (the median of OpenRouter endpoints that publish a cache-read price), so a model is credited the same caching share wherever it is sold; a route without a cache-read price gets no cache-hit rate and no discount. Every such assumption is listed on the price itself — click any underlined price to see the exact inputs, sources and dates behind it. Raw list-price mode skips all of this and simply blends list prices at the fixed input:output ratio you choose.

The value map and its green line

The value map plots each model's capability score against its adjusted cost per task; cheaper models sit to the right, so the most attractive models are in the top-right corner. The green line joins the models that no cheaper model beats on capability — the Pareto frontier — with one small tolerance (Florian, 17 Sep 2026): a model stays on the line unless a model at the same price or cheaper is ahead of it by half a point of capability or more, that is 0.5 % of the score scale (on Elo boards, which have no fixed scale, 0.5 % of the plotted range). Without it, two models that are a tenth of a point apart — a difference no benchmark resolves and nobody can see on the chart — would look like a clear win for one of them.

Benchmaxxing signal

We split benchmarks into public “headline” tests that labs quote in launch posts and “held-out” tests whose questions are private, brand-new or newer than the models; indexes built from other boards, judge- or vote-graded boards and legal, finance and medical specialist boards are left out. For every headline/held-out pair in the same topic (for example GPQA Diamond against CritPt in science), the model is ranked among the models that took both tests, and the first part of the score is the average of how much higher it ranks on the headline test, in percentile points. With few pairs that part is pulled toward zero. Tags come in three levels on the published score (one decimal): light from +3.0, medium from +6.0 and very strong from +12.0, and the level follows that score alone. How much evidence stands behind a tag is shown next to it, never used to hide it: a tag built on fewer than ten comparisons, or one whose gap does not stay above zero when its benchmarks are resampled, is marked ◔ uncertain and says which of the two it is.

The second part is how jagged the model looks inside one topic. Two boards that test the same thing should put a model in roughly the same place; when they do not — strong on one coding board, weak on the next, strong on the one after — the shape looks targeted even if its headline and held-out boards happen to average out. For every topic we take the mean distance between the model's ranks on two of its boards, in percentile points, weigh the topics by how many independent comparisons each contributes, and compare the result with the catalog average (13.1 points today). At weight 0.3 a model that is one standard deviation more uneven than the catalog gains about 1.3 points, and an unusually consistent model loses about as much. Being jagged between topics still counts for nothing: that is specialisation.

How to read it: plus means better on famous public tests than on tests nobody can train for, minus the other way round, and zero means no sign. Differences between topics — strong at coding, weaker at maths — are specialisation and do not count. In detail: n is the number of distinct headline boards plus held-out boards in the model's pairs minus one; a score needs n ≥ 6 and pairs in at least two topics; the pull toward zero is n / (n + k) with k estimated from the catalog (how much score variance falls as n grows, clamped to 6–50); the interval resamples the model's headline and held-out boards separately, 400 times, and a tag whose 80 % interval reaches below zero is marked uncertain rather than dropped (Florian, 17 Sep 2026); the jaggedness part shifts that interval with the score, so what the interval quantifies is the gap part's sampling uncertainty. We also simulated a catalog in which nobody targets benchmarks — the same boards, the same coverage, the same capability levels, and every bit of today's unevenness assumed to be noise: the gap alone would still tag 17 % of models there, and the two parts together 20 %, with the extra tags falling on mid-table models, because ranks are bounded and a model in the middle of the field has the most room to spread. That is the price of the second part, and the reason it is weighted at 0.3 rather than 1. It is a screen, not proof: a positive gap fits benchmark-targeted training, but it also fits a model that is simply weaker at long agent work, which several held-out boards lean toward. Model names, labs, openness and prices are never inputs; a tier is decided from benchmark facts alone. See the flagged models and their radars.

The composite score

Epoch ECI and Software Engineering ECI: Epoch AI, ‘Epoch Capabilities Index’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/eci’ [online resource]. Accessed 2026-09-15. Data: Epoch AI (CC BY)

Benchmarks are not on a common scale, so we do not average raw scores. The composite is rank-based: each result becomes a percentile on one common scale, and those are averaged. The scale is the AA Intelligence Index, which covers the widest field of models. The other benchmarks cover different fields — DesignArena and the Coding Agent Index test mostly frontier models — so a plain percentile there would put the same model 20–30 points lower and punish a model for being measured on a selective board. Each of them is therefore linked to the common scale through the models measured on both: a result that ranks at the same position among those models as a given AA Intelligence percentile counts as that percentile. Differences at the top are rank differences, so 99 against 97 is not a 2 % capability gap. The composite uses seven slots: AA Coding, source-matched AA Coding Agent v1.4 (the pinned 9 September snapshot; AA now publishes v1.5, which the Benchmarks table shows), AA Intelligence, Epoch general ECI, Epoch Software Engineering ECI, DesignArena Web Apps (agentic) and DesignArena Full-Stack. The DesignArena inputs are its agentic Web Apps and Full-Stack boards; the page DesignArena calls Frontend is a different, broader board with its own ratings. AA values are clamped to 0–100. A DesignArena board qualifies at an app-selected minimum of 200 battles, aligned with the source's typical preliminary/reliability threshold; its Elo is converted to the expected score against a fixed Elo 1000 opponent. That conversion keeps every model in the same order, so it does not change the composite, which uses ranks only. Percentiles are taken over every model row in the catalogue, deprecated ones included, so hiding deprecated models with a filter does not move anyone's score.

Effort settings are scored separately. The three Artificial Analysis slots (AA Coding, AA Intelligence and the Coding Agent Index) are measured per effort setting, so a result counts only for the configuration that was measured — a non-reasoning setting never borrows its max setting's index. Epoch ECI, Software ECI and DesignArena publish one result per model family, so that result is shared with the family's other settings and marked as attached. It is shared only when the family's published values agree (otherwise only the family representative's value is used), never as a family maximum and never from a retired configuration.

Thin evidence must not become an advantage. Every missing slot inherits that model's own mean observed percentile, producing a base score exactly equal to the mean of its available percentiles. A final dominance-safe projection then prevents missing data from reversing an otherwise unambiguous comparison: when one model covers every reliable slot of another measured model and is no worse in any shared slot, the less-covered model is lowered to 0.1 points below it; the better-measured model is never moved, so adding a thinly measured row cannot change a well-measured score. The unadjusted base and any adjustment are shown separately in model details, and the model page says “dominance-adjusted from” when the change exceeds one point. Models with fewer than three of the seven inputs (exact or attached) are ranked by their score like every other model, but carry a “Thin data” badge with their input count: treat their position as uncertain. A model with no reliable observed slot receives the neutral fallback 50; its zero evidence coverage stays distinct from a measured score and is excluded from capability charts. The #benchmarkscolumn counts the distinct versioned benchmarks a model has a usable result for, which is a broader set than the seven composite slots.

The score takes the Benchmaxxing signal into account, marginally: a model whose headline benchmarks run ahead of its held-out ones loses 1 point per percentile point of that positive signal (w = 1, applied after the steps above); a negative or missing signal earns no bonus. The weight is kept marginal: it is the smallest round value at which the signal changes the #1 spot on the 17 September 2026 data. The Options checkbox “Include Benchmaxxing signal in the score” (on by default) switches it off everywhere the score is used.

Category scores

Category scores are currently unavailable: each category has fewer than two anchors that meet the coverage rule. The individual benchmarks remain in the benchmark table. When a category qualifies, you can pick its score alongside the Main Composite. Each is the weighted average of that category's anchor benchmarks, on a 0–100 scale: a saturated anchor counts at half the weight of the others (see Saturated below). The anchor set is fixed and published here, so two models' category scores always cover the same benchmarks; a model is scored only when it has a result on every anchor, otherwise it has no score for that category rather than an average over an easier subset. Each benchmark counts at its newest published version. A category is only offered when at least two of its benchmarks are on a 0–100-style, higher-is-better scale and are measured for at least 60 % of the featured model families — which is why Reasoning and Vision, with one qualifying benchmark each today, have no category score. That bar applies to every anchor and is re-measured on each build: when a board's coverage falls below it the board leaves the anchor set, and a category left with one qualifying board leaves the list. Long context left it on 28 September 2026 for exactly that reason. Coding, Agentic and Science followed on 30 September when their newest anchors fell below the same bar. The boards that have left are recorded with their measured coverage in data/category-score-anchors.json. Category scores are raw benchmark results, not percentiles, and every category has its own anchors — compare models within one category, not a Coding score with a Science score.

Provider data policy

“Trains or keeps your data” is taken from OpenRouter's published provider table: a provider passes when it appears under both of OpenRouter's own filters, “Does not train” and “Zero retention”. While the option is unchecked, providers that fail that test are removed from every evaluation on this site. Two things are stated openly rather than hidden. Chutes is treated as satisfying both, because OpenRouter miscategorises it. And providers that OpenRouter does not list at all — the European sovereign hosts among them — have no published verdict to read; they are kept and marked as unknown, because we will not assert a data policy we have not read.

Subscriptions and company use

Every cost ranking on this site uses API prices; that is the clean baseline. Subscriptions such as ChatGPT Plus/Pro or Claude Pro/Max are a separate topic, in the folded note “Subscription costs may differ” below the ranking. They can make the effective cost per task much lower for heavy users, but they are not fixed token bundles: session, weekly and other limits vary and are mostly unpublished, so their effective cost per task depends on how much you use them and is never a universal number. They are not universally business-safe equivalents of API access either: account sharing, resale and automation are restricted, rate limits apply, and commercial-use eligibility depends on the provider, the plan, the region and the contract. We read the terms served to us (EEA): Anthropic's consumer terms (Claude Pro/Max) say “Non-commercial use only”, and Google says “Only personal Google Accounts can sign up for Google AI plans”, so with “buying for a company” set those plans are set aside and business seats are shown instead; Team, Enterprise and API access are the safer commercial categories. Cursor's terms explicitly allow use on behalf of an entity. GitHub steers organisations to Copilot Business without forbidding individual plans, so that is flagged as depending on plan and contract. OpenAI and xAI refuse automated reads of their pages; we do not work around that, so their prices and terms are marked as not collected. For a plan whose price we have, you can estimate its cost per task yourself: (monthly fee + any extra usage charges) ÷ the tasks you complete per month. Every input is your assumption and the result is labelled as a subscription estimate, apart from API costs. The plan list also shows the break-even point — the plan's price divided by the adjusted API cost per task of that vendor's best-scoring model in your view.

Matching models to prices

Coding Agent results are attached to an exact or explicitly audited model/reasoning identity; every harness result is retained and their median is used; like the AA indices it is never copied to another effort setting. Family-scoped Epoch ECI and Intelligence.ai / DesignArena results are stored once on the deterministic collapsed-family representative; the composite and the model pages share them with the family's other settings as attached values (see the composite score above), and the provenance note states explicitly that this does not identify the tested effort setting. Raw DesignArena score views continue to show Elo. Stable model ids and repositories are preferred over fuzzy names, so distinct releases, modes, context tiers and serving routes never share the wrong price. See the repository README and data/SCRAPING.md for how each source is collected and refreshed.

Benchmark tags and the comparison table

The Benchmarks table groups every benchmark into one category and tags its name. AA marks results measured by Artificial Analysis; Arena marks human head-to-head votes scored as Elo. Headline, Niche and Community describe prominence, not quality: headline benchmarks are the ones model release posts and technical reports usually cite, niche ones come from a company or research lab and are rarely cited, community ones are run by an independent person or project. These tiers are editorial: one model drafted them, a different model reviewed them, and both the tags and the categories live in data/benchmark-taxonomy.json. A table cell shows the latest published result for that exact model configuration — measured results are preferred, a developer's own report is marked †, and a missing result stays a dash. Bars compare values within one row only; the bold value is the row's best, and ties are all bold.

Wherever this site counts benchmarks, it counts boards: one benchmark family at one version, with at least one published result. A board can occupy several rows of the comparison table — one per harness a benchmark was run through, plus the measured cost a run of it reported — and those rows are still one benchmark. The registry lists every version we have ever read, including versions no current model has a result for, so it can be larger than the number the page shows.

Two further tags are not editorial. Saturated — “Top models sit near this benchmark's ceiling; it separates weaker models, not the best.” — is measured from the results we hold: the benchmark is on a bounded, higher-is-better scale, we have independently measured results for at least five models, and the mean of the five best results is at or above 90 % of the benchmark's ceiling. Nothing is asserted from a claim, and a benchmark we cannot assess (an Elo board, an open points scale, fewer than five measured models) is not called unsaturated — it simply carries no tag. Judged — “A preference or judge score, not task accuracy.” — marks benchmarks whose number ranks or rates outputs by preference or quality: human head-to-head votes, an Elo from a judge panel, a rubric graded by a judge model. A judge that only checks whether an answer is correct (an equality checker, a majority vote on accuracy) is not judged, because the ground truth still decides. Each Judged classification quotes the benchmark's own registry text; the classifications and the quotes live in data/benchmark-caveats.json and a test fails if a quote is not found verbatim in its source.

Both tags change how a category composite is built. A preference or judge score never averages with task accuracy: when a category shows both kinds, only the task-accuracy rows make its composite (a category whose qualifying rows are all judged gets a composite of those, and says so). A saturated benchmark still counts, at half the weight of an unsaturated one. When the Science category score qualifies, this makes its CritPt and GPQA Diamond average weighted rather than flat. The same rule governs the selectable category scores in data/category-score-anchors.json; a judged benchmark may not be an anchor at all, and the build fails rather than silently changing a published score if one is ever reclassified. None of the seven slots of the Benchmark Heaven Main Composite is saturated today; two of them (DesignArena Web Apps and Full-Stack) are preference scores, and they enter as their own separate, percentile-normalised slots rather than averaged into task accuracy.

Where the benchmark's own source states a task or question date window, or a contamination control (a private or held-out set, a semi-private set, a periodically refreshed task set), the benchmark's (i) repeats it. Where the source states nothing, the field says exactly that — we do not infer a date window from a benchmark's name. Every row also names the version it is and the date we last verified its results from the source.