It is August, which makes this roughly the halfway mark, and I have lost count.
Somewhere around the fourth frontier release in six weeks I stopped keeping a mental table of which model was ahead. They arrive faster than anyone outside a lab can evaluate them. Each one lands with its own benchmark card, its own effort setting, its own quiet choice about how a failure gets counted, and by the time you have worked out whether two of the numbers can even be compared, a third model has shipped. Half of them are open weights now. That used to be the distinguishing feature, and it no longer narrows the field at all.
So I sat down to do the thing I had been putting off since spring, which was to work out properly which model we should be standardising on.
Then yesterday Mark Zuckerberg published fourteen pages arguing that American open-source AI needs fewer restrictions to compete with China, and shipped Muse Glimmer to prove he meant it. Meta has committed to opening the weights of Muse Spark 1.2 as well. That was the model it launched in April as its first closed, proprietary frontier system, the one that was supposed to end the Llama era.
That era lasted four months.
Most of the commentary has read this as a geopolitical story, and it is one. But if you build systems for a living, the geopolitics is downstream of something more useful. The question “which model should we use” has quietly stopped having an answer, and not because the answer is hard. It’s the wrong shape of question.
Your model choice is now a routing policy. Everything below is why, and how to build one.
Two scoreboards
The AGI race has two scoreboards and only one of them gets reported.
The reported one is capability. On the Artificial Analysis Intelligence Index, Claude Opus 5 sits at the top. Kimi K3 lands fourth, a couple of points behind GPT-5.6 Sol and Claude Fable 5. That is 2.8 trillion parameters, the largest open-weight release anyone has ever made, on Hugging Face since 27 July. It’s the best result an open-weight model has ever posted, and it’s still fourth. On the hardest reasoning evaluations the closed frontier keeps a lead of roughly three to eight points, and has for a year.
So there’s no parity at the top. Anyone telling you otherwise is reading launch slides.
The unreported scoreboard is what engineers actually chose when they had to ship. A year ago Chinese open-weight models carried under two percent of the tokens flowing through OpenRouter. They now carry more than forty-five percent. Four of the five most-used models are Chinese. Llama, the open-weight leader two years ago, has fallen off the rankings entirely.
Nobody live-tweets a routing curve. It went vertical anyway.
These two scoreboards disagree, and the disagreement is the whole story. The frontier labs are winning a contest about the ceiling. The people building things are answering a question about the floor, and the floor came up so fast that the ceiling became somebody else’s problem.
You only need one of these scoreboards, and it isn’t the one on the front page. Capability rankings tell you which models are worth putting in a bake-off. They can’t tell you what to ship. The second scoreboard you have to build yourself. What clears your bar, at what cost, on your workload. It’s the only one your bill responds to.
Why the floor is the interesting number
Every production system I have built in twenty years has been an exercise in finding the cheapest thing that clears the bar. Not the best thing. The cheapest thing that clears the bar, because the bar is set by the workload and everything above it is money you set on fire at scale.
Break the aggregate benchmark scores apart and you can see exactly where the bar got cleared. Kimi K3 takes the Frontend Code Arena outright. It leads on BrowseComp, SpreadsheetBench and Automation Bench, the evaluations closest to the work a business actually does. What it doesn’t win is frontier reasoning. Which is a precise description of most production workloads: a thin layer of genuinely hard decisions sitting on a mountain of tool calls, extractions, classifications, retries and formatting.
Then look at the price column. DeepSeek V4 Pro has been running about fifty times cheaper per Intelligence Index task than the model at the top of it. Fifty. There’s no three-point benchmark delta that survives that ratio on a workload you run a billion times a month.
The closed labs can read this too. OpenAI split GPT-5.6 into three tiers and then cut the cheapest by eighty percent at the end of July. That’s what defending a position looks like. You don’t cut eighty percent off a product that’s winning.
For the first time the capability curve and the cost curve have separated far enough that routing between them is worth real engineering effort. Two years ago the spread was too narrow to bother with; you picked a vendor and moved on. It isn’t narrow now.
What a model routing layer actually looks like
Here’s the part that matters on Monday.
1️⃣ Tier by failure cost. The instinct is to route by category, coding here, summarization there, and that’s the wrong axis. Route by what a wrong answer costs you. A bug-fix suggestion a human reviews before merging is cheap to get wrong. A migration script that runs unattended against production isn’t, even though both are “coding.” Sort your calls into expensive to be wrong and cheap to be wrong, and you’ll find the second bucket is eighty to ninety percent of your tokens and almost none of your risk.
2️⃣ Put the volume on weights you host. The cheap-to-be-wrong tier is where open weights pay for themselves, and not because they’re free. You control the serving stack, the batching, the quantization and the latency floor. It’s also the tier where you can afford to run two models and compare, since a second opinion costs you almost nothing.
3️⃣ Keep the frontier for the thin layer. Long-horizon agentic work, anything where the model has to hold a plan across many steps, anything unattended, anything where a confident wrong answer propagates. Pay for it and don’t feel bad. It’s a small fraction of your tokens.
4️⃣ Make the eval set the gate. A router without one is a config file full of guesses, and what you’re shipping is vibes. You need a fixed suite of representative calls from your own workload, scored the way you actually judge output, run against every candidate model. Public benchmarks tell you which models are worth putting in the bake-off. They can’t tell you which one wins on your distribution, and the gap between those two things is where most model-selection decisions go wrong. Published scores depend on effort level, attempt count, tool availability and how failures are counted. None of that matches your environment.
5️⃣ Route on measured cost per completed task. A model at a fifth of the price that needs three attempts and a longer reasoning trace is not cheaper. Per-token pricing is the number vendors publish; cost-per-successful-completion is the number that hits your bill. They diverge constantly, and the divergence is often the opposite of what the price table suggests.
6️⃣ Every tier should degrade to the tier above it on failure, and that escalation belongs in a first-class code path with its own metrics rather than a try/except that quietly retries against something expensive. If you can’t answer “what fraction of calls escalated last week and what did that cost,” your router is a liability.
7️⃣ Self-host anything touching regulated data. The hosted APIs for these models run in China, and “the weights are open” is not a data residency argument. Downloading them is what makes the compliance conversation tractable in the first place. Licences vary far more than people assume. Alibaba keeps Qwen open under Apache 2.0 while putting its best tier behind an API, and Moonshot shipped K3 under its own bespoke licence rather than MIT. Read them before you standardize.
8️⃣ Assume every choice you make is temporary. The half-life of “obviously the best model” is about six weeks right now. Between late April and mid-June, four frontier-competitive open models shipped within weeks of each other. Anything hardcoded against a specific provider is technical debt with a launch calendar attached. The abstraction layer isn’t premature optimization anymore; it’s the only part of the stack with a stable lifespan.
What survives the next release
Those fourteen pages will read as a period piece inside a year, and so will every number I quoted above. All of it has a short shelf life. That’s the point.
What doesn’t expire is the shape of the problem. There is a ceiling and there is a floor, they move at different speeds, and the gap between them is wide enough to route through. If it ever closes again, a router is how you’ll find out, because you’ll be measuring both ends of it every day.
So stop asking which model to use. That question has had no stable answer for about a year, and the release calendar it depends on isn’t yours. Ask what a wrong answer costs you instead, and let that sort your traffic.
The reason to do it is organisational. A router every team calls is one place where “which model, for what, at what cost” gets decided, so two teams asking similar questions get similar answers and somebody can finally say what the organisation’s AI costs. Treat it as infrastructure, beside the API gateway and the feature flags. That is what makes it reasonable to put a model inside a real workflow, because the trust you need lives in the layer around the model rather than in the model itself. Keep that layer moving at the pace the releases do and the next launch stops being a threat: a model nobody has used is a candidate you can score overnight, and adopting it, open weights or closed, becomes a decision you make on your own evidence.
Build your own scoreboard. It’s the only one that matters to you.