Multi-Model Routing: Send Each Task to the Right Model

At the end of July, OpenAI cut the price of its GPT-5.6 Luna model by up to 80 percent, down to about 20 cents per million input tokens and a dollar twenty per million output. The same week it said its models now reach more than a billion people and two million businesses. Every AI newsletter ran the same take: inference is basically free now, the race to the bottom is on, margins are dead. Forbes literally called it a race to the bottom.

Here is the durable version, the one that still holds after the price war forgets which model was cheapest this month. Cheaper models do not lower your bill. Cheaper models lower the price of one option on a menu you are probably not reading. If your product sends every request to your single best model, a price cut three tiers below it changes nothing for you, because you never route anything there.

I have made this mistake with real money. I wired an early product straight to the most capable model I could find, told myself I would optimize later, and watched the invoice climb while most of the traffic was work a model costing forty times less would have nailed. The fix was not a cheaper model. The fix was a decision I had skipped: which model should this specific request go to?

That decision, made automatically, per request, is multi-model routing. It is the least glamorous and most underrated piece of AI infrastructure a builder can own in 2026. This is a builder’s guide to what it is, the framework for making the routing call, the four patterns you can actually ship, and the failure modes that will bite you if you route by vibes. If you want the wider map of where AI value is moving, I keep that in the AI opportunity map. This post is the piece of it most teams skip.

What this covers

• The default tax: why one model for everything is the expensive choice
• The framework: the routing layer and the three routing questions
• The model ladder: the tiers you are routing across
• The four routing patterns you can actually ship
• The confidence cascade in detail
• The eval gate: you cannot route what you cannot grade
• The small-model surprise nobody prices in
• The contrarian take: routing is a measurement problem
• What to do Monday morning
• FAQ

The default tax: one model for everything is the expensive choice

Most AI products are built the way I built mine. You pick a model that is good enough for the hardest thing your product does, you point everything at it, and you ship. It works. That is exactly why it is dangerous. The default never announces itself as a decision, so nobody ever revisits it.

Call it the default tax. It is the money you spend routing easy work to an expensive model because you never built the thing that would send it somewhere cheaper. The tax is invisible on any single request. A few cents here, a fraction of a cent there. It only shows up in aggregate, at the bottom of an invoice, after you have scaled enough that it hurts.

The size of the tax is set by two facts about the 2026 model market. The first is the spread. Prices across general-purpose models now span roughly 140 times, from around three and a half cents per million tokens at the cheap end to five dollars for standard frontier tiers, and output prices stretch from under thirty cents to seventy-five dollars per million at the top. A cheap classification model and a frontier reasoning model are not 10 percent apart on price. They are two orders of magnitude apart. Routing the wrong request to the wrong end of that range is not a rounding error. The cheap end keeps getting cheaper too, subsidized by a price war I traced in the economics of the AI free tier, which only widens the gap between what you could pay and what you do.

The second fact is that the spread keeps widening and reshuffling. New models land every few weeks, each at a different price and quality point. The team that hardcoded one model in early 2026 is, by mid-year, paying frontier rates for tasks that three newer and cheaper models now handle. This is the same commoditization I wrote about in the AI commoditization clock, seen from the buyer’s chair. Model quality at any given price point drops on a clock. If you are not set up to move work down the ladder as it opens up, commoditization is happening to your competitors and not to you.

Here is the part that should sting. In most production workloads, the share of requests that truly need your best model is small. When researchers measured this carefully, they found that a good router can hold 95 percent of frontier-model quality while sending the expensive model only 14 to 26 percent of requests. Flip that around. Something like three quarters of what you are paying top dollar for could go somewhere cheaper with no quality loss a user would notice. You are not buying capability. You are buying the absence of a routing decision.

The reason this persists is not laziness. It is that the alternative sounds like a research project. Building a router feels like it needs a machine-learning team, a training pipeline, and a month you do not have. It does not. The core of it is a framework and a config file. Let me give you the framework first, because the framework is what makes every later decision obvious.

The framework: the routing layer and the three routing questions

Stop thinking about model choice as a setting. Start thinking about it as a layer. The routing layer sits between your application and the model providers. Your code asks it for a completion. It decides which model that completion comes from, and your application never hardcodes a model name again. That single architectural move, pulling the model choice out of your business logic and into one place, is 80 percent of the value before you have written a line of routing logic. It also quietly solves the vendor problem I covered in the AI vendor lock-in playbook. When the model name lives in one config instead of scattered across your codebase, switching costs collapse from a migration to an edit.

Once you have a layer, the only question left is how it decides. And every good routing decision comes down to three questions asked of each request. I call them the three routing questions, and they are the hero of this whole framework because they turn a fuzzy judgment into a checklist.

Difficulty is the obvious one. A lot of what your product does is not hard. Pulling a date out of an email, tagging a support ticket, deciding whether a message is a question or a complaint, turning a paragraph into structured fields. These are not reasoning problems. A small model does them at full quality for a fraction of a cent. Reserve the expensive model for the genuinely hard work: multi-step reasoning, long-context synthesis, code that has to run, judgment calls with no clean answer.

Stakes is the one teams forget, and it is the reason difficulty alone is not enough. Two requests can be equally easy and still deserve different models, because a wrong answer costs different amounts. A mislabeled internal note is a shrug. A wrong number in a customer invoice is a refund and an apology. When the cost of being wrong is high, you buy the more capable model even for easy work, because you are not paying for capability, you are paying for a lower error rate. Routing on stakes is how you avoid the trap of optimizing your bill straight into a customer incident.

Latency is the tie-breaker that changes the math in both directions. If a human is watching a cursor blink, a slow frontier model is a worse product even when it is a smarter one, and a fast small model wins on experience. If the work is a background job nobody is waiting on, you can send it to a slower, cheaper path, or batch it, or run a cascade that tries two models in sequence. The same request routes differently depending on whether someone is waiting, and most teams never encode that.

Three questions. Difficulty, stakes, latency. Run every request through them, even informally at first, and the model choice stops being a default and starts being a decision. Now you need to know what you are choosing between.

The model ladder: the tiers you are routing across

You do not route across forty models. You route across three or four tiers, and you keep one or two named models in each tier so you can swap the model without changing the routing logic. Think of it as a ladder. Most requests should ride the bottom two rungs. The top rung is for the work that earns it.

The exact model names in each rung will be stale by the time you read this, and that is the point. You do not route to “GPT-5.6 Luna” or “Claude Haiku 4.5” in your code. You route to “small tier” and let a config file say which model that is this week. When a provider cuts a price 80 percent overnight, you change one line and the savings flow through every request already pointed at that rung. The teams that felt nothing when Luna’s price dropped are the ones who never built the rung to catch it.

Notice the fourth row. The fine-tuned specialist is the rung most builders never climb to, and it is the one with the widest moat, so I will come back to it. For now, hold the shape in your head: a handful of tiers, most traffic riding the cheap ones, the model names swappable underneath. That is what a routing layer routes across. The next question is how it decides which rung a given request takes.

The four routing patterns you can actually ship

There are exactly four ways to make the routing decision, and they trade off against each other on a simple axis: how much do you spend deciding, versus how good the decision is. You will end up using more than one. Here they are, cheapest-to-decide first.

Static rules are the pattern everyone underrates. You route by something you already know about the request: which endpoint it came from, which feature it belongs to, whether it is a background job or a live chat. The extraction endpoint always goes to the small tier. The reasoning endpoint always goes to frontier. There is no cleverness here and that is the strength. It costs nothing to decide, it never misfires in a surprising way, and it captures most of the available savings on day one. I have never seen a team that jumped straight to a learned router without first leaving easy money on the table that static rules would have swept up.

Classifier routing is what you add when static rules run out. Some requests hit the same endpoint but vary wildly in difficulty. A support inbox gets both “reset my password” and “your API returned a 500 on this specific payload and here is the trace.” A tiny model or an embedding model looks at the incoming text and predicts which tier it needs, before you spend a cent on the real answer. This is the pattern the research community formalized. A router trained on human preference data held 95 percent of frontier quality while sending the strong model only 14 percent of the queries. And the property that makes it worth building is transfer: a router trained on one strong-weak model pair kept working when the underlying models were swapped at test time. You are not retraining every time a provider ships a new model. The router learns “hard versus easy,” which is durable, not “GPT-this versus Claude-that,” which is not.

Cascade is the most accurate pattern on cost, and it deserves its own section, so I will keep this short here. You try the cheap model first, check whether the answer is good enough, and only escalate to the expensive model if it is not. Done well, this is how FrugalGPT reported up to 98 percent cost reduction. The catch is that hard requests now pay two model calls instead of one, and they wait longer, which is exactly why latency was one of your three questions.

Fallback is the pattern that does not belong with the other three, and putting it in the same mental bucket is how teams cause outages. Fallback is about availability, not cost. When your primary provider returns a 500 or times out, you retry on a different provider so the user still gets an answer. It shares plumbing with routing, the same layer, the same config, but it is answering a different question. Cost routing asks “which model should handle this.” Fallback asks “my chosen model is down, now what.” Conflating “pick the cheapest” with “survive an outage” is a documented way to turn a provider blip into a product incident. Keep them as separate concerns that happen to live in the same layer.

You will run three of these at once in a mature system: static rules as the base layer, a classifier for the messy endpoints, a cascade for the background work, and fallback wrapped around all of it for uptime. You do not build them in that order, though. You build static rules, ship, measure, and add the next pattern only where the numbers say the simpler one is leaving money behind.

The confidence cascade in detail

The cascade is worth slowing down on because it is where routing stops being a lookup table and starts needing judgment. The whole pattern hinges on one hard question: how do you know the cheap model’s answer was good enough to keep?

There are a few honest ways to answer “good enough,” and they vary by task. For structured output, the cheapest check is mechanical: did the answer parse as valid JSON, did it fill every required field, did the values fall in the allowed ranges. If it did not, escalate. For tasks where the model can rate its own certainty, you can ask it, though self-reported confidence is a weak signal and you should treat it as one input, not the verdict. For the highest-stakes work, you use a second model as a judge, which starts to look a lot like the machine-checks-machine pattern I described in the AI verification layer. The verification layer and the routing layer are cousins. One decides which model answers, the other decides whether to trust the answer, and a serious cascade uses both.

The failure you have to design against is the drifting check. If your confidence test is too eager to escalate, the cascade quietly sends everything to the frontier model and you are paying more than if you had never built it, plus the latency of the failed first attempt. If it is too reluctant, bad cheap answers slip through to users. A cascade with a miscalibrated gate is worse than no cascade at all, because it adds cost and latency while pretending to save. This is why you cannot ship a cascade without the ability to see what it is doing, which is the same argument I made about silent failures in AI agent observability. A router you cannot observe is a router that is lying to you about its savings.

Reserve cascades for two situations. Background jobs where the extra latency on hard requests does not touch a user. And tasks where “good enough” has a cheap, mechanical test, like schema validation, so the confidence check itself does not cost more than the model call it is guarding. If you find yourself using an expensive judge to decide whether to use an expensive model, you have built a machine that spends money to think about spending money. Fall back to a classifier or a static rule.

The eval gate: you cannot route what you cannot grade

Here is the sentence that separates teams who save money with routing from teams who cause incidents with it. You cannot route what you cannot grade. Every routing decision is a bet that a cheaper model is good enough for this request, and the only way to know if you are winning the bet is to measure quality, not to feel it.

The risk in production AI has quietly moved. A few years ago the thing that broke was throughput: the system fell over under load. Now the thing that breaks is silent quality regression. The system keeps returning answers, they keep looking plausible, and quality has dropped in a way no exception will ever catch. Routing is a direct source of exactly this failure, because a misroute produces a bad answer while every system stays green. Nothing throws. A user just gets a worse response than they would have, and you find out from a churn number three weeks later.

So the eval gate is not optional infrastructure you add later. It is the thing that makes routing safe to ship at all. In practice it looks like this. You keep a set of 50 to 500 representative requests with known-good outputs, the kind of cases your product actually sees. Any change to routing logic, a new model in a tier, a tweaked classifier, a looser cascade threshold, has to run against that set in continuous integration before it merges. If quality drops below your threshold, the change does not ship. That is the gate.

Then you roll out the way you would roll out any risky change. Shadow-test first: mirror a slice of real traffic to the new routing, log both the old and new answers, and score them against each other for a few days before a single user sees the new path. When the numbers hold, ramp behind a flag, 1 percent, then 10, then 50, then 100, watching both quality and cost at each step. This is not routing-specific ceremony. It is how you ship anything that can regress silently, and routing is the definition of something that can regress silently.

The uncomfortable implication, which I will make the centerpiece of the contrarian section, is that the eval set is the hard part and the router is the easy part. Teams spend their energy on the clever routing logic and skip the boring eval harness, then wonder why routing feels risky. It feels risky because they are flying blind. Build the grader first. The router is downstream of it.

The small-model surprise nobody prices in

The fourth rung on the ladder, the fine-tuned specialist, is where routing crosses from cost savings into something closer to a moat, and almost nobody builds it because it sounds harder than it is.

Here is the surprise. On a narrow, well-defined task, a small model that you have fine-tuned on your own examples routinely matches a frontier model used out of the box. Fine-tuned models in the three-to-nine-billion-parameter range hit frontier-level accuracy on classification and structured extraction, the exact work that makes up a huge share of most production traffic. Serving a seven-billion-parameter model runs ten to thirty times cheaper than a seventy-to-hundred-and-seventy-five-billion model, and cuts infrastructure cost by up to three quarters. And the training itself is no longer a project. A LoRA fine-tune on a thousand to a hundred thousand of your own examples now costs under five dollars and finishes in hours on a single GPU.

Sit with that. For less than the price of a coffee and an afternoon, you can build a model that does one of your high-volume tasks as well as the frontier and runs at a small-tier price forever. This is not a lab result. It is a Tuesday. The reason it is a moat and not just a saving is that the fine-tuned model is trained on data only you have, your tickets, your documents, your labeled outcomes, so a competitor routing to a general model cannot match your cost-quality point on that task without your data. Cheap inference is available to everyone. A specialist trained on your own exhaust is not, and that scarcity is exactly the kind of durable premium I argued for in the case for the atoms premium: the copyable parts race to zero, and the value collects around what cannot be copied.

Routing is what makes the specialist usable. You do not fine-tune one giant model to do everything, which would be slow and brittle. You fine-tune small specialists for your two or three highest-volume narrow tasks, and the routing layer sends exactly those tasks to them while everything else rides the general tiers. The specialist is a rung on the ladder, and the router is the thing that knows when to reach for it. This is the same asset-from-your-own-data logic I traced in the wider AI opportunity map, applied to the model layer instead of the product layer.

The contrarian take: routing is a measurement problem

Almost everyone who talks about multi-model routing talks about it as a model problem. Which model is best, which router is smartest, which gateway has the most integrations. They are optimizing the wrong noun.

Routing is a measurement problem wearing a model problem’s clothes. The hard part was never choosing between models. The hard part is knowing, for your specific traffic, which requests a cheaper model can handle without anyone noticing. That is a question about your data and your quality bar, and it can only be answered by measuring, not by reading a benchmark. Two products with the same tech stack will have different optimal routing because they have different traffic and different stakes. The benchmark on the model provider’s website cannot tell you where your line is. Only your eval set can.

This flips the whole build order. The instinct is to start with the router, because the router is the interesting engineering. The correct move is to start with the grader, the boring one. Once you can score a batch of requests against known-good answers, routing becomes almost mechanical. You send a slice of traffic to a cheaper model, you score it, and the model tells you where the line is instead of you guessing. Without the grader, every routing decision is a guess dressed up as an optimization, and guesses in production show up as the silent regressions I keep coming back to.

There is a strategic version of this too, and it connects to the platform risk I wrote about in AI platform risk for founders. When your model choice lives in a routing layer governed by an eval set, you are not exposed to any single provider’s roadmap, price, or decision to compete with you. A provider raises prices, you reroute. A provider ships a competing feature, your routing does not care, it was never loyal to them. The eval set is your bargaining power over the whole model market, because it lets you swap any model for any cheaper one the moment the numbers say you can. Teams that hardcoded a model gave up that power without noticing. The routing layer is how you keep it.

So the contrarian one-liner is this. Do not ask which model is best. Ask which requests you can prove a cheaper model handles, and build the thing that proves it. The router is a consequence of measurement, not a substitute for it.

What to do Monday morning

This is not a research program. You can start capturing the default tax this week. Here is the order I would run it in, and it deliberately front-loads the cheap wins.

First, log your traffic by task. For one week, tag every model call with what it was for: extraction, chat, reasoning, summarization, whatever your real buckets are. You are looking for one number per bucket, the share of your spend that goes to your most expensive model. Most teams have never seen this breakdown and it is usually a shock. That number is your default tax, made visible.

Second, pull the model name out of your code. Even before any routing logic, move the model choice into one config file or one small module that every call goes through. This is the routing layer with a single hardcoded route. It buys you nothing yet on cost, but it is the foundation, and it immediately cuts your switching cost the way I described in the vendor lock-in playbook. Do this even if you never route another thing.

Third, build the eval set before the router. Collect 50 to 100 real requests per major task with outputs you trust. This is tedious and it is the highest-value hour you will spend, because it is the grader that makes every later routing change safe. If you only do one thing from this list, do this one.

Fourth, ship static rules for the obvious buckets. Your extraction and classification traffic goes to the small tier. Your genuine reasoning traffic stays on frontier. Run it against the eval set, confirm quality holds, ramp it slowly. This alone captures most of the money, because the obvious buckets are usually most of the volume.

Fifth, only now add a classifier or a cascade, and only on the buckets where your logs show the simple rules are leaving money behind. Measure, do not assume. And keep fallback as a separate concern for uptime, wired into the same layer but answering the availability question, never confused with the cost question.

Sixth, revisit the config monthly. The model market moves fast enough that the right model in each tier changes every few weeks. When a price drops 80 percent overnight, the only work you should have to do is change one line and watch the eval set stay green. That is the whole payoff of building the layer: the market does the price cuts, much of it funded by the compute subsidy propping up cheap inference right now, and you just catch them.

None of these six steps needs a machine-learning team. They need a week of logging, an afternoon of eval collection, and the discipline to measure before you optimize. The teams that will spend the next few years overpaying are not the ones who lack the skill. They are the ones who never made the model choice a decision.

FAQ

What is multi-model routing?
Multi-model routing is the practice of automatically sending each AI request to the most appropriate model instead of pointing everything at one model. A routing layer sits between your application and the model providers and decides, per request, which model tier should answer, based on how hard the request is, what a wrong answer costs, and whether a human is waiting. Done well, it holds quality while cutting cost by routing most traffic to cheaper models and reserving your best model for the work that needs it.

How much can multi-model routing actually save?
On the traffic you route, savings of 60 to 85 percent are realistic without a quality drop users notice, because a good router can hold about 95 percent of frontier-model quality while sending the expensive model only 14 to 26 percent of requests. Cascade patterns on suitable tasks have reported up to 98 percent cost reduction. The real number depends entirely on your traffic mix: the more of your volume is easy work currently routed to an expensive model, the larger the win.

Do I need to train a machine-learning model to route?
No. The highest-value routing pattern is static rules, where you route by task type or endpoint using logic you already understand, and it captures most of the available savings with zero training. You only add a learned classifier when the same endpoint gets requests of wildly different difficulty. Even then, you can often use an off-the-shelf router or a small model as the classifier rather than training your own.

What is the difference between a cascade and a classifier router?
A classifier decides which model to use before answering, by predicting the difficulty of the request. A cascade decides after answering, by trying the cheap model first, checking whether the answer is good enough, and escalating to a stronger model only if it is not. Classifiers are faster because they commit once. Cascades can be more accurate on cost because they only pay for the strong model when the cheap one demonstrably fails, but they add latency on hard requests that get answered twice.

Is multi-model routing the same as failover?
No, and treating them as the same causes outages. Cost routing answers “which model should handle this request.” Failover answers “my chosen model is down, how do I still return an answer.” They share the same layer and config, but they solve different problems. If you conflate “pick the cheapest model” with “survive a provider outage,” a routine provider blip can turn into a product incident. Keep them as separate concerns that happen to live in the same place.

What is an LLM gateway and do I need one?
An LLM gateway is a piece of infrastructure that normalizes many model providers behind one interface and centralizes routing, fallback, budgets, logging, and rate limits. Open-source options like LiteLLM let you self-host a proxy across a hundred-plus providers; marketplaces like OpenRouter give access to four hundred-plus models through one endpoint; others focus on production safety features like guardrails and audit trails. You do not strictly need a gateway to start, since your first routing layer can be a config file, but a gateway saves you from rebuilding routing, fallback, and observability yourself as you scale.

How do I stop routing from quietly hurting quality?
Build an eval set before you build the router. Keep 50 to 500 representative requests with known-good outputs, and require every routing change to pass a quality threshold against that set in continuous integration before it merges. Then roll changes out gradually: shadow-test against real traffic for a few days, then ramp behind a flag from 1 to 10 to 50 to 100 percent while watching quality and cost. Routing failures are silent by nature, so observability and evals are what make routing safe, not optional extras.

When should I fine-tune a small model instead of routing to a general one?
Fine-tune a small specialist when you have one narrow, high-volume task and your own labeled examples for it, such as a classification or extraction job you run millions of times. A fine-tuned model in the three-to-nine-billion-parameter range often matches frontier accuracy on that specific task while running at a fraction of the cost, and a LoRA fine-tune now costs under five dollars and takes hours. Because it is trained on data only you have, it also becomes a durable advantage a competitor cannot copy by routing to a general model. Route everything else to the general tiers and reserve specialists for your highest-volume narrow work.