The Expensive Default

Every product built on a large language model eventually runs into a question that has nothing to do with prompts or fine-tuning. A request arrives, and something has to decide which model answers it. Send everything to the most capable model available and the bill grows in proportion to traffic, whether or not the traffic needed that much capability. Send everything to a cheaper model and some fraction of users get worse answers than the product could have given them.

The engineering answer to that tradeoff is a router, a lightweight system placed in front of the language models that reads each incoming request and assigns it to one of them. Early academic work on this problem found that many questions do not need the strongest available model at all. A quality-aware router trained to send only the harder queries to a large model reduced calls to that model by up to 40 percent with no measurable drop in response quality ^[1]. A follow-up framework refined the idea using human preference data collected from model comparisons, and on one benchmark it matched 95 percent of a strong model's score while sending only 13.4 percent of requests to that model ^[2]. The premise looked sound. Most requests are easier than the hardest request a product will ever see, and a router that can tell the difference should save money without anyone noticing.

Reaching for the Expensive Model Anyway

Deployed routers turn out to behave differently from the ones described in the papers that introduced the idea. A 2026 study tracking router behavior as the allowed budget per query increases found something researchers had not previously isolated as a distinct failure. Instead of spreading requests across the available models based on difficulty, the call rate of the single most expensive model climbs steadily and eventually saturates near 100 percent, the reading pushed toward the top of the scale regardless of whether the query in front of it needed that much, even though a hindsight-optimal router on the same benchmark uses that model for fewer than 20 percent of queries under the same budget ^[3]. The researchers behind that finding named it routing collapse, and traced the cause to something structural rather than a training bug. Most queries have several models bunched close together in quality, and when the top two or three candidates are nearly tied, a routing model's small prediction errors are enough to flip which one looks best. As the budget grows and more models become affordable, that instability consistently pulls the decision toward the strongest, most expensive option, because a router comparing near-ties has no reliable way to settle on the cheaper one instead.

The opposite failure shows up just as often. A benchmark spanning more than 400,000 queries across 21 datasets and 33 models found that current routers, even ones sold as commercial products, still fail to land on the one correct model for a meaningful slice of requests. On the subset of test queries where only one to three of the available models could answer correctly, two widely used routing methods hit just 23 to 25 percent accuracy at identifying which one ^[4]. So a router can spend too much by defaulting to the strongest model when it did not need to, and separately miss the model that actually had the right answer when specificity mattered. Both problems come from the same root cause. Comparing models that perform almost identically on most requests is a harder discrimination task than comparing models with a wide quality gap, and most real traffic falls into the first category.

Naming the Target Instead of Guessing

Most routers ask an operator to set an indirect parameter, a cost threshold or a confidence cutoff, and observe afterward what accuracy that setting happened to produce. The result lands somewhere near the outcome the operator wanted, rarely exactly on it. One router built specifically against this complaint takes the opposite approach, accepting a target accuracy as a direct input rather than something inferred from unrelated knobs ^[5]. Internally it tracks how far recent decisions have drifted from the requested target and adjusts its own aggressiveness in real time, letting a single trained system serve a whole range of accuracy requirements without retraining for each one.

The reported results suggest the direct-target approach closes much of the gap described above. Tested against a baseline that also tried to enforce a minimum accuracy level, the target-based router met its stated floor consistently, where the baseline met it only 22 percent of the time. On one benchmark it reached within 1.3 percent of the best achievable accuracy while cutting cost by as much as 89.8 percent compared to always using the strongest model ^[5]. The gain is not that the underlying models changed. It is that the system was finally asked to hold a specific, checkable commitment instead of a proxy for one.

Where the Discipline Has to Live

Cost and quality are not the only axis worth tracking, and treating them as the whole problem hides a third variable that shows up the moment routing decisions reach production. Two models can score almost identically on both accuracy and price and still differ enormously in how long a response takes to arrive. One documented pair produced comparable results at comparable cost with response times of 32 seconds and 262 seconds ^[4]. A router optimizing on two dimensions has no way to notice that gap, and a user waiting eight times longer for an equivalent answer will notice it regardless of what the router's dashboard says about savings.

None of this argues against routing. It argues against treating a cost threshold as a stand-in for the outcome a product actually promises its users. A routing layer is only as trustworthy as the evaluation underneath it, the same argument that applies to any system where one model's output stands in for a judgment about quality ^[2], a point explored further in Grading on a Shared Curve. A platform doing model routing should treat the accuracy floor as the product commitment and the routing table as the implementation detail underneath it, not the other way around. The question worth asking before shipping a router is not how much it saves on average. It is what happens on the request where the cheap model was wrong and nobody was checking.