Chapter 131 · Production

Model routing, and using a cheaper model where it suffices

Three ways to decide which model handles a question, why a classifier router underperforms, and the escalation pattern that took 24% off the bill with quality measured flat.

8 min read

Chapter 130 attributed 24% of a 61% cost reduction to routing. This chapter is how, and the honest version includes the approach that did not work.

The premise is uncontroversial: "what is the status of PO-4471" and "compare the return terms across our three largest suppliers and flag anything inconsistent" are not the same task, and paying the same per token for both is a choice.

Three routing strategies

Strategy	Decides	Accuracy cost	Complexity
By question class	Before generation, from the question	Highest risk	Low
By task within a run	Per internal step	Low	Medium
Escalation	Start cheap, retry expensive on a trigger	Lowest	Medium

Why a classifier router underperforms

// The obvious design: classify the question, pick a model.
const tier = await classify(question)      // 'simple' | 'complex'
const model = tier === 'simple' ? CHEAP : STRONG

It fails because the classifier has to predict difficulty from the question alone, before retrieval, and difficulty is mostly a property of what comes back.

// Atlas's classifier against ground truth (did the cheap model actually
// produce an acceptable answer?), 400 questions:
//   accuracy:            0.78
//   routed-down wrongly: 14%   -> a worse answer the user sees
//   routed-up wrongly:    8%   -> money, no harm

Fourteen percent of questions sent to the cheap model where it was not sufficient. The downside is asymmetric and lands on the user, which is the same asymmetry chapter 111 found for over-answering: the cheap failure is invisible to the person making the decision and visible to the person receiving the answer.

// Worse, the misroutes are not random. They concentrate on questions
// that LOOK simple and are not -- a short question about a near-miss
// pair of documents is six words long and needs the stronger model to
// disambiguate. Chapter 62's adjacent-clause failure, arriving through
// the router.

Escalation, which is what shipped

// Start on the cheap model. Escalate on a signal that the cheap answer
// is inadequate -- a signal available AFTER the attempt, which is the
// whole advantage.
export async function answerRouted(q: string, ctx: Ctx): Promise<Answer> {
  const cheap = await generate(q, ctx, { model: CHEAP })
  if (!shouldEscalate(cheap, ctx)) return cheap

  const strong = await generate(q, ctx, { model: STRONG })
  await metrics.increment('routing.escalated', { reason: escalationReason(cheap, ctx) })
  return strong
}

// The triggers, in order of how often they fire. Every one is a
// deterministic check that already exists for another reason.
export function shouldEscalate(a: Answer, ctx: Ctx): boolean {
  if (!SCHEMA.safeParse(a.raw).success) return true          // chapter 57
  if (a.citations.some((n) => !ctx.context[n - 1])) return true  // chapter 122
  if (unsourcedSpecifics(a.text, ctx.sources).length) return true // chapter 110
  if (a.abstained && ctx.retrievalScore > 0.7) return true   // declined with good context
  if (ctx.nearMissDetected) return true                      // competing documents
  if (ctx.intent === 'comparison' || ctx.intent === 'action') return true  // always strong
  return false
}

Atlas's escalation rate: 23%. So 77% of questions are answered by a model costing roughly a fifth as much, and the 23% pay for two generations.

// The arithmetic, which is why the rate matters more than the price gap.
//   cheap:  $0.0068   strong: $0.0317
//   0.77 x 0.0068 + 0.23 x (0.0068 + 0.0317) = $0.0141
// Against $0.0317 flat. Break-even is at roughly 68% escalation, so
// there is a lot of headroom before this stops paying.

Latency: p50 improved from 1.9 s to 1.4 s, because three quarters of answers come from a faster model. p95 worsened from 5.4 s to 6.8 s, because an escalated answer pays for both. That trade was acceptable and it is not automatic — a system whose users care more about the worst case than the median should read it the other way.

Quality, measured

// Chapter 108's paired comparison, strong-only against routed.

Slice	Strong only	Routed	Change	Interval
Overall	0.892	0.889	-0.3	[-2.1, +1.6]
lookup	0.931	0.934	+0.3	[-1.4, +2.1]
policy	0.864	0.858	-0.6	[-3.8, +2.4]
comparison	0.861	0.859	-0.2	[-2.9, +2.6]
out-of-scope	0.781	0.774	-0.7	[-5.1, +3.8]
action	0.883	0.881	-0.2	[-3.0, +2.5]

Flat everywhere, which is the result the escalation design was built to produce: the triggers catch the cases where cheap is inadequate, so the cheap model only keeps answers it handled.

Note that comparison and action escalate unconditionally, so their rows are nearly tautological. They are included rather than hidden, because a reader checking the table should be able to see which slices the design has already decided about.

Routing inside a run

// The second strategy, and it composes with escalation. Several internal
// steps have no judgement in them at all.
const STEP_MODELS = {
  rewriteQuery: CHEAP,          // chapter 46 — mechanical
  classifyIntent: CHEAP,
  summariseForRecord: CHEAP,    // chapter 95's narrative field
  extractClaims: CHEAP,         // chapter 110's extraction stage
  generateAnswer: 'routed',
  verifyClaim: STRONG,          // the judge; chapter 106 pinned it
  decideTool: STRONG,           // chapter 66 — the expensive mistake
}

Two rows deserve comment. The judge stays strong because chapter 106's calibration is specific to the judge model and a cheaper one needs a full recalibration. And tool selection stays strong because a wrong tool choice is chapter 71's problem rather than a quality problem.

Operating it

// Three things to watch, each of which has fired.
const WATCH = [
  { metric: 'escalation rate', normal: '20-26%',
    means: 'above 35% the cheap model is not earning its place; below 12% the triggers have stopped firing' },
  { metric: 'escalation reason mix', normal: 'schema 4%, citations 6%, intent 11%, near-miss 2%',
    means: 'a jump in schema failures is a cheap-model version change' },
  { metric: 'cheap-model answers rated unhelpful', normal: 'within 2 points of strong',
    means: 'a trigger gap — a failure class with no deterministic signal' },
]

The third is the real risk of the whole design and it needs saying plainly: escalation only catches failures that have a detectable signal. A cheap model producing a fluent, well-cited, schema-valid, subtly wrong answer escalates nothing. That is why the comparison above is run on every model change rather than once, and why comparison and action are routed unconditionally rather than by trigger.

Escalation converts a prediction problem into a detection problem. It is a better problem, and it is not a solved one — the failures with no signal still get through.

Key points

Routing by a classifier predicts difficulty before retrieval, and difficulty is mostly a property of what retrieval returns. Atlas's classifier scored 0.78 and misrouted 14% downward.
Misroutes concentrate on questions that look simple. A six-word question about two near-identical clauses needs the stronger model.
Escalation decides after the attempt, using signals that already exist: schema validity, citation resolution, unsourced specifics, abstention despite good context, near-miss detection, and intent.
Atlas escalates 23%. Break-even is around 68%, so there is wide headroom.
Cost per answer fell from $0.0317 to $0.0141. p50 latency improved 1.9 s to 1.4 s; p95 worsened 5.4 s to 6.8 s.
Quality was flat across every slice, with intervals spanning zero. That is the design working, not an absence of measurement.
Route internal steps separately. Query rewriting, intent classification and claim extraction are mechanical; the judge and tool selection stay strong.
Watch the escalation rate and the reason mix. Above 35% the cheap model is not earning its place; below 12% a trigger has stopped firing.
Escalation catches failures that have a signal. A fluent, well-cited, subtly wrong cheap answer escalates nothing, which is why comparison and action route unconditionally.

Next: caching: prompts, embeddings and whole responses — the other 38%.