Chapter 85 · Agent loops and control

Routing between specialised agents

The router is the cheapest component and the one with the most leverage. Four routing methods ranked by cost, the asymmetry that decides the threshold, and the misroute that cannot be recovered.

11 min read

Routing has been assumed since chapter 2 and measured repeatedly: 71% of Atlas's traffic to retrieval, 18% to a single tool, 5% to a workflow, 6% to the loop, with the loop costing eight times the baseline.

This chapter builds the router properly, because it is the component with the most leverage per line of code in the whole system.

What is being routed

Not "which agent" in the sense of separate services. Four paths through one system.

export type Route =
  | { kind: 'question'; sources?: string[] }        // retrieval, 71%
  | { kind: 'lookup'; tool: string }                // one tool call, 18%
  | { kind: 'workflow'; name: string }              // a fixed sequence, 5%
  | { kind: 'planning'; complexity: 'normal' | 'high' }  // the loop, 6%

Keeping it to four is deliberate. Every additional route is a classification boundary that can be crossed wrongly, and the gains from finer routing are small compared to the misroute cost.

Four methods, by cost

Method	Latency	Cost	Accuracy on Atlas
Patterns	<1 ms	0	0.79
Embedding nearest-neighbour	55 ms	~0	0.88
Small model classifier	180 ms	low	0.96
Large model classifier	600 ms	high	0.97

// The cascade: cheap methods handle what they are confident about, and
// only the remainder pays for a model call.
export async function route(question: string, history: Turn[]): Promise<Route> {
  // 1. Patterns, for the shapes that are unambiguous. ~34% of traffic,
  //    at zero cost and 0.99 precision on the shapes it matches.
  const byPattern = routeByPattern(question)
  if (byPattern) return byPattern

  // 2. Nearest neighbour against labelled historical questions. Free
  //    once the embedding exists, and chapter 46 already computes it.
  const nearest = await routeByNeighbour(question)
  if (nearest.confidence > 0.85) return nearest.route

  // 3. The small model, for the rest. ~22% of traffic.
  return routeByModel(question, history)
}

// Patterns are written for precision, not coverage. A pattern that
// matches 5% of traffic correctly is worth more than one that matches
// 40% at 0.8 — because the cascade catches what it declines.
const PATTERNS: Array<[RegExp, Route]> = [
  [/^(what|where|when|how) (is|are|do|does) .{0,60}(policy|procedure|rule|terms)\b/i,
   { kind: 'question' }],
  [/\b(how many|how much|stock|on hand|available)\b.{0,40}\bSKU-\d{4}\b/i,
   { kind: 'lookup', tool: 'check_inventory' }],
  [/\bwhy is\b.{0,40}\b(past due|overdue|on hold)\b/i,
   { kind: 'workflow', name: 'investigate_past_due' }],
]

The asymmetry that sets the threshold

Misroutes are not equally expensive, and the router's thresholds should reflect that.

// The cost matrix, in cost index. Rows = actual, columns = routed to.
//
//              question  lookup  workflow  planning
//  question      1.0      1.4      2.1       8.3    ⚠️ 8x for nothing
//  lookup        ~fail    1.4      2.1       8.3
//  workflow      ~fail    ~fail    2.1       8.3
//  planning      ~fail    ~fail    ~fail     8.3

Two readings.

Routing up is expensive but recoverable. A question sent to the loop costs eight times as much and produces a correct answer.

Routing down is cheap and usually fails. A planning question sent to retrieval produces a confident partial answer from the wrong path — and the user has no signal that a better path existed.

Bias the router upward on uncertainty. An unnecessary loop costs money; a missing one costs a wrong answer that looks right.

// Which makes the threshold asymmetric rather than a single confidence cut.
export function decide(scores: Record<RouteKind, number>): Route {
  const best = maxBy(entries(scores), ([, s]) => s)!
  const planning = scores.planning

  // ⚠️ Escalate to planning on a much lower bar than it would win on.
  // 0.30 against a best of 0.55 still routes to planning.
  if (planning > 0.30 && best[1] - planning < 0.30) return { kind: 'planning', complexity: 'normal' }
  return routeFor(best[0])
}

Recovering from a misroute

The cheap paths can detect their own insufficiency and escalate, which makes a downward misroute recoverable.

// Retrieval escalates when it cannot cover the question. One wasted
// retrieval (~200ms, no model call) is a cheap insurance premium.
export async function questionPath(question: string, scope: Scope): Promise<Answer> {
  const chunks = await retrieveAndRerank(question, { scope, limit: 6 })

  // ⚠️ Escalate on structural signals, not on "the answer looks thin".
  if (chunks.kind === 'no_relevant_results' && mentionsLiveData(question)) {
    return escalate({ kind: 'lookup' }, question, scope)
  }
  if (classifyHops(question) === 'sequential') {
    return escalate({ kind: 'planning', complexity: 'normal' }, question, scope)
  }
  return generate(question, chunks.results)
}

// And escalation is bounded: one hop, never a chain of them.
async function escalate(to: Route, question: string, scope: Scope, from?: Route) {
  if (from) throw new Error('double escalation')      // ⚠️ one hop only
  logger.info('route.escalated', { from: 'question', to: to.kind, question })
  return runRoute(to, question, scope, { escalatedFrom: 'question' })
}

The log line is what makes the router improvable: escalations are labelled training data for the next version of the classifier, free.

-- Which the router's own improvement loop consumes.
select question, escalated_from, route_taken, count(*)
  from route_log
 where escalated_from is not null and logged_at > now() - interval '30 days'
 group by 1,2,3 order by 4 desc limit 30;

The model classifier

<!-- src/generate/prompts/route.md — kept short, because it runs on
     every request that reaches it. -->
Classify this question into exactly one route.

question  — answerable from policy, specification or contract documents
lookup    — needs one live value: stock, order status, a discount tier
workflow  — a known investigation: why an account is past due
planning  — needs several lookups where later ones depend on earlier results

Rules:
- If it could be answered from documents, choose question.
- If it mentions current, live or today's data, it is not question.
- When torn between lookup and planning, choose planning.

Question: {{question}}
Recent turns: {{history}}
Answer with one word.

The third rule encodes the asymmetry in the prompt as well as in the thresholds, and it is the line that moved Atlas's planning recall from 0.84 to 0.93 without hurting the other routes.

// ⚠️ Constrained to the four words, so an unparseable route is impossible.
const route = await callModel({
  model: models.routing,
  messages: [{ role: 'user', content: prompt('route', { question, history }) }],
  maxTokens: 3,
  // Where available, constrain the output set rather than parsing it.
})

Measuring the router

// evals/routing.ts — no execution, deterministic given a fixed classifier
// temperature. Fast enough to run on every PR.
export async function scoreRouting(cases: RouteCase[]) {
  const rows = await Promise.all(cases.map(async (c) => ({
    id: c.id,
    expected: c.expect,
    actual: (await route(c.question, c.history ?? [])).kind,
  })))

  return {
    accuracy: mean(rows.map((r) => +(r.actual === r.expected))),
    // ⚠️ The confusion matrix matters more than the accuracy, because the
    // cells are not equally expensive.
    confusion: confusionMatrix(rows),
    // The cell that matters most: planning questions routed down.
    missedPlanning: rows.filter((r) => r.expected === 'planning' && r.actual !== 'planning'),
  }
}

routing — 180 cases

accuracy 0.96

confusion (expected -> actual):
  question -> lookup        3    cheap, recoverable
  question -> planning      2    ⚠️ 8x cost, correct answer
  lookup   -> question      1    recovered by escalation
  planning -> question      1    ⚠️ NOT recovered — wrong answer shipped

One cell is the problem. The planning -> question case escaped escalation because it had no sequential marker — "is the Henderson discount right?" reads like a question and needs two dependent lookups.

Adding that phrasing to the escalation signals fixed it, and the general lesson is that the confusion matrix should be read cell by cell against the cost matrix rather than summarised as accuracy.

The failure mode: the router that was improved into a cliff

Atlas's router reaches 0.96. Someone adds two routes — comparison and aggregate — on the reasonable grounds that those are distinct shapes needing distinct handling.

Accuracy falls to 0.89. Not because the new routes are bad, but because six classes is a harder problem than four, and the three nearest-neighbour boundaries that were previously unambiguous now have competitors.

Worse, the two new routes are each about 2% of traffic, so the 7 points of accuracy were paid to improve the handling of 4%.

// The rule that follows: a new route must justify itself against the
// accuracy it costs on the existing ones.
export async function evaluateNewRoute(candidate: Route, cases: RouteCase[]) {
  const before = await scoreRouting(cases)
  const after = await scoreRouting(cases, { routes: [...ROUTES, candidate] })

  const trafficShare = cases.filter((c) => c.expect === candidate.kind).length / cases.length
  const accuracyCost = before.accuracy - after.accuracy

  logger.info('route.evaluation', {
    candidate: candidate.kind, trafficShare, accuracyCost,
    // ⚠️ The honest comparison: gain on the new route's traffic against
    // loss across everything else.
    worthIt: trafficShare * 1.0 > accuracyCost,
  })
}

Atlas's two candidate routes were both handled instead as parameters on existing routes — { kind: 'planning', shape: 'comparison' } — which gave the specialised handling without adding a classification boundary.

Every route is a boundary that can be crossed wrongly. Prefer a parameter on an existing route to a new route, and make a new one prove it earns the accuracy it costs everywhere else.

Key points

Four routes, not more. Every route is a boundary that can be crossed wrongly.
Cascade from patterns to nearest neighbour to a small model. Patterns handled 34% of traffic at zero cost.
Write patterns for precision, not coverage. The cascade catches what they decline.
Misroutes are asymmetric: routing up costs money, routing down produces a wrong answer that looks right. Bias upward.
Let cheap paths escalate on structural signals, one hop only, and log every escalation — it is free labelled data.
Encode the asymmetry in the classifier prompt too. One rule moved planning recall from 0.84 to 0.93.
Read the confusion matrix cell by cell against the cost matrix. One cell — planning routed to question — is the only unrecoverable one.
A new route must earn the accuracy it costs on the existing ones. Two candidate routes became parameters instead.

Next: handling partial failure mid-task.