Chapter 133 · Production

Latency budgets and where the time goes

A measured breakdown of 2.9 seconds, the three thresholds that change user behaviour, why time to first token matters more than total time, and the tail that is a different problem.

7 min read

Chapter 59 streamed for perceived latency. This chapter is the actual latency, measured, and the budget that decides which parts are worth attacking.

Where 2.9 seconds goes

Atlas, p50, instrumented from chapter 128's span tree.

Span	p50	p95	Share of p50
resolve identity	12 ms	31 ms	0.4%
rewrite query	180 ms	420 ms	6.2%
embed query	22 ms	48 ms	0.8%
vector search	31 ms	180 ms	1.1%
rerank	8 ms	22 ms	0.3%
assemble context	4 ms	9 ms	0.1%
generate (step 1)	1,100 ms	3,400 ms	37.9%
tool calls	290 ms	2,100 ms	10.0%
generate (step 2)	1,000 ms	2,900 ms	34.5%
validate	24 ms	61 ms	0.8%
total	2,900 ms	10,400 ms

Generation is 72% of p50 and the whole retrieval stack is 2.3%. That ratio is the single most useful thing in this chapter, because retrieval optimisation is where engineering instinct goes — it is the part that looks like a database problem, and it is 67 milliseconds.

// Which means the latency levers are, in order of effect:
const LEVERS = [
  'Fewer generation steps.',      // chapter 131's workflow promotion
  'A faster model.',              // chapter 131's routing: p50 1.9 s -> 1.4 s
  'Fewer output tokens.',         // output length is the strongest predictor
  'Parallel tool calls.',         // chapter 88
  'Everything else.',             // ~3% combined
]

Output length dominates generation time

// Time to generate is roughly linear in output tokens and barely
// sensitive to input tokens, because input is processed in parallel and
// output is produced one token at a time. Atlas, measured:
const GENERATION_MS = {
  perInputToken: 0.014,     // 6,000 input tokens -> ~84 ms
  perOutputToken: 3.6,      // 284 output tokens -> ~1,020 ms
  fixedOverhead: 180,
}

A 257× difference per token. That reframes several decisions made earlier for other reasons:

// Chapter 109's model upgrade wrote 35% more output for the same
// questions, which was ~90 ms of latency nobody attributed to it.
// An explicit length target took output 284 -> 196 tokens and removed
// ~320 ms, which is larger than the entire retrieval stack.

Input tokens cost money. Output tokens cost money and time. The two optimisations are not the same, and the second one is the one users feel.

Three thresholds

// Measured against Atlas's own behaviour data rather than taken from a
// general usability source — the numbers differ by context, and an
// internal tool with no alternative has more tolerance than a consumer
// product.
const THRESHOLDS = {
  200:   'feels immediate. Only reachable for a cache hit.',
  1000:  'attention holds. No visible waiting behaviour.',
  4000:  'tab switching begins. Measured: 31% of sessions over 4 s had a focus loss.',
  10000: 'abandonment. 12% of runs over 10 s were abandoned before completion.',
}

The four-second number is the one that shaped Atlas's design. Past it, a third of users leave the tab, and a user who has left the tab does not read the answer carefully — which showed up as a measurable fall in citation expansion on slow answers.

Time to first token

// The metric that correlates with satisfaction, and it is not total
// time. Atlas, before and after streaming was added:
const TTFT = {
  before: { ttft: 2900, total: 2900, satisfaction: 71 },
  after:  { ttft: 640,  total: 3100, satisfaction: 83 },
}

Total time got worse by 200 ms — streaming adds overhead — and satisfaction rose twelve points. The budget is therefore written against TTFT rather than total:

const BUDGET = {
  ttft:  { p50: 800, p95: 2000 },
  total: { p50: 3000, p95: 10000 },
}

// And the design consequence: everything before the first token is on
// the critical path for the metric that matters, which makes the 180 ms
// query rewrite the most expensive non-generation span in the system.
// Atlas made it conditional rather than unconditional:
const needsRewrite = hasPronoun(question) || question.split(/\s+/).length < 5
// Fires on 34% of questions. Saves 180 ms on the other 66%, and the
// eval suite showed retrieval recall unchanged at 0.94.

That is the highest-return latency change in the chapter and it came from asking which spans sit before the first token rather than which spans are slow.

The tail is a different problem

// p99 is 31 seconds against a p50 of 2.9. The causes are not "the same
// thing, slower" — they are categorically different.
const TAIL_CAUSES = {
  multiStep: '58% — 6+ step runs. Chapter 83 is the fix, not tuning.',
  toolBackend: '22% — a slow internal API. Not ours to optimise.',
  providerVariance: '14% — occasional 8s generations with no local cause.',
  retrievalOnLargeFilter: '6% — a user with very broad group membership.',
}

// Which means tail work is capability work, and the honest handling of
// the part that is not ours is to show it.
if (elapsed > 4000) ui.show('Still working — checking the orders system.')

Naming the slow dependency in the progress message moved abandonment on runs over 10 seconds from 12% to 4%. The run did not get faster; the user stopped assuming it had hung. That is the cheapest latency improvement available and it is not a latency improvement.

A budget that is enforced

// Chapter 81 set per-run time limits for control. The same number also
// shapes the answer, by degrading rather than failing.
const DEADLINE = 25_000

async function withDeadline(run: Run) {
  const remaining = () => DEADLINE - run.elapsed()
  // Each step receives the remaining budget and plans within it.
  if (remaining() < 6_000) run.hint('Answer now with what you have; do not call further tools.')
  if (remaining() < 2_000) return run.finishFromContext()
}

The hint at six seconds is what produces a useful partial answer rather than a timeout. Chapter 90 listed failing to thread the remaining budget into generation as one of the seven production bugs, and this is the positive version of that fix.

Key points

Generation is 72% of p50; the entire retrieval stack is 2.3%. Retrieval optimisation is where instinct goes and it is 67 milliseconds.
Output tokens cost 3.6 ms each; input tokens cost 0.014 ms. A 257× difference, so length targets are a latency lever and context size is not.
An explicit output length target removed 320 ms — more than the whole retrieval stack.
Three thresholds: 1 s holds attention, 4 s starts tab switching (31% focus loss), 10 s starts abandonment (12%).
Budget against time to first token, not total. Streaming made total 200 ms worse and satisfaction twelve points better.
Everything before the first token is on the critical path. Making query rewriting conditional saved 180 ms on 66% of questions with recall unchanged.
The tail is categorically different: 58% is multi-step runs, 22% a slow backend, 14% provider variance. Tail work is capability work.
Naming the slow dependency in a progress message cut abandonment on 10-second runs from 12% to 4%, with no speed change.
Thread the remaining deadline into every step, and hint the model to finish from context before the budget is gone.

Next: rate limits, retries and backoff against a provider — where a lot of the tail actually comes from.