Chapter 59 · The model layer

Streaming responses, and what it changes upstream

Streaming halves perceived latency and breaks validation, because you have shown the reader text you have not checked. Four changes that make both work.

10 min read

Atlas's p50 is 1,774ms, inside the four-second target. Streaming does not make it faster — it makes it feel about three times faster, because the reader sees the first words at 600ms instead of a blank screen for two seconds.

It also breaks three things built earlier in this section. This chapter is about keeping both.

What streaming conflicts with

Built in	Assumes	Streaming gives
Validation	A complete response	Partial text, already displayed
Structured output	A parseable whole	A fragment of JSON
Citations	Resolved before display	`[2]` shown before the source list exists
Fallback	Replacing the answer	Half the answer is already read

The fourth is the real problem. A fallback that replaces an answer is clean when nothing was shown; replacing three sentences the reader has already read is not.

Change 1: order the output so the checkable parts come first

The single most effective change, and it is a prompt change rather than a code one.

<!-- Metadata BEFORE the prose. The decision whether to stream at all is
     then made from the first ~20 tokens. -->
<found>true</found>
<citations>1,2</citations>
<answer>
Damaged goods may be returned within 30 days of delivery [1]...
</answer>

// Buffer until the metadata is complete, validate it, then stream the prose.
// ~20 tokens of delay buys the ability to refuse before anything is shown.
export async function* streamAnswer(question: string, sources: Retrieved[]) {
  let buffer = ''
  let headerDone = false

  for await (const token of callModelStreaming(question, sources)) {
    buffer += token

    if (!headerDone) {
      const header = tryParseHeader(buffer)
      if (!header) continue                       // still buffering
      headerDone = true

      // ⚠️ Semantic validation on the metadata, BEFORE any prose is shown.
      // This is where chapter 57's citation-range check now lives.
      const issues = validateHeader(header, sources)
      if (issues.some(isFatal)) {
        yield { kind: 'fallback', answer: FALLBACK_ANSWER, issues }
        return
      }

      // Sources can be rendered now, so [1] is a live link from the first
      // moment it appears in the prose.
      yield { kind: 'sources', sources: header.citations.map((n) => sources[n - 1]) }
      if (!header.found) { yield { kind: 'abstention', text: ABSTENTION_TEXT }; return }

      buffer = buffer.slice(buffer.indexOf('<answer>') + 8)
    }

    yield { kind: 'token', text: token }
  }
}

This inverts the usual ordering instinct — people put the answer first because it is what matters — and it is why chapter 56 preferred delimiters over JSON. A JSON object with answer as the first key cannot do this.

Change 2: validate what you cannot pre-check, after the fact

Some checks need the finished text: inline-marker agreement, the figure check. Those run after streaming completes, and the response has to carry their result.

// Stream a terminal event, and let the interface annotate rather than
// replace. "This answer failed a check" below text already read is honest;
// deleting the text is worse.
const final = { kind: 'complete', text: full, warnings: postValidate(full, sources) }

// ⚠️ A post-hoc fatal issue cannot un-show the answer. So anything fatal
// must be checkable from the header — which is a constraint on schema
// design, not on the streaming code.
const FATAL_CHECKS = ['citation_out_of_range', 'abstained_but_cited'] as const
// Both are decidable from <found> and <citations> alone. Deliberately.

That constraint is the design consequence worth carrying: if a check must be able to stop an answer, its inputs have to be in the header. Anything else is an annotation.

Change 3: errors mid-stream

A stream can fail after 80% of an answer. The connection drops, the provider errors, a timeout fires.

export async function* streamWithRecovery(question: string, sources: Retrieved[]) {
  let emitted = ''
  try {
    for await (const event of streamAnswer(question, sources)) {
      if (event.kind === 'token') emitted += event.text
      yield event
    }
  } catch (error) {
    logger.error('stream.failed', { emittedChars: emitted.length, error: String(error) })

    // ⚠️ Never retry into the same stream. A retry restarts generation, and
    // the reader sees the answer begin again mid-sentence.
    yield {
      kind: 'interrupted',
      // Ending mid-sentence is worse than ending early. Trim back to the
      // last complete sentence, so what remains reads as finished.
      keep: trimToLastSentence(emitted),
      message: 'The answer was cut short. Please ask again.',
    }
  }
}

const trimToLastSentence = (text: string) => {
  const cut = Math.max(text.lastIndexOf('. '), text.lastIndexOf('.\n'))
  return cut > text.length * 0.3 ? text.slice(0, cut + 1) : text
}

Trimming to the last complete sentence is a small touch that changes how a failure reads: a truncated paragraph looks like a bug, and a short paragraph plus "the answer was cut short" looks like a system that noticed.

Change 4: the metrics change meaning

// Total latency stops being the number anyone experiences.
interface StreamMetrics {
  ttftMs: number          // ⚠️ the number that matters
  tokensPerSecond: number
  totalMs: number
  completed: boolean      // did it finish, or was it interrupted?
  abandonedAtMs?: number  // the reader navigated away mid-stream
}

-- p95 time-to-first-token is the SLO for a streaming interface. Total
-- duration is a cost metric, not an experience metric.
select percentile_cont(0.5) within group (order by ttft_ms) as p50_ttft,
       percentile_cont(0.95) within group (order by ttft_ms) as p95_ttft,
       round(100.0 * count(*) filter (where not completed) / count(*), 2) as interrupted_pct,
       round(100.0 * count(*) filter (where abandoned_at_ms is not null) / count(*), 2) as abandoned_pct
  from answer_log
 where answered_at > now() - interval '7 days';

Atlas's numbers after streaming:

	Before	After
p50 total	1,774 ms	1,810 ms
p50 time to first token	—	640 ms
Perceived wait	~1,800 ms	~640 ms
Abandoned mid-answer	—	3.1%

Total latency went up slightly — the header buffer costs about 35ms — and the experience improved by roughly a factor of three. The abandonment number is new information that did not exist before: 3% of readers get what they need from the first sentence and leave, which is a signal that answers could be shorter.

Where the first token actually comes from

Worth knowing, because the instinct is to optimise the model and the model is not the problem.

Stage	Contribution to TTFT
Embed the question	60 ms
Hybrid retrieval	14 ms
Rerank	200 ms
Model, prompt processing	~300 ms
Header buffer	35 ms
First prose token	~640 ms

The reranker is a third of it. For a system where TTFT is the SLO, that reframes chapter 44's trade — it saved total time by halving the context, and it costs perceived time by delaying the first token.

// Which suggests an optimisation that only makes sense once streaming
// exists: show the SOURCES before generation starts. They are known as
// soon as reranking finishes, 300ms before any prose.
yield { kind: 'sources', sources: window }     // at ~340ms
// ...then the prose at ~640ms.

Rendering the source list first gives the reader something real at 340ms, and it is the citation UI they would otherwise see last.

The failure mode: the answer that was shown and then contradicted

Atlas streams. A question about payment terms produces: "Payment is due within 30 days of invoice [1]." — fully rendered, read, and then the post-validation figure check fires because the cited source is the superseded clause and the amendment says 45.

The interface appends a warning. The reader has already read "30 days" and has probably already acted on it.

The fix is not better post-validation. It is that this check should never have been post-hoc: supersession is a property of the retrieved chunk, known before generation starts.

// ⚠️ Anything decidable before generation must be decided before generation.
// Streaming makes this a correctness requirement rather than a preference.
export function preflightSources(sources: Retrieved[]): PreflightResult {
  const superseded = sources.filter((s) => s.supersededBy)
  if (superseded.length) {
    // Resolve it: swap in the current text, or drop the chunk. Do not pass
    // a known-superseded source to a model whose output you will stream.
    return { kind: 'resolve', superseded }
  }
  const conflicts = detectConflicts(sources)
  if (conflicts.length) return { kind: 'flag_conflict', conflicts }
  return { kind: 'ok' }
}

The general rule, and it is the one streaming forces you to take seriously:

Streaming converts every post-hoc check into an apology. Move every check you can ahead of the first token, and accept that whatever remains can only annotate.

That pressure is useful. Working through which checks can move forward found three in Atlas's pipeline — supersession, conflict detection, citation range — that were post-hoc only because nothing had forced the question.

Key points

Streaming does not reduce latency; it reduces perceived latency by about three times.
Put metadata before prose so the fatal checks run on ~20 buffered tokens, before anything is shown. JSON with answer first cannot do this.
If a check must be able to stop an answer, its inputs belong in the header. Everything else can only annotate.
Never retry into an open stream. Trim to the last complete sentence and say it was cut short.
Time to first token becomes the SLO. Total duration becomes a cost metric.
Abandonment mid-stream is new information: 3% leaving after the first sentence means answers could be shorter.
The reranker is a third of TTFT. Render the source list at 340ms, before generation starts.
Streaming turns every post-hoc check into an apology — which is a useful forcing function for moving checks earlier.

Next: token budgets and context assembly — deciding what fits when everything wants space.