Chapter 61 · The model layer

What to put in context and what to leave out

The budget is allocated; now spend it. How to render a chunk, what metadata earns its tokens, and the four things that look helpful and make answers worse.

10 min read

Chapter 60 decided how much space each claimant gets. This chapter is about what goes in the 3,000 tokens allocated to retrieved content.

The question is narrower than it sounds, and it has a measurable answer: every token in context either helps the model answer or competes with the tokens that do.

Rendering a chunk

Six chunks, roughly 420 tokens each, and the wrapper around each one costs about 25. That wrapper is worth getting right.

// src/generate/render.ts
export function renderSource(chunk: Retrieved, n: number): string {
  const parts = [`[${n}] ${chunk.title}`]

  // The heading path: ~8 tokens, and it is what makes a fragment
  // interpretable. Chapter 32 measured its effect on retrieval; it does
  // the same job again here, for the model rather than the index.
  if (chunk.headingPath.length) parts.push(` — ${chunk.headingPath.join(' > ')}`)

  // Freshness, only when it is actually stale. An "updated 2026-01-14" on
  // every source trains the reader to ignore all of them.
  const stale = freshnessNote(chunk, chunk.source)
  if (stale) parts.push(` (${stale})`)

  // Parse warnings, same rule: only when present. Chapter 27's OCR caveat.
  if (chunk.parseWarnings.length) parts.push(` ⚠️ ${chunk.parseWarnings[0]}`)

  return `${parts.join('')}\n${chunk.content}`
}

[1] Returns and Credits Policy — Returns > Damaged goods
Damaged goods may be returned within 30 days of delivery, provided the damage
is reported within 48 hours of receipt...

[2] SKU-4411 Specification — Availability (last updated 2025-08-02) ⚠️ OCR confidence 0.86
Standard lead time is 6 weeks from order confirmation...

Conditional metadata is the rule throughout. A field that appears on every source is noise the model learns to skip; a field that appears on one is a signal.

What earns its tokens

Measured by ablation: remove it, score the test set, see what happens.

Included	Tokens	Δ correct	Verdict
Heading path	8	+0.06	Keep
Document title	5	+0.03	Keep
Freshness, when stale	7	+0.02	Keep
Parse warning, when present	9	+0.01	Keep
Source system name	4	0.00	Drop
Chunk ordinal	3	0.00	Drop
Retrieval score	4	−0.02	Drop
Last-modified on every source	7	−0.01	Drop

Two negative rows are worth explaining, because both are things people add deliberately.

The retrieval score hurts. Shown score: 0.61, the model treats the number as a confidence signal and defers to the top-ranked chunk even when a lower-ranked one answers the question. It converts a ranking into an instruction.

A date on every source hurts. With dates everywhere the model starts preferring recency over relevance, and on a corpus where the authoritative policy is five years old and unchanged, that is exactly backwards.

The model will use any field you give it, including ones you meant as debugging information. A field in context is an instruction whether or not you intended one.

Ordering the sources

// Rank order, best first. Three alternatives were measured and all were
// worse or equal.
const ordered = window   // already sorted by rerank score

Ordering	Correct	Note
Rank, best first	0.91	—
Rank, best last	0.90	Recency effect; no real gain at six chunks
Grouped by document	0.91	Equal, and better when chunks are adjacent
Chronological	0.87	Ignores relevance entirely

Grouping by document ties on the aggregate and wins on the multi-chunk subset, where two chunks from one document answer the question together.

// So: group by document, order groups by their best chunk, order within a
// group by ordinal. Adjacent chunks then read as continuous prose.
export function orderSources(chunks: Retrieved[]): Retrieved[] {
  const groups = [...groupBy(chunks, (c) => c.documentId).values()]
  groups.sort((a, b) => Math.max(...b.map(s)) - Math.max(...a.map(s)))
  return groups.flatMap((g) => g.sort((x, y) => x.ordinal - y.ordinal))
  function s(c: Retrieved) { return c.rerankScore ?? c.score }
}

The four things to leave out

The question, repeated

<!-- ❌ The question appears three times: in the instruction, before the
     sources, and after them. It is already last, next to generation. -->

Measured at −0.01 and 40 wasted tokens. It feels like emphasis and reads as padding.

Instructions about the sources, inside the sources

// ❌ Per-source guidance. The model now has six instructions competing
//    with the six rules in the system prompt.
`[1] ${title}\nNOTE: this is the most relevant source, prefer it.\n${content}`

Chunks below the relevance floor

// ❌ "It might help." It will not, and it is a near-miss distractor, which
//    chapter 52 measured as the expensive kind.
const context = [...relevant, ...marginal.slice(0, 3)]

If a chunk is below the floor, sending it is strictly negative: it cannot be the answer and it can be mistaken for it.

Anything the model must not act on

// ❌ The worst version. Sending forbidden content with an instruction not
//    to use it is the prompt-as-permission pattern chapter 13 rejected,
//    and it also puts the content in your logs and traces.
const context = [...permitted, ...forbidden.map((c) => `[RESTRICTED] ${c.content}`)]

Whole-document escalation

Chapter 52 named this as one of three legitimate uses of a long window. Here is the trigger.

// Three or more chunks from one document, and the document fits: send it
// whole. Cross-clause references then resolve without any special handling.
export async function maybeEscalate(chunks: Retrieved[], budget: number) {
  const byDoc = groupBy(chunks, (c) => c.documentId)
  const [docId, group] = maxBy([...byDoc.entries()], ([, g]) => g.length)!
  if (group.length < 3) return { kind: 'chunks' as const, chunks }

  const tokens = await documentTokens(docId)
  if (tokens > budget * 0.7) return { kind: 'chunks' as const, chunks }

  // ⚠️ Keep the other documents' chunks. Escalating to one whole document
  // must not drop the evidence that came from elsewhere.
  const others = chunks.filter((c) => c.documentId !== docId)
  return { kind: 'whole_document' as const, docId, tokens, others }
}

Measured on Atlas's contract questions — the clause 7.2 / 7.3 case — this moved correctness from 0.84 to 0.95 on that subset, and it fires on 4% of questions.

The assembled prompt

export function buildAnswerPrompt(prepared: PreparedQuery, window: Window): string {
  return [
    SYSTEM_PROMPT,                                  // cached prefix
    EXAMPLES,                                       // cached prefix
    // ---- cache boundary, per chapter 60 ----
    renderHistory(window.history),
    'Sources:',
    orderSources(window.chunks).map(renderSource).join('\n\n'),
    // ⚠️ The ORIGINAL question, not the rewritten one. Chapter 46.
    `Question: ${prepared.original}`,
  ].filter(Boolean).join('\n\n')
}

Every line in that function is a decision from an earlier chapter, and the comment on the last one is the one most easily lost in a refactor.

The failure mode: the helpful summary

Atlas's context is improved with a one-line summary above each source, generated at ingest: "This section covers returns of damaged goods within the standard warranty period."

It reads well. Measured, correctness fell from 0.91 to 0.86.

Two causes, and they compound.

The summary is lossy and authoritative-looking. Where it says "within the standard warranty period" and the clause says "within 30 days of delivery", the model sometimes answers from the summary — and the summary is a paraphrase of a paraphrase.

It goes stale invisibly. It was generated at ingest from the chunk text, and when the clause is amended the chunk is re-ingested and re-summarised — but the content hash covers the chunk, so a summary generated from a previous parser version persists against unchanged content. Chapter 33 listed ingest-time summaries as metadata not to attach, and this is why.

// ❌ Derived text in context, competing with the source it was derived from.
`[${n}] ${title}\nSummary: ${chunk.summary}\n${chunk.content}`

// ✅ Only the source. If the model needs orientation, the heading path
//    gives it in 8 tokens and cannot disagree with the content.
`[${n}] ${title} — ${chunk.headingPath.join(' > ')}\n${chunk.content}`

The general rule this produces is worth more than the specific finding:

Never put derived text beside the text it was derived from. The model has no way to know which is authoritative, and the derived version is shorter, so it often wins.

That rule also rules out: a model-generated title, an extracted "key fact", a highlighted sentence, and a relevance explanation. All four have been tried on this corpus and all four cost correctness.

Key points

Every token in context either helps or competes. Measure by ablation rather than reasoning about it.
Make metadata conditional. A field on every source is noise; a field on one is a signal.
The heading path is the highest-value addition at eight tokens: +0.06 correctness.
Retrieval scores and per-source dates both hurt. The model reads a score as a confidence instruction and dates as a recency preference.
Group sources by document, order groups by best chunk, order within a group by ordinal.
Leave out: the repeated question, per-source instructions, chunks below the floor, and anything the model must not act on.
Escalate to a whole document when three chunks come from it and it fits. +0.11 on the contract subset, firing on 4% of questions.
Send the original question, not the rewritten one.
Never put derived text beside its source. Summaries, extracted facts and highlighted sentences all cost correctness.

Next: citations — making an answer checkable, which is what makes it trustworthy.