Chapter 52 · The model layer

Context windows, and why more is not the answer

A million-token window does not make retrieval optional. What actually degrades as context grows, measured, and the three cases where a long window genuinely helps.

10 min read

The most common proposal when a retrieval system gives a wrong answer: the context window is huge now — why not put everything in?

It is a reasonable question and the answer is specific. This chapter measures what happens as context grows, and names the three cases where a long window is genuinely the right tool.

What a large window does not change

Atlas's corpus is 44,387 chunks at roughly 420 tokens each: about 18.6 million tokens. A one-million-token window holds 5% of it.

// The arithmetic that ends most versions of this discussion.
const corpusTokens = 44_387 * 420        // 18,642,540
const windowTokens = 1_000_000
console.log(`${((windowTokens / corpusTokens) * 100).toFixed(1)}% of the corpus`)  // 5.4%

So "put everything in" is not available, and the question becomes which 5% — which is retrieval. A larger window changes how much retrieval returns, not whether it is needed.

Three further things it does not change:

Permissions. A context assembled without the permission filter is a leak regardless of its size. The filter is a where clause, not a budget.

Freshness. Stale chunks are stale at any volume.

Cost. Input tokens are billed. A 100,000-token context at 800 questions a day is forty times the bill of a 2,520-token one for the same answers.

What degrades as context grows

Three effects, and they are distinguishable.

Position sensitivity

Information in the middle of a long context is used less reliably than information at either end. The effect is well documented, it varies by model, and it is measurable on your own task in an afternoon.

// scripts/position-test.ts
// Put the answer at a known position among N distractors, vary the position,
// measure correctness. Run it on YOUR model and YOUR prompt.
export async function positionSensitivity(model: string, distractorCount: number) {
  const answer = SUPPORTING_CHUNK
  const distractors = sampleDistractors(distractorCount)
  const results: Array<{ position: number; correct: number }> = []

  for (const position of [0, 0.25, 0.5, 0.75, 1]) {
    const index = Math.round(position * distractorCount)
    const context = [...distractors.slice(0, index), answer, ...distractors.slice(index)]
    const scores = await Promise.all(
      QUESTIONS.map(async (q) => +((await generate(q, context, { model })).text.includes(EXPECTED))),
    )
    results.push({ position, correct: mean(scores) })
  }
  return results
}

Atlas's measured result, mid-tier model:

Chunks in context	First	25%	Middle	75%	Last
6	1.00	1.00	1.00	1.00	1.00
20	1.00	0.98	0.95	0.98	1.00
60	0.98	0.92	0.84	0.91	0.97
200	0.95	0.81	0.68	0.79	0.94

At six chunks there is no effect at all. At 200 chunks, a passage in the middle is found two thirds of the time.

⚠️ The six-chunk row is the important one. The entire position-sensitivity literature is irrelevant to a system that sends six well-ranked passages, which is another reason reranking earned its place.

Distractor interference

Separate from position, and the effect that matters more at moderate sizes.

// Hold position constant, vary how many NEAR-MISS chunks accompany the answer.
// Random distractors are easy; topically adjacent ones are not.
const nearMiss = await retrieveAndRerank(question, { limit: 60 })
  .then((r) => r.results.filter((c) => !supporting.has(c.id)))

Distractors	Random	Topically adjacent
5	1.00	0.97
20	0.99	0.89
60	0.97	0.78

Random distractors barely matter. Adjacent ones — the late-returns clause beside the damaged-goods clause — degrade sharply, and those are exactly what deep retrieval returns.

Adding more retrieved context adds more near-misses, which is the expensive kind of distractor. Recall bought by depth is partly paid for in precision at the model.

Instruction dilution

The third effect, and the one that surprises people: as context grows, the system prompt's instructions are followed less reliably.

// Citation discipline against context size, same prompt throughout.
// 6 chunks:    citations valid 0.99
// 20 chunks:   citations valid 0.97
// 60 chunks:   citations valid 0.91
// 200 chunks:  citations valid 0.78

A fifth of answers citing a non-existent source at 200 chunks is a worse failure than a missing answer, for the reason chapter 51 gave: it looks verified.

The three cases where a long window helps

Not an argument against long context — an argument against using it as a substitute for retrieval. Three places it is the right tool.

1. A whole document that must be read as one

A contract where clause 7.3 qualifies 7.2 is genuinely a whole-document question. Chunking cannot solve it and neighbour expansion only partly can.

// Escalate to the whole document when retrieval keeps hitting the same one.
export async function maybeWholeDocument(results: Retrieved[], budget: number) {
  const byDoc = groupBy(results, (r) => r.documentId)
  const [docId, chunks] = maxBy(Object.entries(byDoc), ([, c]) => c.length)!

  // Three or more chunks from one document, and the document fits: send it
  // whole. Cross-clause references then resolve themselves.
  if (chunks.length >= 3) {
    const size = await documentTokens(docId)
    if (size <= budget) return { kind: 'whole_document', docId, tokens: size }
  }
  return { kind: 'chunks' }
}

2. Conversation history

A long session accumulates turns, and summarising them loses detail. A long window lets you keep them verbatim — which is section 8's subject.

3. Few-shot examples for a structured task

Twenty worked examples of a difficult output format cost tokens and are more reliable than a longer instruction. Chapter 55 covers when that trade is worth it.

Choosing the window size

// Measured, like everything else. The knee is where correctness stops
// rising and cost keeps going.
for (const k of [3, 4, 6, 8, 12, 20]) {
  const scored = await scoreEndToEnd(cases, { rerankKeep: k })
  console.log(k, round(scored.correct), round(scored.grounded), scored.tokensIn)
}

Chunks	Correct	Grounded	Input tokens	Cost index
3	0.86	0.99	1,260	0.5
4	0.89	0.99	1,680	0.67
6	0.91	0.98	2,520	1.0
8	0.91	0.97	3,360	1.33
12	0.90	0.95	5,040	2.0
20	0.87	0.91	8,400	3.3

Correctness peaks at six and falls beyond eight. Groundedness falls monotonically. Adding context past the knee makes the system worse and more expensive at the same time, which is the single most useful fact in this chapter.

⚠️ Six is Atlas's number, with a reranker in front of it. Without reranking the knee is higher, because more chunks are needed to catch the answer — the two parameters are coupled and must be tuned together.

The failure mode: the window that grew by accident

Nobody decides to send 200 chunks. The context grows one addition at a time, and each addition is justified.

// ❌ How it actually happens. Each line was a reasonable commit.
const context = [
  ...retrievedChunks,              // 6
  ...conversationHistory,          // 12 turns by the afternoon
  ...userPreferences,              // added for personalisation
  ...recentDocuments,              // "helpful context"
  ...toolDefinitions,              // 14 tools, section 6
  ...examples,                     // few-shot, chapter 55
]
// 2,520 tokens at launch. 31,000 tokens four months later, and the
// groundedness number has been drifting down the whole time.

// ✅ A budget, enforced, with priorities. Chapter 60 builds this properly;
// the assertion is what matters now.
const BUDGET = { total: 6_000, retrieved: 3_000, history: 1_500, tools: 1_000, system: 500 }

export function assembleContext(parts: ContextParts): string {
  const assembled = buildWithinBudget(parts, BUDGET)

  // ⚠️ Fail loudly rather than truncating silently. A context that quietly
  // dropped the retrieved chunks to fit the history is the worst outcome
  // and is invisible in the answer.
  if (assembled.droppedCategories.includes('retrieved')) {
    throw new Error('context budget dropped retrieved chunks — history is too long')
  }
  return assembled.text
}

And the regression test that catches the drift, which is the real defence:

test('context size has not grown', async () => {
  const sizes = await Promise.all(
    SAMPLE_CASES.map((c) => measureContextTokens(c)),
  )
  // A committed number, raised deliberately with a reason in the commit.
  expect(percentile(sizes, 0.95)).toBeLessThan(6_500)
})

Context size is a budget, not a limit. A limit is what the model allows; a budget is what you have decided the answer is worth, and only one of the two gets smaller when you are not looking.

Key points

A million-token window holds 5% of Atlas's corpus. "Put everything in" is not available; the question is still which chunks.
A large window changes nothing about permissions, freshness or cost.
Three distinct degradations: position sensitivity, distractor interference, instruction dilution. All are negligible at six chunks.
Topically adjacent distractors hurt far more than random ones, and deep retrieval returns exactly those.
Citation validity falls from 0.99 to 0.78 between 6 and 200 chunks. That failure looks verified.
Correctness peaks at six chunks and falls beyond eight. Past the knee, more context is worse and more expensive.
The window size and the reranker's keep are coupled. Tune them together.
Long context is right for a whole document that must be read as one, verbatim conversation history, and few-shot examples.
Contexts grow by accident, one justified addition at a time. Enforce a budget, fail loudly on dropped retrieval, and regression-test the size.

Next: the system prompt as an engineering artifact — versioned, reviewed, and tested like code.