Chapter 31 · Ingestion

Chunk size, overlap, and measuring instead of guessing

Everyone quotes 512 tokens with 10% overlap. Here is the experiment that takes an afternoon and tells you the number for your corpus, plus what overlap actually buys.

11 min read

The previous chapter chose a strategy per source. This one sets the numbers, and it is the first chapter in the series that is an experiment rather than a design.

The received wisdom is 512 tokens with 10–20% overlap. It is a reasonable starting point and it is somebody else's answer, measured on somebody else's corpus. The experiment below takes an afternoon and produces yours.

What you need before starting

The retrieval test set from chapter 18: questions keyed to document and heading, with distractors. That keying is what makes this experiment possible at all — a test set keyed to chunk IDs is invalidated by the very thing being varied.

# evals/retrieval.yaml — 34 cases, which is enough to see a real effect and
# not enough to see a small one. Be honest about that in the conclusions.
- question_id: ret-001
  question: "How long do customers have to return damaged goods?"
  supporting:
    - document: "Returns and Credits Policy"
      heading: ["Returns", "Damaged goods"]
  distractors:
    - document: "Returns and Credits Policy"
      heading: ["Returns", "Late returns"]

The three metrics

Not one number. The three answer different questions and they move in different directions.

// src/evals/retrieval-metrics.ts

/** Did the supporting chunk appear anywhere in the top k?
 *  The ceiling on everything downstream: if it is not retrieved, nothing
 *  later can recover it. */
export function recallAtK(results: Retrieved[], supporting: Set<number>, k: number) {
  return results.slice(0, k).some((r) => supporting.has(r.id)) ? 1 : 0
}

/** How far down was it? 1 / rank, averaged. Distinguishes "first" from
 *  "eleventh", which recall@12 does not. */
export function reciprocalRank(results: Retrieved[], supporting: Set<number>) {
  const i = results.findIndex((r) => supporting.has(r.id))
  return i === -1 ? 0 : 1 / (i + 1)
}

/** How much of the retrieved context was useful? Recall can be bought by
 *  retrieving more, and this is the price. */
export function precisionAtK(results: Retrieved[], supporting: Set<number>, k: number) {
  const top = results.slice(0, k)
  return top.filter((r) => supporting.has(r.id)).length / top.length
}

Recall is the one that matters most, and it is also the one that is trivially gamed by raising k. Report all three or the experiment lies to you.

The experiment

// scripts/experiment-chunking.ts
// A grid over size and overlap. Each cell re-chunks, re-embeds and re-scores
// the whole corpus, so run it on a sample.
const SIZES = [256, 512, 800, 1200, 2000]
const OVERLAPS = [0, 0.1, 0.25]

export async function run() {
  // ⚠️ Sample the corpus, not the test set. 300 documents including every
  // document the test set references keeps a cell at ~2 minutes.
  const corpus = await sampleCorpus({ documents: 300, mustInclude: testSetDocuments() })

  const results: Cell[] = []
  for (const size of SIZES) {
    for (const overlap of OVERLAPS) {
      await rebuildIndex(corpus, { size, overlap })          // into a scratch schema
      const scored = await scoreTestSet({ k: 12 })
      results.push({
        size, overlap,
        recall12: mean(scored.map((s) => s.recall12)),
        mrr: mean(scored.map((s) => s.rr)),
        precision12: mean(scored.map((s) => s.precision12)),
        chunks: await countChunks(),
        tokensPerQuery: await meanContextTokens(),
      })
      console.table(results)
    }
  }
}

Two practical notes that save an afternoon.

Build into a scratch schema, not the live one. search_path makes this a one-line change and it means the experiment cannot leave the index in a half-rebuilt state if it is interrupted.

Include chunks and tokensPerQuery in the output. Those are the cost columns, and a configuration that buys two points of recall for 60% more tokens per query is a decision, not a win.

A representative result

From Atlas's policy and specification corpus, structural strategy, 300 documents:

Size	Overlap	Recall@12	MRR	Precision@12	Chunks	Tokens/query
256	0	0.79	0.41	0.09	18,400	3,100
256	0.25	0.85	0.44	0.09	23,900	3,100
512	0	0.88	0.58	0.11	9,600	6,200
512	0.1	0.91	0.61	0.11	10,400	6,200
800	0.1	0.91	0.57	0.10	6,800	9,700
1200	0.1	0.88	0.49	0.08	4,700	14,500
2000	0.1	0.82	0.38	0.06	2,900	24,100

Four things to read out of a table like this, and the fourth is the one people miss.

Recall peaks in the middle. Small chunks fragment the answer; large ones dilute the vector, exactly as chapter 30 predicted.
MRR peaks earlier and falls faster than recall. 800 matches 512 on recall and is clearly worse on rank. If you are sending 12 chunks to the model this matters less; if you are sending 5, it is the deciding column.
Tokens per query scale linearly with size while recall does not. 2000-token chunks cost 3.9× the tokens of 512 for worse retrieval — the clearest possible argument against "just use big chunks, the context window is huge".
Precision is low everywhere. 0.11 at k=12 means roughly one useful chunk in nine. That is normal, and it is the number that justifies reranking in chapter 44 — the win there is precision, and this table shows the headroom.

What overlap actually buys

Overlap exists for one reason: a boundary falling mid-answer. It helps measurably and less than people expect.

// Overlap at the block level, not the token level. Repeating half a sentence
// helps nothing; repeating the preceding paragraph helps.
export function withOverlap(chunks: Chunk[], ratio: number): Chunk[] {
  if (ratio <= 0) return chunks
  return chunks.map((chunk, i) => {
    if (i === 0) return chunk
    const previous = chunks[i - 1]
    const take = Math.floor(previous.tokens * ratio)
    const tail = lastTokens(previous.content, take, { snapToSentence: true })
    return {
      ...chunk,
      // ⚠️ Overlap goes into embedText and into content, but the chunk's
      // identity stays its own text. Otherwise dedupe at retrieval breaks.
      content: `${tail}\n\n${chunk.content}`,
      embedText: `${chunk.headingPath.join(' > ')}\n\n${tail}\n\n${chunk.content}`,
      tokens: chunk.tokens + take,
    }
  })
}

Three observations from the table and from using it:

0 to 10% is the whole effect. 0.88 to 0.91 recall. Worth taking.
10% to 25% buys nothing at 512 tokens and costs 10% more chunks and storage. At 256 it does help, because at that size boundaries land mid-answer far more often — which is a reason to use 512, not a reason to use 25% overlap.
Overlap partly substitutes for good boundaries. The gain is largest with fixed-size chunking and smallest with structural, because structural boundaries are already where the author put them. If overlap is buying you a lot, that is evidence your splitting is in the wrong place.

A large overlap benefit is a diagnostic, not a win. It means the chunker is cutting through answers often enough that patching the cuts helps.

Per-source numbers, not one global number

The experiment is per source, because the sources are different shapes. Running it once on the policy corpus and applying the result to spreadsheet rows is how the registry ends up with a number that was never measured for the source it governs.

// The outcome lands in the registry, with the date it was measured.
{
  id: 'policy_docs',
  chunkStrategy: 'structural',
  targetTokens: 512,
  overlapRatio: 0.1,
  // ⚠️ Measured, not chosen. Re-run when the embedding model changes.
  chunkingMeasuredOn: '2026-11-03',
  chunkingRecall12: 0.91,
}

Recording the measured recall beside the parameters is worth the two lines. It is the baseline that a later change has to beat, and without it the next person to touch this is guessing again.

When to re-run it

Three triggers, and only three:

Trigger	Why
The embedding model changes	Different models have different optimal granularity. This is the big one.
A new source shape is added	Its documents are not like the ones you measured on.
The corpus grows by an order of magnitude	More near-neighbours makes discrimination harder at every size.

Not: a bad answer. A bad answer gets classified by chapter 10's three buckets first, and only retrieval:absent across many cases points back here.

The failure mode: tuning on the questions you tuned with

The experiment has 34 cases. Running a grid of 15 configurations against 34 cases and picking the best is selecting on noise as much as on signal — with 34 binary outcomes, the difference between 0.88 and 0.91 is one question.

This is the chunking-specific form of overfitting, and the mitigation is standard and usually skipped.

// Hold out from the start. Tune on train, report on test, once.
const { train, test } = splitTestSet(cases, { trainRatio: 0.7, seed: 42 })

const best = gridSearch(train)                    // 24 cases
const confirmed = await scoreTestSet(test, best)  // 10 cases, scored ONCE

// ⚠️ If the held-out score is much worse than the training score, the winner
// was noise. Prefer the simplest configuration within the noise band instead.
if (confirmed.recall12 < best.recall12 - 0.1) {
  logger.warn('chunking.overfit', { train: best.recall12, test: confirmed.recall12 })
}

And the practical rule that matters more than the statistics:

Prefer the simplest configuration that is within the noise band of the best one. If 512/0 scores 0.88 and 512/0.1 scores 0.91 on 34 cases, that is one question's difference, and the simpler configuration has fewer chunks, lower storage and no overlap logic to be wrong. Take the complexity only when the gap is clearly larger than one or two cases.

The honest summary of the Atlas table above is: 512 tokens, structural, overlap somewhere between 0 and 10%, and anything above 1200 is clearly worse. That last clause is the finding with real confidence, and it is the one that saves money.

Key points

Measure on your corpus. 512/10% is a reasonable prior and somebody else's measurement.
Report recall@k, MRR and precision@k together. Recall alone is gamed by raising k.
Include chunk count and tokens-per-query. They are the cost of every recall point.
Sample 300 documents including every one the test set references. Build into a scratch schema.
Recall peaks in the middle; MRR peaks earlier and falls faster. Large chunks cost linearly and retrieve worse.
Low precision everywhere is normal and is the headroom reranking will use.
Overlap's whole effect is in the first 10%, at the block level. A large overlap benefit means your boundaries are wrong.
Measure per source and record the result and date in the registry beside the parameters.
Hold out 30% of cases. With 34 cases, three points of recall is one question — prefer the simplest configuration inside the noise band.

Next: keeping document structure through the chunker — the heading path, and why a chunk has to know where it came from.