Chapter 30 · Ingestion

Chunking: the decision that quietly determines quality

A chunk is the unit you retrieve, so it bounds what a search can possibly find. Four strategies, what each one is good at, and why the right answer is rarely one of them alone.

12 min read

Chapter 6 named chunking as one of the two expensive boundaries, alongside permissions. This chapter is why, and what the options are.

The stakes in one sentence: a chunk is the unit you retrieve, so no search can return anything smaller than a chunk or assemble anything that spans two of them badly split. Retrieval quality has a ceiling, and chunking sets it.

What goes wrong at each extreme

The second failure of the large chunk is the less obvious and the more damaging. An embedding is one vector for the whole chunk, so a 4,000-token chunk covering eight topics produces a vector that is near the centroid of all eight and close to none of them. A question about one of the eight matches it weakly, and loses to a smaller chunk that is mostly about something else.

Large chunks do not merely waste context. They dilute the signal that retrieval ranks on.

Four strategies

Fixed-size

Split every N tokens, with an overlap. Simple, uniform, structure-blind.

export function chunkFixed(text: string, size = 800, overlap = 100): string[] {
  const tokens = tokenize(text)
  const chunks: string[] = []
  for (let i = 0; i < tokens.length; i += size - overlap) {
    chunks.push(detokenize(tokens.slice(i, i + size)))
  }
  return chunks
}

Good: predictable cost, works on anything, trivially correct. Bad: splits mid-sentence and mid-table, and ignores every boundary the author put there.

Worth having as a baseline to measure against, which is chapter 31's job. It is rarely the best choice and it is never the worst.

Recursive character splitting

Split on the largest available separator, and fall back to smaller ones when a piece is still too big.

const SEPARATORS = ['\n## ', '\n### ', '\n\n', '\n', '. ', ' ']

export function chunkRecursive(text: string, max = 800, depth = 0): string[] {
  if (countTokens(text) <= max) return [text]
  const separator = SEPARATORS[depth]
  if (!separator) return chunkFixed(text, max, 0)     // out of separators

  const parts = text.split(separator)
  const out: string[] = []
  let buffer = ''

  for (const part of parts) {
    const candidate = buffer ? buffer + separator + part : part
    if (countTokens(candidate) <= max) { buffer = candidate; continue }
    if (buffer) out.push(buffer)
    // A single part larger than max recurses to a finer separator.
    buffer = countTokens(part) > max ? '' : part
    if (!buffer) out.push(...chunkRecursive(part, max, depth + 1))
  }
  if (buffer) out.push(buffer)
  return out
}

Good: respects sentence and paragraph boundaries most of the time, cheap, a sensible default. Bad: operates on a string, so it has already lost the structure that chapter 23's block contract preserved.

Structural

Split on the document's own structure: headings, sections, clauses, table boundaries. This is what the block pipeline exists for.

// Split at the heading level that produces pieces nearest the target, rather
// than at a fixed depth. A document with one h2 and forty h3s needs h3.
export function chunkStructural(blocks: Block[], target = 800): Chunk[] {
  for (const depth of [2, 3, 4]) {
    const pieces = splitAtHeadingDepth(blocks, depth)
    const sizes = pieces.map((p) => countTokens(blockText(p)))
    const median = percentile(sizes, 0.5)
    // Accept the first depth whose median lands in a sane band around target.
    if (median >= target * 0.4 && median <= target * 1.5) {
      return pieces.flatMap((p) => splitOversized(p, target))
    }
  }
  return splitAtHeadingDepth(blocks, 2).flatMap((p) => splitOversized(p, target))
}

Good: chunk boundaries are where the author put topic boundaries, which is exactly right. Tables stay whole. The heading path comes free. Bad: produces wildly uneven sizes — a 40-token section beside a 6,000-token one — and depends on structure the parser recovered, which for PDFs is inferred rather than given.

Choosing the heading depth per document rather than globally is what makes this work across a mixed corpus, and it is the part most implementations skip.

Semantic

Split where the topic changes, detected by embedding successive sentences and cutting at similarity troughs.

export async function chunkSemantic(sentences: string[], target = 800): Promise<string[]> {
  const vectors = await embed(sentences)
  // Distance between each adjacent pair; a peak is a topic boundary.
  const distances = vectors.slice(1).map((v, i) => 1 - cosine(v, vectors[i]))
  const threshold = percentile(distances, 0.85)

  const chunks: string[] = []
  let buffer: string[] = []
  for (let i = 0; i < sentences.length; i++) {
    buffer.push(sentences[i])
    const atBoundary = distances[i] > threshold
    if ((atBoundary && countTokens(buffer.join(' ')) > target * 0.4) ||
        countTokens(buffer.join(' ')) > target * 1.5) {
      chunks.push(buffer.join(' ')); buffer = []
    }
  }
  if (buffer.length) chunks.push(buffer.join(' '))
  return chunks
}

Good: finds boundaries in documents with no structural markers at all — meeting transcripts, scanned prose, resolution notes. Bad: costs an embedding per sentence at ingest, which on 45,000 chunks' worth of sentences is a real number, and it is a threshold to tune on top of the one you were already tuning.

⚠️ Use it where structure is genuinely absent. On a well-structured policy document it underperforms structural splitting and costs twenty times as much.

What to actually do

Route by source shape, which the registry already records.

Source shape	Strategy	Why
Policy, specification documents	Structural, recursive fallback	Real headings, authored boundaries
Contracts	Structural on clause numbers	A clause is the natural unit
Spreadsheet rows	One row per chunk	Chapter 15
Spreadsheet notes	Recursive	Short prose, no structure
Ticket notes	Whole row, or semantic if long	p50 of 380 characters fits in one
Scanned PDFs	Recursive, then semantic	Inferred structure is unreliable

// The dispatcher. One function, and the strategy is data on the source.
export function chunkDocument(doc: TransformedDocument, source: SourceDefinition): Chunk[] {
  const chunks = (() => {
    switch (source.chunkStrategy) {
      case 'structural': return chunkStructural(doc.blocks, source.targetTokens)
      case 'recursive':  return chunkRecursiveBlocks(doc.blocks, source.targetTokens)
      case 'row':        return chunkPerRow(doc.blocks)
      case 'semantic':   return chunkSemanticBlocks(doc.blocks, source.targetTokens)
    }
  })()

  // Invariants every strategy must satisfy, checked once here rather than
  // trusted in four places.
  return chunks.map(enforceInvariants)
}

The invariants

Whatever strategy produced them, these must hold. Checking centrally is what lets you change strategies without re-checking everything.

function enforceInvariants(chunk: Chunk): Chunk {
  // 1. Never empty, never whitespace. An empty chunk embeds to a vector near
  //    the origin, which is weakly similar to everything.
  if (chunk.content.trim().length < 20) {
    throw new SkipChunk(`chunk too short: ${chunk.content.length} chars`)
  }

  // 2. Never larger than the embedding model's input limit. Silent truncation
  //    by the provider means the tail of the chunk is not in its own vector.
  if (chunk.tokens > EMBEDDING_MAX_TOKENS) {
    throw new Error(`chunk of ${chunk.tokens} tokens exceeds the model limit`)
  }

  // 3. A table is whole, or it is not a table chunk.
  if (chunk.kind === 'table' && !isCompleteTable(chunk.content)) {
    throw new Error('table chunk is incomplete — the splitter cut it')
  }

  // 4. The heading path is carried. Chapter 32.
  if (chunk.kind === 'prose' && chunk.headingPath.length === 0 && chunk.ordinal > 0) {
    chunk.warnings.push('no heading path — structure may not have parsed')
  }
  return chunk
}

The second invariant is the one that silently degrades a corpus. Most embedding APIs truncate over-long input rather than erroring, so a 1,200-token chunk against an 8,000-token limit is fine and a 9,000-token chunk is embedded as its first 8,000 tokens — with the remainder present in the stored text, retrievable by nothing.

Why this is expensive to change

Re-chunking means re-embedding, which is the entire cost of an ingest. For Atlas that is tens of dollars and about an hour, which is nothing.

The real cost is that it invalidates measurement. Every retrieval number you have — the test set scores, the tuned score floor, the window size — was measured against one chunking, and changing it resets all of them. That is why chapter 18 insisted the retrieval test set be keyed to document and heading rather than chunk ID: the test set has to survive the experiment it exists to support.

At a larger scale the money becomes real too. Re-embedding ten million chunks is a budget conversation, which is how organisations end up living with a chunking decision made in week two by whoever wrote the first loader.

The failure mode: the clause that lost its condition

A contract reads:

7.2  Payment is due within 30 days of invoice.
7.3  Clause 7.2 does not apply to accounts on a managed credit plan,
     for which payment is due within 60 days.

Chunked at clause level — which is the correct strategy for contracts — these are two chunks. A question about payment terms retrieves 7.2, which is a complete, grammatical, authoritative statement. The answer is 30 days. For a managed-credit account, it is wrong.

Neither a larger chunk nor a smaller one fixes this reliably: 7.3 might be three pages later, under a different heading, in an amendment.

Two things help, and they are both about what travels with a chunk rather than about where it is cut.

// 1. Carry cross-references forward, detected at transform time.
const REF = /\bclauses?\s+(\d+(?:\.\d+)*)/gi

export function linkCrossReferences(chunks: Chunk[]): Chunk[] {
  const byClause = new Map(chunks.map((c) => [c.clauseNumber, c]))
  for (const chunk of chunks) {
    for (const [, number] of chunk.content.matchAll(REF)) {
      const target = byClause.get(number)
      // A clause that is referred to gains a note pointing at the referrer.
      if (target) target.content += `\n\n(See also clause ${chunk.clauseNumber}.)`
    }
  }
  return chunks
}

// 2. Retrieve neighbours. If a chunk is returned, consider its siblings —
//    cheap, because they are in the same document and already indexed.
const expanded = await db.query(
  `select * from chunks
    where document_id = any($1)
      and ordinal between $2 - 1 and $2 + 1`,
  [documentIds, ordinals],
)

Neighbour expansion is chapter 45's territory and it belongs on the list here because it is the mitigation that most often turns a chunking problem into a non-problem. It is also the reason ordinal is in the schema from chapter 8: without a stable within-document position, you cannot ask for the chunk next to this one.

Key points

A chunk is the unit of retrieval, so chunking sets the ceiling on retrieval quality.
Large chunks dilute the embedding, not just the context window. That is the worse of the two failures.
Four strategies: fixed, recursive, structural, semantic. Route by source shape from the registry rather than picking one globally.
Choose heading depth per document, not globally. It is what makes structural splitting work on a mixed corpus.
Use semantic chunking only where structure is genuinely absent. It costs an embedding per sentence.
Enforce invariants centrally: minimum length, the model's input limit, whole tables, a heading path. Silent truncation at the embedding API is the quiet one.
Re-chunking is cheap in money and expensive in invalidated measurement. Key the test set to document and heading.
A correctly split clause can still lose the condition that qualifies it. Link cross-references at transform time and expand to neighbours at retrieval.

Next: chunk size and overlap — measuring the numbers instead of copying someone else's.