Chapter 27 · Ingestion

Parsing PDFs, and why it is harder than it looks

The stage that destroys more retrieval quality than any other. Three populations of PDF, how to tell them apart, and the checks that catch a parse that silently produced nonsense.

12 min read

A PDF describes where marks go on a page. It does not describe a document's structure, and frequently does not describe its reading order. Everything difficult in this chapter follows from that.

This is the stage where the dry run's warning about product specifications becomes real: 1,900 specification documents, 40% of them supplier PDFs, 760 with no usable text layer.

Three populations, and the triage

Triage first, because the three have different costs and only the first is free. Running everything through OCR because some of it needs OCR is the common shortcut and it is expensive in money and in quality — OCR of a native PDF is strictly worse than reading its text layer.

// src/ingest/parse/triage.ts
export type PdfClass = 'native' | 'awkward' | 'scanned'

export async function triagePdf(bytes: Buffer): Promise<PdfClass> {
  const doc = await pdfjs.getDocument({ data: bytes }).promise
  const pages = Math.min(doc.numPages, 5)     // a sample is enough
  let chars = 0
  let items = 0

  for (let i = 1; i <= pages; i++) {
    const content = await (await doc.getPage(i)).getTextContent()
    items += content.items.length
    chars += content.items.reduce((n, it) => n + ('str' in it ? it.str.length : 0), 0)
  }

  // ⚠️ Per page, not in total. A 300-page scan with one text cover page has
  // plenty of characters in absolute terms and no text anywhere else.
  const charsPerPage = chars / pages
  if (charsPerPage < 100) return 'scanned'

  // Many tiny fragments means positioned glyphs rather than text runs: a
  // multi-column layout, or a PDF generated by a drawing tool.
  const avgItemLength = chars / Math.max(items, 1)
  if (avgItemLength < 4) return 'awkward'

  return 'native'
}

Measuring per page is the detail that matters. The absolute-character test passes on exactly the documents it should catch.

Native: the easy case, with one trap

# The quickest useful check, before writing any code. -layout preserves
# columns; without it, a two-column page interleaves into nonsense.
pdftotext -layout sample.pdf - | head -50

The trap is reading order. A PDF's text objects are in the order they were drawn, which for a two-column layout is often column-interleaved: line one of column A, line one of column B, line two of column A. Extracted naively, every sentence is spliced with an unrelated one, and the result is fluent-looking garbage that chunks and embeds without complaint.

// Sort by position, not by draw order. Coarse bucketing by y, then x.
function orderTextItems(items: TextItem[], pageWidth: number): TextItem[] {
  const LINE_TOLERANCE = 3    // points

  // Detect columns by looking for a vertical gap in x-coverage.
  const columns = detectColumns(items, pageWidth)

  return items
    .map((it) => ({ ...it, column: columns.findIndex((c) => it.x >= c.from && it.x < c.to) }))
    .sort((a, b) =>
      a.column !== b.column ? a.column - b.column        // column order first
      : Math.abs(a.y - b.y) > LINE_TOLERANCE ? b.y - a.y  // then down the page
      : a.x - b.x,                                        // then across the line
    )
}

Recovering structure that was never there

A PDF has no headings. It has text that is larger and bolder, and the structure the chunker needs has to be inferred from that.

// Infer heading levels from font size clustering, not from absolute sizes.
// Absolute thresholds break on the first document with a different template.
export function inferHeadings(items: TextItem[]): Block[] {
  const sizes = items.map((i) => Math.round(i.fontSize * 2) / 2)
  const counts = new Map<number, number>()
  for (const s of sizes) counts.set(s, (counts.get(s) ?? 0) + 1)

  // Body text is whatever size covers the most characters. Everything
  // meaningfully larger is a heading, ranked by size.
  const body = [...counts.entries()].sort((a, b) => b[1] - a[1])[0][0]
  const headingSizes = [...counts.keys()].filter((s) => s > body * 1.15).sort((a, b) => b - a)

  return items.map((item) => {
    const depth = headingSizes.indexOf(Math.round(item.fontSize * 2) / 2)
    return depth >= 0
      ? { type: 'heading', depth: depth + 1, text: item.str }
      : { type: 'paragraph', text: item.str }
  })
}

Clustering relative to the document's own body size is what makes this portable across templates. A supplier's specification sheet with 8pt body text and a company policy with 11pt both work, and neither needs a configuration entry.

⚠️ This is heuristic and it will be wrong on some documents. That is acceptable because of what chapter 32 does with it: a missed heading degrades the heading path, which degrades retrieval slightly. A missed heading does not corrupt content.

Scanned: OCR, and its costs

// OCR is a separate pipeline: slower, costly, and it introduces errors that
// nothing downstream can detect. Keep it out of the main path.
export async function ocrDocument(bytes: Buffer): Promise<ParsedDocument> {
  const pages = await rasterise(bytes, { dpi: 300 })   // below 300, accuracy falls off
  const results = await Promise.all(pages.map((p) => ocr(p)))

  const meanConfidence =
    results.reduce((n, r) => n + r.confidence, 0) / results.length

  // ⚠️ Low-confidence OCR produces plausible words that are the wrong words.
  // An indexed document of plausible wrong words is worse than no document.
  if (meanConfidence < 0.80) {
    throw new SkipDocument('parse', `OCR confidence ${meanConfidence.toFixed(2)} below 0.80`)
  }

  return {
    blocks: results.flatMap((r) => r.blocks),
    parseWarnings: meanConfidence < 0.92
      ? [`OCR confidence ${meanConfidence.toFixed(2)} — verify before relying on figures`]
      : [],
    // ... rest of the document
  }
}

Two numbers there, and the gap between them is deliberate. Below 0.80 the document is skipped. Between 0.80 and 0.92 it is indexed with a warning that travels with it, because an OCR document that is mostly right is useful and should not be silently trusted on an exact figure.

The specific risk with OCR is digits. 0 and 8, 1 and 7, 5 and 6 are the common confusions, and a specification is mostly numbers. A tolerance of "0.5mm" read as "0.6mm" is not detectable by any downstream check.

// So: for OCR'd sources, carry the warning into retrieval, as chapter 19
// carries a freshness stamp.
if (chunk.parseWarnings.length) {
  sources.push(`[${n}] ${title} ⚠️ ${chunk.parseWarnings[0]}\n${chunk.content}`)
}

The checks that catch a bad parse

A parse that produces nonsense does not throw. It produces a string, and every stage after it works perfectly on garbage. These four checks cost nothing and catch most of it.

export function validateParse(doc: ParsedDocument, bytes: Buffer): void {
  const text = doc.blocks.map(blockText).join(' ')

  // 1. Yield. Text should be a plausible fraction of the file size.
  const yieldRatio = text.length / bytes.length
  if (yieldRatio < 0.001) {
    throw new SkipDocument('parse', `yield ${yieldRatio.toExponential(1)} — likely empty`)
  }

  // 2. Word shape. Garbled extraction produces very long or very short tokens.
  const words = text.split(/\s+/).filter(Boolean)
  const avgWord = words.reduce((n, w) => n + w.length, 0) / Math.max(words.length, 1)
  if (avgWord > 18 || avgWord < 2) {
    throw new SkipDocument('parse', `mean word length ${avgWord.toFixed(1)} — likely garbled`)
  }

  // 3. Whitespace loss. "Thereturnswindowis30days" extracts as one token.
  const spaceRatio = (text.match(/ /g)?.length ?? 0) / Math.max(text.length, 1)
  if (spaceRatio < 0.08) {
    throw new SkipDocument('parse', `space ratio ${spaceRatio.toFixed(3)} — spacing lost`)
  }

  // 4. Replacement characters: an encoding or embedded-font failure.
  const bad = (text.match(/\uFFFD/g)?.length ?? 0) / Math.max(text.length, 1)
  if (bad > 0.01) {
    throw new SkipDocument('parse', `${(bad * 100).toFixed(1)}% replacement characters`)
  }
}

The third check is the one that earns its place most often. Losing inter-word spacing is a common failure with embedded subset fonts, and the resulting text is a single enormous token per line — which embeds to something meaningless and retrieves for nothing, forever, silently.

Reading the triage report before building

# Run this over the corpus before writing the parser. It is twenty minutes
# and it sizes the work.
for f in ./export/specs/*.pdf; do
  chars=$(pdftotext -f 1 -l 5 "$f" - 2>/dev/null | tr -d '[:space:]' | wc -c)
  pages=$(pdfinfo "$f" 2>/dev/null | awk '/^Pages:/ {print $2}')
  echo "$((chars / 5)) ${f}"
done | awk '$1 < 100 {s++} $1 >= 100 {n++} END {print "scanned:", s, "native:", n}'

For Atlas: 760 scanned, 1,140 native. That ratio decides three things — whether OCR is in the first slice at all, roughly what the backfill costs, and whether specification questions can be answered before the OCR pipeline exists.

The answer for Atlas was to ship with the 1,140 and record the 760 as a not yet in the exclusion register, with the two blocked questions named.

The failure mode: the table that became a column of numbers

Specification PDFs are mostly tables, and a table extracted by a text-stream parser becomes a run of values with the headers somewhere above.

Operating temperature Storage temperature Humidity
-20 to 60 -40 to 85 5 to 95

Chunked, embedded and retrieved, this produces an answer: "The operating temperature range is -40 to 85°C." Confident, cited, and off by a specification that matters.

The fix is not a better chunker. It is to detect tables during parsing, as chapter 15 required, and to emit a table block rather than paragraphs.

// Detect by x-position clustering: a table is text aligned into columns
// across several consecutive lines.
export function detectTables(lines: Line[]): Block[] {
  const blocks: Block[] = []
  let run: Line[] = []

  for (const line of lines) {
    const alignsWithRun =
      run.length > 0 && columnPositions(line).length === columnPositions(run[0]).length

    if (alignsWithRun) { run.push(line); continue }
    if (run.length >= 3) blocks.push(toTableBlock(run))   // 3+ aligned lines
    else blocks.push(...run.map(toParagraph))
    run = [line]
  }
  return blocks
}

And the check that proves it is working, which matters more than the heuristic:

// A specification document with no table blocks is a parse failure, whatever
// the yield check said. Assert against what the source shape should produce.
if (source.expectsTables && !doc.blocks.some((b) => b.type === 'table')) {
  doc.parseWarnings.push('no tables detected in a source that should contain them')
}

Assert on the shape a parse produces, not only on whether it produced something. A specification with no tables and a contract with no clause numbers are both successful parses of nothing useful.

Key points

Triage into native, awkward and scanned before building. Only the first is free, and OCR of a native PDF is worse than reading its text layer.
Test extractable characters per page, not in total.
Sort text by position, not draw order. Column-interleaved extraction produces fluent garbage.
Infer headings by clustering font sizes relative to the document's own body size, not against absolute thresholds.
OCR below 0.80 confidence is skipped; between 0.80 and 0.92 it is indexed with a warning that travels into retrieval. Digits are where OCR errors hurt.
Four cheap parse checks: yield ratio, mean word length, space ratio, replacement characters. The space-ratio check catches the common subset-font failure.
Detect tables in the parser and emit table blocks. A table flattened into a text run produces confident, wrong specifications.
Assert on the shape a parse should produce, not only that it produced something.

Next: spreadsheets, and the tables inside documents — merged cells, formulas, and the sheet that is really five tables.