Chapter 29 · Ingestion

Email and chat archives

The source excluded from version one. What it takes if you ever connect it: threading, quote stripping, per-message permissions, and the authority problem that has no technical fix.

12 min read

Atlas excludes sales email, decided in chapter 5 and justified in chapter 12 and chapter 16. Nothing here reverses that.

This chapter exists because the exclusion is recorded as revisit: after version one rather than never, and because conversations are the most requested source in almost every project of this kind. If you are going to connect them, these are the four problems, in the order they bite — and the fourth has no technical solution, which is the reason the recommendation stands.

Problem 1: a message is not a document

The unit is the thread. A single message is unintelligible alone: "Yes, that works — go ahead" retrieves for nothing and answers nothing.

// src/ingest/parse/thread.ts
// Group by thread, then order by time. Both are harder than they look.
export function groupIntoThreads(messages: RawMessage[]): Thread[] {
  const byId = new Map(messages.map((m) => [m.messageId, m]))
  const roots = new Map<string, RawMessage[]>()

  for (const message of messages) {
    // References is the reliable header; In-Reply-To is often missing and
    // Subject matching merges unrelated threads that share a subject line.
    const rootId = message.references?.[0] ?? message.inReplyTo ?? message.messageId
    // Walk up in case the first reference is itself a reply.
    let root = rootId
    for (let i = 0; i < 20 && byId.get(root)?.references?.[0]; i++) {
      root = byId.get(root)!.references![0]
    }
    roots.set(root, [...(roots.get(root) ?? []), message])
  }

  return [...roots.entries()].map(([id, msgs]) => ({
    id,
    messages: msgs.sort((a, b) => +a.date - +b.date),
    // ⚠️ Not the root's subject: threads get renamed mid-conversation and
    // the last subject is usually the accurate one.
    subject: msgs[msgs.length - 1].subject,
  }))
}

The depth-limited walk matters. Mail clients produce References chains that occasionally loop, and an unbounded walk hangs the ingest on one malformed message out of 400,000.

Problem 2: every message quotes every previous message

A ten-message thread naively concatenated contains the first message ten times. Embedded, it swamps the index with near-duplicates of the same text, and retrieval returns five chunks that are the same paragraph quoted at five depths.

// Strip quoted text before anything else. These patterns cover most clients;
// the residual is handled by the dedupe below rather than by more patterns.
const QUOTE_MARKERS = [
  /^>+ /m,                                            // plain quoting
  /^On .{10,80} wrote:$/m,                            // Gmail, Apple Mail
  /^-{2,}\s*Original Message\s*-{2,}$/im,              // Outlook
  /^From:.*\nSent:.*\nTo:.*\nSubject:.*$/im,           // Outlook header block
  /^_{10,}$/m,                                        // Outlook divider
]

export function stripQuoted(body: string): string {
  let cut = body.length
  for (const marker of QUOTE_MARKERS) {
    const match = marker.exec(body)
    if (match && match.index < cut) cut = match.index
  }
  return body.slice(0, cut).trim()
}

// Belt and braces: drop any paragraph that already appears earlier in the
// thread. Catches the clients the patterns miss.
export function dedupeThread(messages: ParsedMessage[]): ParsedMessage[] {
  const seen = new Set<string>()
  return messages.map((m) => ({
    ...m,
    body: m.body
      .split(/\n{2,}/)
      .filter((para) => {
        const key = normalise(para)
        if (key.length < 40) return true        // short lines repeat legitimately
        if (seen.has(key)) return false
        seen.add(key)
        return true
      })
      .join('\n\n'),
  }))
}

Signatures and disclaimers are the same problem at a smaller scale, and they are worse than they look: a six-line legal disclaimer on 400,000 messages is a substantial fraction of the corpus, all identical, all embedded.

// Learn the signatures rather than listing them. Any trailing block that
// appears on more than ~20 messages from the same sender is a signature.
export function learnSignatures(messages: ParsedMessage[]): Map<string, Set<string>> {
  const counts = new Map<string, Map<string, number>>()
  for (const m of messages) {
    const tail = m.body.split(/\n{2,}/).slice(-1)[0]
    if (!tail || tail.length > 400) continue
    const perSender = counts.get(m.from) ?? new Map()
    perSender.set(normalise(tail), (perSender.get(normalise(tail)) ?? 0) + 1)
    counts.set(m.from, perSender)
  }
  return new Map(
    [...counts].map(([sender, tails]) => [
      sender,
      new Set([...tails].filter(([, n]) => n >= 20).map(([t]) => t)),
    ]),
  )
}

Problem 3: permissions are per message, not per thread

This is where email breaks the model from chapter 13 outright.

The document-level classification that every other source uses does not exist here. Three options, and only the third is sound.

Option	Verdict
Most permissive message governs the thread	❌ Leaks message 4 to everyone on message 1
Least permissive governs	⚠️ Safe, and hides almost everything
Chunk per message, permission per chunk	✅ Correct, and breaks thread coherence

-- Which means the schema from chapter 8 has to change for this source:
-- permissions move from documents to chunks, for conversations only.
alter table chunks add column principals text[];   -- null except for conversations

create index chunks_principals on chunks using gin (principals)
  where principals is not null;

-- And retrieval gains a branch, which is the real cost.
where d.deleted_at is null
  and (
    (c.principals is null and d.classification = any($class)
       and (d.account_id is null or d.account_id = any($accounts)))
    or (c.principals is not null and c.principals && $principals)
  )

That or is the price of connecting conversations: the clean single-axis filter from chapter 13 becomes two filters with different shapes, and every future change to permissions has to be correct in both branches. It is the strongest engineering argument for the exclusion, separate from the privacy one.

Problem 4: the authority problem, which has no fix

Four sentences from a real thread shape:

"I think we could probably do 15% for them." "Let's go with 15%." "Correction — 15% needs VP sign-off. 12% is the floor I can approve." "Confirmed at 12%, contract amended."

Retrieval returns one of these. All four are grammatical, confident assertions containing a number. Nothing in the text marks which is a musing, which is a decision, which is a correction, and which is the outcome. Recency helps and is not sufficient — thread order is not decision order when a conversation branches.

Three partial mitigations, offered as partial:

// 1. Carry position and time into the chunk, so the model can at least see
//    that a message is early in a long thread.
content: `Message ${i + 1} of ${thread.messages.length} · ${fmtDate(m.date)}\n` +
         `From ${role(m.from)} to ${m.to.map(role).join(', ')}\n\n${m.body}`

// 2. Weight recency within a thread at retrieval time.
score: baseScore * (0.85 + 0.15 * (messageIndex / threadLength))

// 3. The one that works: prefer the system of record. If a contract exists,
//    the contract's number is authoritative and the thread is context.
if (hasAuthoritativeSource(question)) demote(conversationChunks)

The third is not a parsing technique. It is an admission that conversations should be a supplement to a system of record and never the answer, and it is why the right handling for Atlas's freight question in chapter 18 was to create a document, not to connect the mailbox.

A conversation records that something was discussed. Only a system of record establishes what is true.

If you connect it anyway: the minimum

A checklist, in the order the chapters established them.

A scoped mailbox, not an estate. One shared mailbox, allow-listed by the rule in chapter 16 — never per-person inboxes.
A date floor. Anything older than the current contract generation is superseded context.
Per-message chunks with per-message principals, and the retrieval branch tested by the parity check from chapter 14.
Quote, signature and disclaimer stripping, measured: report the ratio of retained to raw characters per run.
Demotion against systems of record, so a contract always outranks a thread about the contract.
An erasure path, because a mailbox is the most likely source of a deletion request.

// The measurement for point 4, which also detects a stripper that broke.
logger.info('thread.stripped', {
  rawChars, retainedChars,
  ratio: +(retainedChars / rawChars).toFixed(3),   // expect ~0.2-0.4
})

A ratio near 1.0 means the stripping silently stopped working and you are about to embed every quote ten times. A ratio near 0.05 means it is eating real content. Both are visible in one number, and neither is visible any other way.

The failure mode: the thread that was never the decision

A rep asks what discount was agreed with Henderson. The corpus contains both the contract — 12%, signed — and the thread above.

Retrieval returns the contract chunk and two thread chunks. The thread chunks are more conversational, more specific-sounding, and mention 15% twice against 12% once. The model writes: "The agreed discount is 15%, though one message mentions 12% as a floor."

The contract was cited. The answer is wrong. Nothing in the pipeline failed — the thread genuinely says 15%, twice, in a message that genuinely is in the thread.

This is the failure that cannot be engineered away at parse time, because the defect is in the source rather than in the handling of it. The only reliable defence is the one in point 5: a hard precedence rule, enforced before the model sees anything.

// Not a prompt instruction. A filter, for the same reason permissions are a
// filter — a precedence the model is asked to respect is a preference.
export function applyPrecedence(chunks: Retrieved[]): Retrieved[] {
  const hasAuthoritative = chunks.some((c) => AUTHORITATIVE.has(c.sourceId))
  return hasAuthoritative
    ? chunks.filter((c) => !CONVERSATIONAL.has(c.sourceId))
    : chunks
}

const AUTHORITATIVE = new Set(['customer_contracts', 'policy_docs', 'pricing_workbook'])
const CONVERSATIONAL = new Set(['sales_email', 'chat_archive'])

Dropping conversational chunks entirely whenever an authoritative one is present is blunt, and bluntness is correct here. The partial version — keeping both and hoping the model weighs them — is precisely the failure above.

Key points

The unit is the thread, assembled from References with a depth-limited walk, not from subject matching.
Strip quotes, then dedupe paragraphs across the thread. Learn signatures per sender rather than listing patterns.
Permissions in email are per message. That forces per-chunk principals and a second branch in the retrieval filter — the strongest engineering argument for excluding conversations.
Nothing in a message's text distinguishes a musing from a decision from a correction. Position and recency help and do not solve it.
A conversation records that something was discussed. Only a system of record establishes what is true.
If you connect it: scoped mailbox, date floor, per-message principals, measured stripping, hard precedence, and an erasure path.
Report the retained-to-raw character ratio. A ratio near 1.0 or near 0.05 are both silent failures otherwise.
Enforce precedence as a filter, not as a prompt instruction. Drop conversational chunks when an authoritative source is present.

Next: chunking — the decision that quietly determines retrieval quality, and the one that is expensive to revisit.