Chapter 19 · The data layer
Freshness, source by source
How current each source has to be, and the rule that decides whether it may be indexed at all. Staleness is the failure that produces confident, well-cited, wrong answers.
10 min read
Chapter 12 gave a rule in one line and deferred the numbers: if the answer changes faster than the sync interval, it cannot come from the index. This chapter turns that into a per-source decision.
Staleness deserves its own chapter because it is the failure mode with no symptom. A stale answer is well-formed, correctly cited, and derived faithfully from a real source. Nothing in the answer, the logs or the eval suite distinguishes it from a correct one. The only defence is a decision made here.
The two numbers per source
For every source, two numbers that are usually easy to obtain and rarely written down.
Change rate — how often the underlying facts change. Measured, not estimated.
Tolerance — how stale an answer may be before it causes harm. This is a business question, not a technical one, and the answer varies enormously across sources in the same company.
-- Change rate, measured. For a relational source this is one query.
select date_trunc('day', updated_at) as day, count(*)
from inventory
where updated_at > now() - interval '14 days'
group by 1 order by 1;# For a document source: how many files changed in the last fortnight.
find ./export/policy -type f -mtime -14 | wc -l
find ./export/policy -type f | wc -lThe decision table
| Change rate | Tolerance | Route |
|---|---|---|
| Slow | Any | Index, sync nightly |
| Fast | Hours or more | Index, sync frequently |
| Fast | Minutes or less | Live query. Never index. |
| Slow | Minutes | Index, but stamp and show the age |
The third row is the one that must be enforced rather than intended, and the fourth row is the one people forget exists — a slowly changing source with a tight tolerance is fine to index as long as the answer carries its timestamp.
For Atlas:
| Source | Change rate | Tolerance | Route |
|---|---|---|---|
| Policy documents | ~2 per month | Days | Index, nightly |
| Product specifications | ~30 per month | Days | Index, nightly |
| Customer contracts | ~12 per week | Hours | Index, hourly |
| Pricing workbook | ~2 per month | Minutes | Index, nightly, stamped |
| Orders and inventory | Thousands per day | Minutes | Live query |
| Support tickets | Hundreds per day | Minutes | Live query |
| Account ownership (permissions) | ~5 per week | Minutes | Index, every 15 min |
Two rows there are counter-intuitive and both are important.
The pricing workbook changes twice a month and has a minutes-level tolerance. Those look contradictory until you consider what happens when it does change: a discount tier is updated the morning of a negotiation, and a rep quoting yesterday's number quotes a number the company no longer offers. Rare and consequential. The handling is not a faster sync — it is to stamp the answer with the source's effective date so a human can notice.
Account ownership is permission data, and it syncs far more often than the content it governs. A revoked grant still in your index is an access-control gap, as chapter 13 argued. The asymmetry is deliberate: the cost of a stale permission is categorically different from the cost of a stale fact.
Sync permissions on a different, faster schedule than content. A stale fact is wrong. A stale grant is a leak.
Stamping, and showing the age
When an indexed source has a tight tolerance, the answer carries the age. This is cheap and it is the single most effective staleness mitigation available.
// Freshness travels with the chunk, from the index into the prompt.
export interface Retrieved {
id: number
content: string
score: number
sourceUpdatedAt: Date // when the SOURCE changed, not when we indexed it
indexedAt: Date // when we last read it
}
/** Per-source tolerance, from the table above. */
const TOLERANCE_HOURS: Record<string, number> = {
policy_docs: 24 * 7,
product_specs: 24 * 7,
customer_contracts: 24,
pricing_workbook: 24,
}
export function freshnessNote(chunk: Retrieved, source: string): string | null {
const ageHours = (Date.now() - chunk.sourceUpdatedAt.getTime()) / 3_600_000
const tolerance = TOLERANCE_HOURS[source]
if (!tolerance || ageHours <= tolerance) return null
return `⚠️ last updated ${chunk.sourceUpdatedAt.toISOString().slice(0, 10)}`
}Keeping sourceUpdatedAt and indexedAt as separate fields matters more than it looks. The age a person cares about is how old the fact is. The age you care about when debugging is how long ago you read it, and a gap between the two is exactly how you detect a sync that has silently stopped.
// Render the stamp into the sources block, so the model can report it.
const sources = chunks.map((c, i) => {
const note = freshnessNote(c, c.source)
return `[${i + 1}] ${c.title}${note ? ` (${note})` : ''}\n${c.content}`
}).join('\n\n')And one line in the prompt, which is the part that makes it work:
If a source carries a "last updated" note, mention the date in your answer and
suggest confirming the current value.Detecting a sync that stopped
This is the failure that causes the most damage per unit of likelihood, because it is silent on both sides: the index keeps serving, retrieval keeps working, and answers keep being well-formed and cited. They are simply from a frozen snapshot.
Two complementary checks. Neither is sufficient alone.
-- 1. Staleness per source. Alert when the oldest sync exceeds the interval.
select source,
max(indexed_at) as last_indexed,
now() - max(indexed_at) as age,
count(*) as documents
from documents
where deleted_at is null
group by source
order by age desc;// 2. A heartbeat the sync writes on every run, including runs that changed
// nothing. This is the check that catches the real failure.
export async function recordSyncRun(source: string, result: SyncResult) {
await db.query(
`insert into sync_runs (source, ran_at, documents_seen, documents_changed, ok, error)
values ($1, now(), $2, $3, $4, $5)`,
[source, result.seen, result.changed, result.ok, result.error ?? null],
)
}The distinction between the two is the whole point. Check 1 looks at documents.indexed_at, which only advances when something changes — so a source with no changes for a week looks identical to a source whose sync has been crashing for a week. Check 2 separates "nothing changed" from "nothing ran".
-- The alert worth having. A source whose last SUCCESSFUL run is older than
-- twice its interval, regardless of whether anything changed.
select source, max(ran_at) filter (where ok) as last_ok
from sync_runs
group by source
having max(ran_at) filter (where ok) < now() - interval '2 hours';The live-query route and its cost
Live querying solves staleness completely and buys three new problems. Worth knowing before choosing it for a borderline source.
| Cost | Detail | Mitigation |
|---|---|---|
| Latency | A live call is in the request's 4-second budget | Hard timeouts, as in chapter 12 |
| Availability | The source being down makes the answer unavailable | Explicit unavailable flag, never an empty result |
| Permissions | A second filter implementation, in the tool | One Scope object, per chapter 13 |
Which is why the rule is a threshold rather than a preference: live query where staleness would be harmful, index everywhere else.
The failure mode: the discount that changed on Tuesday
Concretely, because the general statement does not land.
The pricing workbook is indexed nightly. On Tuesday morning the north-region tier-3 discount changes from 12% to 9%, because a supplier cost increased. The workbook is updated at 9:15.
At 11:00 a rep asks Atlas for the north tier-3 discount. The index holds Monday night's snapshot. Atlas answers 12%, cites the pricing workbook, and is well-formed and confident.
The rep quotes 12% on a $40,000 order. The discrepancy is found at invoicing, three weeks later, and by then it is a customer conversation rather than a correction.
Trace what each defence would have done:
| Defence | Outcome |
|---|---|
| Nightly sync alone | ❌ Wrong answer, no signal |
| Sync every 15 minutes | ✅ Right answer — and 96× the sync cost for a twice-monthly change |
| Stamped with the effective date | ⚠️ "12%, effective 2026-01-01 — confirm as this may have changed." The rep checks. |
| Live query tool | ✅ Right, and the workbook is not a queryable system |
| Stamp and a change-triggered sync | ✅ The answer is right, and stays right between triggers |
The last row is the answer, and the second component is cheaper than it sounds: a file-watch or webhook on one file that triggers a re-ingest of one source.
// A targeted re-ingest on change, rather than a faster schedule for everything.
// The cost is proportional to changes, not to the clock.
watch('./sources/pricing-workbook.xlsx', async () => {
logger.info('sync.triggered', { source: 'pricing_workbook', trigger: 'file_change' })
await ingestSource('pricing_workbook')
})This is the shape of most good freshness solutions: not a faster interval, but a trigger on the small number of sources where latency actually matters, plus a stamp so that the residual window is visible to a human rather than hidden.
Key points
- Record two numbers per source: measured change rate and business tolerance. They are independent, and a slow-changing source can have a tight tolerance.
- If the answer changes faster than the sync interval, the source cannot be indexed.
- Sync permissions faster than content. A stale fact is wrong; a stale grant is a leak.
- Keep
sourceUpdatedAtandindexedAtseparate. The gap between them is how you detect a stopped sync. - Stamp answers from sources with tight tolerance, and tell the model in the prompt to mention the date.
- A heartbeat on every sync run, including no-change runs. Staleness checks on content alone cannot distinguish "nothing changed" from "nothing ran".
- Prefer a change trigger on a few sources over a faster schedule for everything.
Next: the data you should refuse to connect — and how to say so in a way that is heard.