Chapter 132 · Production

Caching: prompts, embeddings and whole responses

Four cache layers with very different risk profiles. The prefix discipline that makes prompt caching work, the embedding cache that is free, and the response cache that is mostly a bad idea.

8 min read

Chapter 130 found 46% of input tokens byte-identical on every request and attributed 38% of the cost reduction to caching them. This chapter is the four layers, ordered by how safe they are.

The ordering matters because they look alike and are not. Two are nearly free and nearly risk-free; one is cheap and subtly dangerous; one is dangerous enough that chapter 121 listed it as a security leak path.

Layer	Saves	Risk	Verdict
Prompt prefix	38% of generation cost	None	Always
Embeddings	All repeat embedding	None	Always
Retrieval results	~45 ms, some vector compute	Staleness	Usually
Whole responses	A whole generation	Leakage, staleness	Rarely

Prompt prefix caching

// The provider caches a prefix of the prompt. The requirement is exact:
// every byte before the first varying byte, in order. One interpolated
// timestamp at the top invalidates all of it.
const prompt = [
  SYSTEM_PROMPT,        // 1,240 — identical
  FEW_SHOT_EXAMPLES,    // 620   — identical
  TOOL_DEFINITIONS,     // 980   — identical per role
  // ---------------- cache boundary: 2,840 tokens ----------------
  renderContext(chunks),
  renderSessionState(state),
  question,
].join('\n\n')

// Four things that silently break it, all of which Atlas did at least
// once.
const BREAKERS = [
  'A date or request id in the system prompt.',
  'Tool definitions serialised from an object with non-deterministic key order.',
  'A user name or role rendered at the top rather than near the question.',
  'A prompt built by string concatenation in a different order on one code path.',
]

// The guard, which costs one assertion and catches all four. It runs in
// development and in the eval suite, not in production.
const prefixHash = sha256(prompt.slice(0, prompt.indexOf(CACHE_BOUNDARY)))
assert(prefixHash === EXPECTED_PREFIX_HASH,
  'prompt prefix changed — prompt caching is now a no-op until this is intentional')

The second breaker is the one that is hardest to see. JSON.stringify over an object preserves insertion order, and a tool registry built from a map iterated in a different order on a different code path produces a different byte sequence with identical semantics. Atlas lost caching for eleven days to exactly that and found it in the cost dashboard rather than anywhere else.

// Which is the operational lesson: cache effectiveness is a metric, not
// a configuration. Emit it.
span.setAttribute('llm.cached_prompt_tokens', response.usage.cachedTokens ?? 0)

Embedding cache

The easiest win in the chapter and frequently missed.

// Embeddings are a pure function of text and model. Content-addressed,
// shared across all users safely — the key is the text itself, so there
// is nothing to leak that the holder does not already have.
const key = `emb:${embeddingModelId}:${sha256(text)}`

	Hit rate	Saving
Query embeddings	31%	$3/month
Ingestion re-embedding	89%	$57/month

The ingestion figure is the real one and it comes from chapter 34's problem: a document is re-fetched, most of its chunks are byte-identical to the previous version, and without a cache every one is re-embedded. Eighty-nine percent of ingestion embedding spend was re-computing vectors that already existed.

// ⚠️ The model id in the key is mandatory and is the one mistake worth
// warning about. An embedding cache keyed on text alone, after an
// embedding model change, silently serves vectors from the old space
// into a new index. Retrieval degrades and nothing errors.

Retrieval cache

// Cheap, useful, and this is where chapter 121's constraint arrives:
// the key must carry the caller's permission scope.
const key = scopedKey(ctx, 'retrieval', `${embeddingModelId}:${corpusSnapshotId}:${sha256(question)}`)

Two components beyond the user scope do the real work:

// 1. corpusSnapshotId. Without it, a cached result survives a corpus
//    update and serves chunks that have been superseded or deleted.
//    Chapter 35's orphan-chunk problem, reintroduced by a cache.
//
// 2. A short TTL anyway, as a backstop. Atlas uses 15 minutes, which is
//    shorter than the ingestion cycle, so a bug in snapshot propagation
//    costs fifteen minutes of staleness rather than a day.

Hit rate 18% after user scoping, saving about 45 ms on those and a little vector compute. Modest, and worth having mostly for the latency.

Response caching

// Tempting: a whole generation saved. Three reasons Atlas serves only a
// narrow slice of traffic from it.

It leaks. The original failure from chapter 121. Scoping the key by user and group hash fixes the leak and takes the hit rate from 23% to 11%.

It goes stale invisibly. An answer is derived from documents that change. A cached answer about payment terms outlives the contract amendment, and nothing in the cache knows.

It defeats the audit trail. A served cache hit produces no retrieval, no policy evaluation and no audit record of what was reached — or it produces a synthetic one, which is worse because it is false.

// What Atlas actually caches: answers to questions whose sources have
// not changed, with the source set in the key, a short TTL, and a real
// audit row recording the cache hit and the original run.
const key = scopedKey(ctx, 'answer', `${sha256(question)}:${sourceVersionHash(sources)}`)
await audit.record('answer.served_from_cache', { runId, originalRunId, key })

sourceVersionHash is what makes it defensible: the key changes when any contributing document changes, so staleness is bounded by ingestion rather than by a TTL. It requires knowing which documents contributed, which the audit record already holds.

Final hit rate: 6%, saving about $17 a month. Kept mostly because it removes the duplicate-question burst that follows a company-wide announcement.

A response cache is the only layer here that can serve a wrong answer or someone else's. Treat it as a feature with a threat model rather than as an optimisation.

Invalidation

// Three strategies, and the ordering is the advice.
const STRATEGY = {
  contentAddressed: 'preferred. Nothing to invalidate; the key changes.',
  versionInKey:     'next best. The corpus snapshot or source hash.',
  explicitPurge:    'last resort. It will be forgotten on one code path.',
}

Every cache in this chapter uses one of the first two. That is not an accident of design taste — an explicit purge is a second system that has to stay in step with the first, and the failure mode is serving stale data with full confidence.

Key points

Four layers with four risk profiles. Prompt prefix and embeddings are always worth it; retrieval usually; whole responses rarely.
Prompt caching needs an exact byte prefix. A date in the system prompt, non-deterministic key order in tool definitions, a user name near the top, or a different concatenation order all silently disable it.
Atlas lost prompt caching for eleven days to tool definitions serialised in a different key order, and found it in the cost dashboard.
Assert the prefix hash in development and in the eval suite, and emit cached-token count as a metric. Cache effectiveness is a metric, not a configuration.
Embedding caches are content-addressed and safe to share. Ingestion hit rate was 89%, saving $57 a month.
Put the embedding model id in the key. Without it, a model change silently serves old-space vectors into a new index with no error.
Retrieval cache keys need the user scope, the corpus snapshot id and a short TTL backstop.
Response caching leaks, goes stale invisibly, and defeats the audit trail. Scope by user, key on a hash of the contributing sources, and write a real audit row for the hit.
Prefer content-addressed keys, then a version in the key. An explicit purge is a second system that will be forgotten on one code path.

Next: latency budgets and where the time goes — the other half of what caching bought.