Chapter 39 · Retrieval

Choosing an embedding model

Leaderboards measure someone else's corpus. The four criteria that decide it, the two-hour bake-off that answers it, and what a migration costs when you change your mind.

11 min read

The previous chapter covered what embeddings do. This one is the procurement decision, which is mostly an evaluation problem and partly a migration-cost problem.

The short version: public benchmarks are a shortlist, not an answer, and the bake-off that produces the answer takes two hours.

Why the leaderboard is a shortlist

Benchmark suites average across dozens of tasks — scientific abstracts, forum posts, product reviews, multilingual retrieval. Your corpus is none of those. Atlas's is internal policy prose dense with SKUs, clause numbers and defined terms, queried by people using internal jargon.

A model two points higher on an average of unrelated tasks can be worse on that. The leaderboard is useful for exactly one thing: narrowing twenty candidates to four.

Use the benchmark to decide what to test. Use your test set to decide what to ship.

The four criteria

In the order they eliminate candidates.

1. Can you run it at all

// Elimination questions, answered before any quality testing.
interface Feasibility {
  residency: string[]       // chapter 17 — does it serve your region?
  dataRetention: string     // zero-retention available?
  selfHostable: boolean     // needed if residency cannot be met
  maxInputTokens: number    // must exceed your p99 chunk, not your target
  rateLimit: number         // does the backfill finish this week?
}

The maxInputTokens check catches a real problem. A model with an 8,192-token limit is fine for 512-token chunks and silently truncates the oversized table chunks that chapter 30 deliberately allowed to exceed the target.

-- Check your actual distribution, not your target.
select max(tokens), percentile_cont(0.999) within group (order by tokens)
  from chunks;

2. Quality on your corpus

The bake-off, below. This is the only criterion that needs real work.

3. Cost at your scale

const chunks = 44_387
const avgTokens = 420
const queriesPerDay = 800

// Ingest is a one-off plus re-embeds; queries are forever but tiny.
const ingestTokens = chunks * avgTokens              // ~18.6M
const queryTokensPerYear = queriesPerDay * 365 * 20  // ~5.8M

At Atlas's scale the whole annual bill is small enough that cost should not decide this. At ten million chunks it decides it outright, and the gap between the cheapest and most expensive credible option is roughly an order of magnitude.

4. Dimensions, and what they cost downstream

Covered in chapter 38: storage, memory and comparison cost scale linearly, quality does not. Prefer the smaller variant unless the bake-off shows a real gap.

The bake-off

// scripts/embedding-bakeoff.ts
// Four candidates, one sample corpus, one test set. Two hours including the
// embedding time, and it ends the argument.
const CANDIDATES = [
  { id: 'a-small', dimensions: 1536, prefix: null },
  { id: 'a-large', dimensions: 3072, prefix: null },
  { id: 'b-base',  dimensions: 768,  prefix: { doc: 'passage: ', query: 'query: ' } },
  { id: 'c-self',  dimensions: 1024, prefix: { doc: 'passage: ', query: 'query: ' } },
]

export async function bakeOff() {
  // Same sample as chapter 31's chunking experiment, for comparability.
  const corpus = await sampleCorpus({ documents: 300, mustInclude: testSetDocuments() })

  for (const model of CANDIDATES) {
    await rebuildIndex(corpus, { model })          // into a scratch schema
    const scored = await scoreRetrieval(cases, { model, limit: 50 })

    console.table([{
      model: model.id,
      recall12: round(scored.recall12),
      recall50: round(scored.recall50),
      mrr: round(scored.mrr),
      // ⚠️ Report the hard subset separately. An average hides exactly the
      // questions that distinguish models.
      recallCodes: round(scored.subset('contains_code').recall12),
      recallNegation: round(scored.subset('negation').recall12),
      msPerQuery: scored.medianLatencyMs,
      indexMb: await indexSizeMb(),
    }])
  }
}

Representative results

Atlas's policy and specification sample, 34 cases:

Model	Dim	recall@12	recall@50	MRR	codes	negation	ms	index
a-small	1536	0.89	0.94	0.59	0.61	0.52	41	273 MB
a-large	3072	0.91	0.96	0.63	0.64	0.55	58	546 MB
b-base	768	0.90	0.95	0.64	0.58	0.51	36	137 MB
c-self	1024	0.86	0.93	0.57	0.55	0.49	22	182 MB

Four readings, and the fourth is the one that matters most.

The spread is two points of recall. On 34 cases that is under one question. None of these models is meaningfully better than the others on this corpus.
b-base at 768 dimensions matches a-small at 1536 and has the best MRR, at half the index size. Dimensions are not buying quality here.
a-large costs double for +0.02. Not worth it at this scale; possibly worth it at ten million chunks where the two points compound across more near-neighbours.
Every model is bad on codes (0.55–0.64) and worse on negation (0.49–0.55). That is not a model-selection problem. No choice in this table fixes it, and both are solved elsewhere — codes by hybrid retrieval, negation by sending the whole passage and measuring groundedness.

The honest conclusion from a table like this is usually: pick on operational grounds, because quality does not distinguish them. For Atlas that pointed at b-base — smallest index, best MRR, zero-retention available, in-region.

⚠️ Report the subsets. An aggregate of 0.89 against 0.90 says "equivalent". The code subset at 0.61 against 0.58 is the number that tells you the whole class of question is unsolved by any of them.

What changing it later costs

The reason this chapter exists before the corpus grows.

Cost	Atlas, 44k chunks	10M chunks
Re-embed everything	~$0.40, 40 min	Thousands of dollars, days
Rebuild the index	2 min	Hours
Re-tune the score floor	10 min	10 min
Re-run the chunking experiment	2 hours	2 hours
Invalidated retrieval measurements	All of them	All of them

The last row is the real cost, as it was for chunking. Every tuned number — floor, window size, hybrid weights, rerank threshold — was measured against one vector space.

Migrating without downtime

-- A second column, populated in the background, swapped atomically.
alter table chunks add column embedding_v2 vector(768);
create index concurrently chunks_embedding_v2
  on chunks using hnsw (embedding_v2 vector_cosine_ops);

// Backfill in batches, resumable by the same rule as chapter 26: advance
// only past durably finished work.
export async function backfillEmbeddings(batch = 500) {
  for (;;) {
    const { rows } = await db.query(
      `select id, content, heading_path from chunks
        where embedding_v2 is null order by id limit $1`, [batch])
    if (rows.length === 0) break

    const vectors = await embed(rows.map(prepareForEmbedding))
    await db.transaction(async (tx) => {
      for (const [i, row] of rows.entries()) {
        await tx.query(`update chunks set embedding_v2 = $2 where id = $1`,
          [row.id, toVector(vectors[i])])
      }
    })
    logger.info('embedding.backfill', { done: rows[rows.length - 1].id })
  }
}

// Then the cutover, which is a config flag and a measured comparison —
// never a deploy and a hope.
const column = config.embedding.activeColumn   // 'embedding' | 'embedding_v2'

// Before flipping: score the test set against BOTH columns and compare.
// After flipping: keep the old column for a week, so rollback is a flag.

Scoring both columns before the flip is the step that makes this safe, and it is possible only because retrieval evaluation is free and deterministic.

Self-hosting: when it is worth it

Three situations, and only three.

Situation	Why
Residency cannot be met by any provider	Chapter 17. No alternative.
Scale makes per-token cost dominate	Around tens of millions of chunks with frequent re-embeds
Fine-tuning on your own domain	Rarely worth it; measure against hybrid retrieval first

The third deserves scepticism. Fine-tuning an embedding model on domain data is a real technique and it typically buys less than hybrid search does, for considerably more work and an ongoing retraining obligation. Try chapter 43 first.

Self-hosting's real cost is not the GPU, it is that embedding becomes a service you operate with its own uptime, versioning and capacity — and an embedding service that is down stops ingestion and degrades every query.

The failure mode: the model that changed underneath you

A provider updates a hosted model behind a stable name. Documents ingested before the change and after it are in different vector spaces, and comparisons across the boundary are meaningless.

It is rare and it is silent: no error, no version change in your code, and a slow drift in quality as the proportion of newly-embedded chunks grows.

// A canary. Embed a fixed string on every ingest run and store the vector.
// If it moves, the model moved.
const CANARY = 'The quick brown fox jumps over the lazy dog.'

export async function checkEmbeddingStability() {
  const [current] = await embed([CANARY])
  const stored = await getStoredCanary()
  if (!stored) { await storeCanary(current); return }

  const drift = 1 - cosineSimilarity(current, stored)
  if (drift > 1e-6) {
    throw new IngestAnomaly(
      `embedding model drift detected: ${drift.toExponential(2)}. ` +
      `The hosted model may have changed. A full re-embed is required ` +
      `before this index is coherent again.`,
    )
  }
}

A dozen lines, run once per ingest, and it converts an undiagnosable quality decline into a specific message. ⚠️ Pin a dated model identifier where the provider offers one — that is the real fix, and the canary is what catches the case where you cannot.

Key points

Benchmarks produce a shortlist of four. Your test set produces the answer.
Eliminate on feasibility first: residency, retention, input limit against your p999 chunk, rate limit against the backfill.
Run the bake-off on the same sample as the chunking experiment, into a scratch schema. Two hours.
Report hard subsets — codes, negation — separately. Aggregates hide the questions that distinguish models.
Expect a two-point spread. When quality does not distinguish candidates, choose on operational grounds.
Codes and negation are bad in every model. Those are fixed by hybrid retrieval and by generation, not by model choice.
The real migration cost is invalidated measurements, not tokens.
Migrate with a second column, a background backfill and a config flag. Score both columns before flipping.
Self-host only for residency, extreme scale, or after hybrid retrieval has been tried.
Pin a dated model identifier, and run a canary embedding to catch silent drift.

Next: vector stores — the differences that are real, and the one everybody shops on that is not.