Chapter 40 · Retrieval

Vector stores: the real differences

They all compute the same distance over the same vectors, so retrieval quality is not the axis. Here are the six axes that actually differ, and the index parameters that matter more than the vendor.

11 min read

Chapter 37 put a zero in the gains table against "a different vector store". This chapter justifies that and then explains what you should choose on instead.

The claim is narrow and defensible: for a corpus of this size, every credible vector store computes the same cosine distance over the same vectors and returns the same ranking. Quality differences come from approximation parameters, which you control in all of them, not from the vendor.

What actually differs

Six axes, in the order they decide the question.

1. Whether you already run it

This one usually ends the discussion, and it should.

Atlas's company already runs Postgres. Adding pgvector is an extension and a migration. Adding a dedicated vector database is a new service with its own availability, backup, access control, upgrade path and on-call rotation — paid forever, for a quality difference of approximately zero at 44,387 chunks.

The best vector store is the database you already operate, until it measurably cannot do the job.

2. Joins against your own data

This is the one that is underrated and that bites hardest.

Every query in this series joins chunks to documents for the permission filter. In Postgres that is one query:

select c.id, c.content, 1 - (c.embedding <=> $1) as score
  from chunks c
  join documents d on d.id = c.document_id
 where d.deleted_at is null
   and d.classification = any($2)
   and (d.account_id is null or d.account_id = any($3))
 order by c.embedding <=> $1
 limit 12;

In an external store the permission data has to be denormalised into the vector record's metadata, which reintroduces exactly the problem chapter 14 warned about — a copy of permission data that drifts — or you fetch candidates and filter afterwards, which is the leak-into-logs pattern chapter 13 rejected.

// ❌ The shape an external store pushes you toward.
const candidates = await vectorDb.query({ vector, topK: 100 })
const allowed = candidates.filter((c) => isPermitted(c.metadata, scope))
// Now: forbidden content is in process memory, and topK=100 might contain
// zero permitted results for a narrowly-scoped user.

That second comment is the practical failure. Pre-filtering in an external store is supported by most of them now and is genuinely a filter inside the search — but the quality of that support varies, and a store that applies the filter after retrieving topK will return nothing for a user whose permitted set is a small fraction of the corpus.

3. Index algorithm and its parameters

This is where real quality differences live, and they are yours to set.

-- HNSW: a navigable graph. Better recall per unit of latency, slower to
-- build, more memory. The default choice for a read-heavy corpus.
create index chunks_hnsw on chunks
  using hnsw (embedding vector_cosine_ops)
  with (m = 16, ef_construction = 64);

-- IVFFlat: inverted lists. Faster build, less memory, lower recall at the
-- same latency. Reasonable when the corpus is rebuilt frequently.
create index chunks_ivf on chunks
  using ivfflat (embedding vector_cosine_ops)
  with (lists = 210);     -- rule of thumb: sqrt(rows) for < 1M rows

Parameter	Effect	Trade
`m` (HNSW)	Graph connectivity	Higher = better recall, more memory, slower build
`ef_construction`	Build-time search width	Higher = better graph, slower build. Build-time only
`ef_search`	Query-time search width	Higher = better recall, slower query. Tune this.
`lists` (IVFFlat)	Number of clusters	More = faster query, lower recall
`probes` (IVFFlat)	Lists searched per query	More = better recall, slower query

-- ef_search is the dial you actually use, and it is per-session.
set hnsw.ef_search = 100;   -- default 40

// Measure it against the test set rather than guessing, the same way as
// everything else in this section.
for (const ef of [20, 40, 80, 160, 320]) {
  await db.query(`set hnsw.ef_search = ${ef}`)
  const scored = await scoreRetrieval(cases, { limit: 50 })
  console.log(ef, round(scored.recall50), scored.medianLatencyMs)
}
// 20   0.89   7ms
// 40   0.93   9ms
// 80   0.94  13ms
// 160  0.94  21ms
// 320  0.94  38ms

That table is the point of the chapter. Four points of recall between ef_search 20 and 80, for four milliseconds — a larger quality difference than any vendor choice in chapter 39's bake-off, available for free, in a setting most teams never touch.

⚠️ Note the plateau. Above 80 nothing improves and latency keeps rising. The useful value is the knee, and it is corpus-specific.

4. Filtered search behaviour

The axis that matters most for a permission-scoped system, and the one benchmarks never cover.

-- A sales rep owning 40 of 4,000 accounts. Their permitted set is roughly 2%
-- of the corpus. What does the index do?
select c.id from chunks c join documents d on d.id = c.document_id
 where d.classification = 'account' and d.account_id = any($accounts)
 order by c.embedding <=> $1 limit 12;

An approximate index searches a neighbourhood of the query vector. If almost nothing in that neighbourhood passes the filter, the search either returns fewer results than asked for or degrades to a scan.

-- Postgres can choose between two plans, and it is worth checking which.
explain (analyze) <the query above>;
-- 'Index Scan using documents_retrieval' then a sort: the filter was
--   selective enough to drive the plan. Exact and fast. Good.
-- 'Index Scan using chunks_hnsw' with a high 'Rows Removed by Filter':
--   the vector index drove it and most candidates were discarded. This is
--   where you silently get fewer than `limit` results.

// The defensive pattern: over-fetch, filter, and detect under-filling.
const OVER_FETCH = 4
const rows = await search({ ...options, limit: options.limit * OVER_FETCH })
const results = rows.slice(0, options.limit)

if (rows.length < options.limit && scope.accountIds.length < 100) {
  // Not necessarily wrong — a narrow scope may genuinely have few chunks —
  // but worth logging, because it is also how approximate search fails.
  logger.warn('retrieval.underfilled', {
    requested: options.limit, got: rows.length, scopeSize: scope.accountIds.length,
  })
}

5. Operational shape

Concern	Postgres + pgvector	Dedicated store
Backup	Your existing backup	A second thing to back up
Transactions	Chunks and documents in one transaction	Two systems, eventually consistent
Access control	Your existing roles	A second access model
Upgrades	Extension version	A service upgrade path
Scaling limit	Hundreds of millions of rows with work	Designed for billions

The transaction row is worth noting against chapter 23's one-transaction-per-document rule. With an external store, a document row and its vectors are written to two systems, and a failure between them leaves the partially-indexed document that rule exists to prevent.

6. Scale

The honest threshold: below about ten million vectors, Postgres is comfortable on ordinary hardware. Between ten and a hundred million it needs attention — memory for the index, partitioning, possibly quantisation. Above that, a purpose-built store starts to earn its operational cost.

Atlas is at 44,387. Most internal corpora are between ten thousand and a few million, which is to say most of these decisions are being made at a scale where they do not matter.

Choosing, as a procedure

// Three questions. They resolve it in most cases.
if (corpusSize < 10_000_000 && alreadyRunning('postgres')) use('pgvector')
else if (needsPerDocumentAcl || joinsAgainstOperationalData) use('pgvector')
else if (corpusSize > 100_000_000) use('a purpose-built store')
else measure()   // the bake-off below

// If you must measure: same shape as the embedding bake-off, and the columns
// are operational rather than quality-based, because quality will tie.
const AXES = [
  'recall@12 at a fixed p95 latency',   // expect a tie
  'p95 latency at a fixed recall',      // the real quality-adjacent number
  'recall under a 2%-selective filter', // the one that differs
  'index build time for a full re-ingest',
  'memory at corpus size',
  'cost per month at corpus size',
]

The third axis is the one to weight heavily for any permission-scoped system, and it is the one no public benchmark reports.

The failure mode: the index that was never rebuilt

Atlas's HNSW index is created during the first migration against an empty table. The first full ingest then inserts 44,387 rows, each one incrementally added to the graph.

An HNSW graph built incrementally from empty is worse than one built over existing data, because early insertions had no neighbourhood to connect into. Recall is a point or two lower, permanently, and nothing indicates it.

-- Build the index AFTER the bulk load, not before it.
-- 1. Ingest with no vector index. Inserts are much faster too.
drop index if exists chunks_hnsw;
-- 2. Load 44,387 rows.
-- 3. Build once, over the full data, in parallel.
set maintenance_work_mem = '2GB';
set max_parallel_maintenance_workers = 4;
create index chunks_hnsw on chunks using hnsw (embedding vector_cosine_ops)
  with (m = 16, ef_construction = 128);

For Atlas that was 2 minutes of build time and +0.015 recall@12 — comparable to the entire difference between the best and worst model in the bake-off, from changing the order of two operations.

// Make it part of the full-ingest path rather than a thing to remember.
export async function fullIngest(sources: SourceDefinition[]) {
  await db.query(`drop index if exists chunks_hnsw`)
  for (const source of sources) await ingestSource(source)
  await db.query(`set maintenance_work_mem = '2GB'`)
  await db.query(`create index chunks_hnsw on chunks using hnsw (...)`)
  await db.query(`analyze chunks`)        // ⚠️ or the planner has no stats
}

The analyze is not optional. A freshly loaded table with stale statistics gives the planner wrong row estimates, and wrong estimates are how it decides to sequential-scan 44,387 rows instead of using the index you just spent two minutes building.

Key points

Every credible store computes the same distance over the same vectors. Quality differences come from parameters you control.
The best vector store is the database you already operate, until it measurably cannot do the job.
Joins against your own data is the underrated axis. An external store pushes you toward denormalised permissions or post-filtering, and both were rejected earlier in this series.
ef_search is the dial that matters. Four points of recall for four milliseconds, with a corpus-specific knee.
Filtered search under a narrow permission scope is where stores genuinely differ, and no public benchmark reports it. Over-fetch and log under-filling.
Below ten million vectors these decisions do not matter. Most internal corpora are below it.
Build the vector index after the bulk load, not before. Incremental build from empty costs permanent recall.
Run analyze after a bulk load, or the planner ignores the index.

Next: your first vector search — and why the results disappoint in four specific, fixable ways.