Chapter 45 · Retrieval

Metadata filters and permission-aware retrieval

The filter has been in every query since chapter 13. Here is what it does to an approximate index, how to keep it fast under a narrow scope, and the other filters worth having.

11 min read

The permission filter has been in every query in this section. This chapter is about what it costs, how it interacts with an approximate index, and the other metadata filters that earn their place.

It assumes the permission model and the metadata schema.

The selectivity problem

A finance user sees everything: the filter removes nothing and the index works as designed. A sales rep owning 40 of 4,000 accounts has a permitted set that is roughly 2% of the corpus, and the neighbourhood the index explores may contain almost nothing they are allowed to see.

-- See which plan you get. Run it as a narrow user, not as yourself.
explain (analyze, buffers)
select c.id from chunks c join documents d on d.id = c.document_id
 where d.deleted_at is null
   and d.classification = 'account'
   and d.account_id = any(array['4471','4480','4492'])
 order by c.embedding <=> $1 limit 12;

Two plans are possible and both can be right:

-- Plan A: the filter drives it.
Limit
  -> Sort (actual rows=12)
      -> Nested Loop
          -> Index Scan using documents_retrieval on documents d
          -> Index Scan using chunks_doc on chunks c
-- Exact, and fast when the permitted set is small. ✅

-- Plan B: the vector index drives it.
Limit
  -> Index Scan using chunks_hnsw on chunks c
      Filter: (d.classification = ...)
      Rows Removed by Filter: 1,847
-- ⚠️ Approximate, and may return fewer than 12 when the filter is selective.

Plan A is what you want for a narrow scope and Plan B for a broad one, and Postgres usually chooses correctly — if the statistics are current. This is where the analyze from chapter 40 stops being hygiene and starts being correctness.

-- Give the planner what it needs to estimate selectivity on the pair.
create statistics documents_scope_stats (dependencies, ndistinct)
  on classification, account_id from documents;
analyze documents;

Partitioning by the filter

For a corpus where most queries are account-scoped, partitioning turns the selectivity problem into a smaller search space rather than a filtered one.

-- Partition by classification: the three classes have very different access
-- patterns and very different sizes.
create table chunks (
  id bigserial, document_id text not null, classification text not null, ...
) partition by list (classification);

create table chunks_public   partition of chunks for values in ('public_internal');
create table chunks_account  partition of chunks for values in ('account');
create table chunks_financial partition of chunks for values in ('financial');

-- One HNSW index per partition. A warehouse supervisor's query touches only
-- chunks_public, which is a quarter of the corpus.
create index on chunks_public using hnsw (embedding vector_cosine_ops);
create index on chunks_account using hnsw (embedding vector_cosine_ops);
create index on chunks_financial using hnsw (embedding vector_cosine_ops);

⚠️ This duplicates classification onto chunks, which chapter 33 argued against. The exception is defensible here because it is a partition key rather than a permission source — but it must be maintained by the same code path, and a check is worth the three lines:

-- The partition key cannot be allowed to disagree with the document.
create or replace function chunks_classification_matches() returns trigger as $$
begin
  if new.classification is distinct from
     (select classification from documents where id = new.document_id) then
    raise exception 'chunk classification does not match its document';
  end if;
  return new;
end $$ language plpgsql;

Partitioning is worth it above roughly a million chunks with skewed access. Below that it is complexity for a few milliseconds.

The other filters worth having

Permissions are mandatory. Four more are optional and each needs justification, because every filter is a chance to exclude the answer.

Source routing

// A question about stock is not answered by policy documents. Narrowing the
// source improves precision — and excludes the answer when the routing is wrong.
const SOURCE_HINTS: Array<[RegExp, string[]]> = [
  [/\b(stock|on hand|inventory|availability)\b/i, ['product_specs']],
  [/\b(clause|section|contract|agreement|amendment)\b/i, ['customer_contracts']],
  [/\b(discount|tier|pricing|margin)\b/i, ['pricing_workbook', 'customer_contracts']],
]

// ⚠️ A hint, never a restriction. Boost the hinted sources; do not remove
// the others, because a wrong classification would otherwise be unrecoverable.

Measured on Atlas: hard source filtering raised precision@6 from 0.38 to 0.44 and dropped recall from 0.97 to 0.89. Boosting instead of filtering raised precision to 0.41 with no recall loss. Boosting won, and the general principle is worth stating:

A filter that can be wrong should be a boost. Only filters that must never be wrong — permissions — belong in the where clause.

Date and recency

-- Only when the question implies a period. A blanket recency filter hides
-- the policy that has not changed since 2019 and is still in force.
and ($dateFrom::date is null or d.source_updated_at >= $dateFrom)

Document kind

-- Useful for "show me the table of..." questions, which chapter 15's table
-- chunks serve directly.
and ($kinds::text[] is null or c.kind = any($kinds))

Excluding superseded

-- Already handled by deleted_at, from chapter 35. Mentioned because it is
-- easy to write a new query path that forgets it — which is what the grep
-- test in chapter 13 is for.
and d.deleted_at is null

Measuring the filter, not just the results

The leakage gate asserts that forbidden chunks never enter the window. Two more checks belong in the retrieval layer itself.

// 1. Did a narrow scope get a full window? Under-filling is how approximate
//    search fails silently under a selective filter.
if (results.length < limit && candidates.length === candidateLimit) {
  // The candidate set was full and the window is not: the filter removed
  // them, which means the index neighbourhood was mostly forbidden content.
  logger.warn('retrieval.filter_starved', {
    scopeAccounts: scope.accountIds.length, got: results.length, limit,
  })
}

// 2. Score parity across scopes. The same question asked by a broad user and
//    a narrow one should return the narrow user's permitted subset in the
//    same relative order — not a different ranking.
test('narrow scope preserves ranking of permitted chunks', async () => {
  const broad = await retrieve(question, { scope: FINANCE, limit: 50 })
  const narrow = await retrieve(question, { scope: ONE_REP, limit: 50 })

  const permittedFromBroad = broad.results.filter((r) => isPermitted(r, ONE_REP))
  // The first few should match. A divergence means the approximate search
  // explored a different neighbourhood, not that permissions differ.
  expect(narrow.results.slice(0, 5).map((r) => r.id))
    .toEqual(permittedFromBroad.slice(0, 5).map((r) => r.id))
})

That second test is the one that catches the approximate-index failure. It passed on Atlas at ef_search = 100 and failed at ef_search = 20 — which is a concrete demonstration that the index parameter from chapter 40 is a correctness setting for narrow scopes, not only a quality one.

The cost, measured

Scope	Permitted share	p50 latency	recall@12	Plan
Finance (all)	100%	12 ms	0.97	B: vector index
Warehouse (public only)	24%	14 ms	0.97	B: vector index
Rep, 40 accounts	2.1%	9 ms	0.96	A: filter-driven
Rep, 3 accounts	0.2%	4 ms	0.98	A: filter-driven

The narrow scopes are faster, because the filter-driven plan examines a small set exactly rather than approximating over a large one. The widespread assumption that permission filtering is a performance cost is backwards at this scale — it is a performance cost only in the middle band, where neither plan is clearly right.

The failure mode: the filter that is correct and the window that is empty

A rep asks a general policy question — "what is the returns window?" — and gets nothing.

Trace it. Their scope is ['public_internal', 'account'] with 40 account ids. Policy documents are public_internal, so they are permitted. The filter is correct.

The problem is the candidate set. Hybrid retrieval fetched 50 per method. This rep's 40 accounts include three large ones with 900 contract chunks between them, and the question's wording — "returns window" — matches contract return-clause language strongly. All 50 vector candidates and all 50 keyword candidates are contract chunks from their own accounts. Permitted, relevant-looking, and not the policy.

The policy chunk ranks 63rd and is never a candidate.

// The fix: stratify candidate generation across sources, so one source
// cannot consume the entire candidate budget.
export async function stratifiedRetrieve(question: string, options: RetrieveOptions) {
  const sources = await permittedSources(options.scope)
  const perSource = Math.ceil(options.candidateLimit / sources.length)

  const lists = await Promise.all(
    sources.map((source) =>
      hybridRetrieve(question, { ...options, source, limit: perSource }),
    ),
  )
  // Fuse across sources with RRF, exactly as across methods in chapter 43.
  return reciprocalRankFusion(lists.map((l) => l.results))
}

	recall@6	precision@6	Affected questions
Flat candidates	0.97	0.38	3 of 34 returned no policy chunk
Stratified by source	0.99	0.37	0

Two points of recall, a trivial precision cost, and — more importantly — it removes a failure that only appears for some users. A flat candidate set makes retrieval quality depend on the shape of the asker's permissions, which is the kind of bug that is invisible in testing because the test user is usually an administrator.

Test retrieval as a narrowly-scoped user. An administrator's results are the easy case and the one every default setup produces.

Key points

A narrow permission scope changes which query plan is correct. Keep statistics current, and add extended statistics on the filter columns.
Partitioning by classification helps above about a million chunks with skewed access. It duplicates the classification, so constrain it against the document.
A filter that can be wrong should be a boost. Only permissions belong in the where clause.
Hard source filtering cost six points of recall for six points of precision; boosting got most of the precision for free.
Log under-filled windows. Filter starvation is how approximate search fails silently under a selective filter.
Test that a narrow scope preserves the ranking a broad scope gives for permitted chunks. It fails at low ef_search, which makes that parameter a correctness setting.
Narrow scopes are often faster, not slower. Permission filtering as a performance cost is backwards at this scale.
Stratify candidate generation across sources, or one source can consume the entire budget for users whose permissions happen to be shaped that way.
Test as a narrowly-scoped user. Administrator results are the easy case.

Next: query rewriting and expansion — fixing the question before it reaches the index.