Chapter 149 · Interface and adoption

Knowing when to expand scope, and when not to

Every expansion adds a source, a permission model and a failure mode, and dilutes the thing that already works. Five tests, the ones Atlas said no to, and the cost of a yes.

8 min read

A system that works attracts requests. Chapter 148 left the warehouse at 11% penetration and the obvious conclusion — index the warehouse management system — is the one this chapter exists to slow down.

The thing nobody counts: every expansion makes the existing system slightly worse. A new source adds candidates to every retrieval, a new permission model adds a way to leak, a new tool adds a way for the model to choose wrong, and new scope widens what users expect and therefore what counts as a failure.

What an expansion actually costs

// Atlas's measured cost of adding one source, from the two it did add.
const COST_OF_A_SOURCE = {
  engineering: '2-4 weeks: connector, parser, chunking, permission mapping',
  ongoing: 'a parse-failure rate to watch, a freshness expectation, ingestion lag',
  retrieval: 'recall on EXISTING questions fell 0.94 -> 0.92 on the first ' +
             'expansion, before reranking was retuned',
  evaluation: '15-20 new eval cases, or the new source is unmeasured',
  security: 'a permission model to map (chapter 14) and a trust tier (chapter 117)',
  support: 'a new category of question that can now fail',
}

The retrieval line is the one that surprises teams. Adding documents adds competition: a question about returns now has to beat passages from a source that did not exist last month, and chapter 62's near-miss problem gets a new supply of near-misses.

An expansion is not additive. It is a trade of some existing quality for some new coverage, and the trade is worth making only when you can say which questions get worse.

Five tests

// Applied before any work. A request failing two of these does not get
// built, and the honest answer is usually "not yet".
const TESTS = [
  { id: 'demand',    ask: 'How many real questions, from the logs, failed for want of this?' },
  { id: 'fit',       ask: 'Is this a retrieval-and-answer problem, or a different product?' },
  { id: 'authority', ask: 'Is this the system of record, or a copy of one?' },
  { id: 'ownership', ask: 'Who owns this data, and have they agreed?' },
  { id: 'failure',   ask: 'What is the cost of a wrong answer here, and is it higher than today?' },
]

-- The demand test is a query, not an opinion. Out-of-scope refusals
-- already carry their reason (chapter 111), so the backlog of unmet
-- demand is sitting in the log.
select regexp_replace(lower(question), '\d+', 'N', 'g') as shape,
       count(*) as asks, count(distinct user_id) as people
  from audit_run
 where refusal_reason = 'not-in-corpus' and at > now() - interval '90 days'
 group by 1 having count(distinct user_id) >= 5
 order by 2 desc limit 20;

Atlas's top unmet demand over a quarter:

Shape	Asks	People
stock location for N	412	31
freight surcharge for N	86	19
who owns account N	71	24
commission on order N	44	6
salary band for N	12	9

What Atlas said yes to, and no to

// "who owns account N" — YES. 71 asks, 24 people, the data is in the
// CRM which is the system of record, the permission model is the one
// already mapped, and a wrong answer costs an email to the wrong
// person. Two weeks, and it passed all five tests cleanly.

// "stock location for N" — the biggest demand by far, and NO, twice.
//
//   fit:       the question is asked on a phone, in a noisy building,
//              with gloves on. It is a scanner interaction, not a
//              typed question. A different product.
//   authority: the WMS is live and changes by the minute. Indexing it
//              produces a stale copy of a system of record — chapter
//              91's routing rule, at system scale.
//
// What shipped instead: a tool that queries the WMS live and returns
// one field. Four days, no ingestion, no index, no freshness problem.
// Warehouse penetration went 11% -> 38%.

That outcome is the pattern worth taking from this chapter. The request was "index the warehouse system"; the demand was answerable with a single live tool call, and chapter 74 had already built the mechanism. Most expansion requests arrive phrased as corpus expansion and are satisfied by a tool.

// "commission on order N" — NO. 44 asks from 6 people, and the
// failure test decides it: a wrong commission figure is a pay dispute.
// It is also finance's system of record. Routed to a link, not an
// answer.

// "salary band for N" — NO, and this one is a policy decision rather
// than a product one. Chapter 16 excluded personal data deliberately
// and 12 asks does not reopen it.

The expansions that are not sources

// Three other shapes of request, each with a different answer.
const OTHER = {
  newUserGroup: 'cheapest expansion available. New people, same corpus, ' +
    'same tools. Usually a permission mapping and a starter set.',
  newAction: '⚠️ the most expensive. A write tool is chapters 71, 72 and 73 ' +
    'again, plus a hazard count that must stay at zero.',
  newModality: 'voice, mobile, a messaging platform. Measured in chapter 141; ' +
    'often cheaper than a source and often higher impact.',
}

Atlas's highest-return expansion in its second year was none of the requested sources: it was putting the assistant inside the order system, which chapter 141 measured at 58% weekly active against 31% standalone. No new data, no new tools, nearly twice the usage.

Saying no well

// A refused request should leave the requester with something.
const NO = [
  'The number. "19 people asked this 86 times" or "6 people asked 44 times" ' +
    'makes the decision legible rather than arbitrary.',
  'The test it failed, named.',
  'The alternative, if there is one. The WMS tool came from a no.',
  'The condition under which it becomes a yes.',
]

The fourth turns a no into a deferral with a trigger, and it is what keeps requesters bringing requests. Atlas's freight surcharge request was declined at 86 asks with the stated condition "when the surcharge schedule has an owner who maintains it" — and six months later finance appointed one and it shipped.

The scope that should shrink

// Rarely considered, occasionally right. Atlas removed one source.
const REMOVED = {
  what: 'an archive of email threads from before 2024',
  why: 'it produced 0.4% of citations, 11% of ungrounded answers, and ' +
       'the worst chunks-per-document ratio of any source (chapter 138)',
  effect: 'recall@12 on the existing test set went 0.92 -> 0.94',
}

Removing a source improved retrieval measurably. A source that contributes few citations and many distractors is a net negative, and the only way to know is to measure citations and ungrounded answers by source — which chapter 110's bySource breakdown exists to do.

Key points

Every expansion makes the existing system slightly worse. Adding documents adds competition, and the first Atlas source cost 0.02 of recall on existing questions.
Count the real cost: 2-4 weeks of engineering, a permanent parse-failure and freshness burden, 15-20 eval cases, a permission mapping, and a new category of failure.
Five tests: demand, fit, authority, ownership, and the cost of being wrong.
The demand test is a query. Out-of-scope refusals carry their reason, so unmet demand is already in the log.
Most expansion requests arrive phrased as corpus expansion and are satisfied by a tool. Atlas's largest demand needed a four-day live lookup, not an index, and took warehouse penetration from 11% to 38%.
Refuse anything whose system of record is live and changing. Indexing it produces a stale copy.
Refuse where a wrong answer is a pay dispute or a legal exposure, regardless of demand.
A new user group is the cheapest expansion; a new write action is the most expensive. A new interface surface is often both cheaper and higher impact than a source.
Say no with the number, the test it failed, an alternative, and the condition that makes it a yes. Atlas's deferred request shipped six months later when the condition was met.
Consider removing a source. Dropping one that produced 0.4% of citations and 11% of ungrounded answers raised recall from 0.92 to 0.94.

Next: what to build next, and how to decide — the last chapter.