Chapter 150 · Interface and adoption

What to build next, and how to decide

The last chapter. A decision procedure driven by the instruments the series has built, the four states a system can be in and what each one needs — and what Atlas would do differently from chapter one.

8 min read

A hundred and forty-nine chapters have built a system and the instruments to see it. This chapter is what to do on an ordinary Monday with all of it running, and it closes the series.

The question "what should we build next" has a bad default answer — whatever was asked for most loudly, most recently — and a good one, which is that the instruments already contain the answer.

Ask the instruments, in this order

// Each returns a candidate, and they are ordered by how much evidence
// stands behind what they return.
const SOURCES_OF_TRUTH = [
  { q: 'What is failing?',        from: 'chapter 107 attribution of eval failures' },
  { q: 'What is being refused?',  from: 'chapter 111 refusal rate by reason' },
  { q: 'Who stopped using it?',   from: 'chapter 148 cohort retention, segmented' },
  { q: 'What is being asked for?',from: 'chapter 149 unmet-demand query' },
  { q: 'What is drifting?',       from: 'chapter 112 weekly run' },
  { q: 'What costs the most?',    from: 'chapter 130 spend by decile' },
]

// Run all six quarterly. It takes an afternoon, and it has produced
// a different priority from the team's intuition three times out of
// four — most memorably when 51% of failures turned out to be
// retrieval while everyone was working on the prompt.

Four states, four priorities

// A system is in one of these. The priority follows from the state,
// and most arguments about what to build next are really disagreements
// about which state you are in.
type State = 'not-trusted' | 'trusted-narrow' | 'trusted-broad' | 'drifting'

State	Signal	Build
Not trusted	Week-12 retention under 30%; citation opens high and flat	Quality and expectations. Nothing else.
Trusted, narrow	Retention good; penetration under 50%; high out-of-scope	Interface and scope. Not quality.
Trusted, broad	Retention and penetration both good	Depth: actions, workflows, the expensive decile
Drifting	Scores falling with no deploy	Stop building. Chapter 112.

// The common mistake is building for the wrong state. A team with 41%
// retention and 11% warehouse penetration is in "trusted, narrow" and
// will spend a quarter improving groundedness from 0.96 to 0.97 — real
// work, correctly executed, against a problem nobody has.

Atlas at month twelve was trusted-narrow, and the quarter that followed shipped an embedded interface, a live WMS tool and a rewritten starter set. No model work at all, and it moved more numbers than any quarter before it.

The ratio worth holding

// Across Atlas's second year, by engineering weeks.
const WHERE_THE_TIME_WENT = {
  retrieval: 0.27,
  data: 0.22,          // ingestion, parsing, freshness, document quality
  interface: 0.19,
  evaluation: 0.12,
  tools: 0.11,
  prompt: 0.05,
  model: 0.04,
}

Nine percent on the model and the prompt combined. That ratio is the most compressed statement of this entire series: the model is the part you do not build, and almost everything that determines whether the system is good is somewhere else.

What Atlas would do differently

An honest list, written as the thing to copy from a hundred and fifty chapters.

const DIFFERENTLY = [
  'Build the eval suite in week one, at 40 cases. It was built in month ' +
    'four, and everything before it was unmeasured.',

  'Put the audit log’s per-document access index in from the start. It ' +
    'cannot be backfilled, and it is what made the one real incident ' +
    'scopeable.',

  'Thread the caller identity through every tool signature on day one. ' +
    'Retrofitting it found a gap on the resumption path that had been ' +
    'open for months.',

  'Write the scope sentence before writing the first prompt. It belonged ' +
    'in the interface and ended up there a year late.',

  'Ship structured starters instead of an empty box. Two thirds of the ' +
    'company formed their impression from the empty box.',

  'Measure the escalation rate before launch, not after. Without the ' +
    'before, the after is an assertion.',
]

Four of six are instrumentation and one is a sentence of copy. None is a model decision.

When to stop

// A system that works does not need continuous investment, and
// pretending otherwise is how a good internal tool becomes an
// expensive one. Three signals that the right move is maintenance.
const STOP_BUILDING = [
  'Retention and penetration flat at a good level for two quarters.',
  'The unmet-demand query returns nothing above the threshold.',
  'The last three changes all had intervals spanning zero.',
]

The third is the clearest and the hardest to act on. Chapter 108 made "no evidence of change" a legitimate outcome; three of them in a row is the system telling you it is done, and the honest response is to move to maintenance — the weekly drift run, the ingestion report, the quarterly instrument review — and put the engineering somewhere it will move a number.

The series, in one page

// What the twelve sections actually argued, compressed.
const SERIES = {
 'Decide whether you need an agent at all. Most questions are search.',
 'The permission model is a data-layer decision, made before retrieval.',
 'Ingestion is the attack surface and the quality ceiling.',
 'Retrieval is its own system, measured on its own. 51% of failures.',
 'The prompt is an artifact. Citations make an answer checkable.',
 'A tool surface is a security boundary with typed arguments.',
 'You give up the call graph. Make it terminate, resume and be legible.',
 'Memory is a liability by default. 71% of sessions are single-turn.',
 'Without a suite, every claim about quality is an opinion.',
'Assume the model is manipulated. Build so it cannot reach anything.',
'Most of the system is deterministic. That is where the grip is.',
'The interface decides what gets asked, and whether anyone returns.',
}

The hardest part was never the model. It was knowing what the company's documents actually say, who is allowed to see them, whether the answer was right, and whether anyone trusted it enough to use.

Where to go from here

// Three honest next steps, depending on where you are.
const NEXT = {
  haveNothing: 'Chapter 7: pick the smallest slice with a real user and a ' +
    'measurable outcome. Chapter 10 defines what success means before you ' +
    'write code.',
  haveAPrototype: 'Chapter 101. Build forty eval cases this afternoon. ' +
    'Everything else is easier once a number exists.',
  haveSomethingRunning: 'Run the six instrument questions above. The ' +
    'priority you get back will probably not be the one you expected.',
}

Atlas is fictional and every number in this series came from building systems of this shape. The specific figures will differ in your company; the ratios — 51% retrieval, 9% model and prompt, 71% single-turn, 31% of injection attempts landing and 5% mattering — have been stable enough across different corpora to be worth planning against until your own measurements replace them.

That is the last recommendation of the series: replace every number here with your own as soon as you can measure it. The reason this was written as one worked example with real figures rather than as a set of principles is that a principle cannot be checked, and a number can.

Key points

Six instruments already contain the answer to what to build next. Run them quarterly; it takes an afternoon.
A system is in one of four states, and most disagreements about priorities are disagreements about which state it is in.
Not trusted: quality and expectations. Trusted but narrow: interface and scope. Trusted and broad: depth. Drifting: stop and diagnose.
Building for the wrong state means executing real work correctly against a problem nobody has.
Atlas's second year: 27% retrieval, 22% data, 19% interface, 12% evaluation, 11% tools, 9% prompt and model combined.
The six things to do differently are four pieces of instrumentation, one identity decision and one sentence of copy. None is a model decision.
Know when to stop. Three consecutive changes with intervals spanning zero is the system saying it is done.
Replace every number in this series with your own. A principle cannot be checked and a number can.

That is the end of the series. If you build one of these, the chapter worth re-reading first is chapter 101 — not because evaluation is the most interesting part, but because every other chapter's advice becomes checkable once you have forty cases and a number.