Chapter 135 · Production

Graceful degradation when the model is down

Four levels between working and gone, the retrieval-only mode that needs no model at all, and the measurement showing users prefer an honest degraded answer to a slow correct one.

7 min read

Chapter 134 ended with Atlas choosing degradation over a second provider. This is what that means in practice, and the argument is stronger than "it is cheaper".

The insight that makes it work: most of the system does not need the model. Chapter 127 drew the deterministic prefix and suffix around a non-deterministic middle. When the middle is unavailable, the prefix is still a working search engine over the company's documents with correct permissions — which is more than most companies had before the assistant existed.

Four levels

type Level = 'full' | 'reduced' | 'retrieval-only' | 'read-only-corpus'

Level	Model	Tools	What the user gets
full	Yes	Yes	Normal
reduced	Cheap only	Read only	Shorter answers, no actions
retrieval-only	No	No	Ranked passages with citations
read-only-corpus	No	No	Document search by title and keyword

// The transitions are automatic on signals and manual by flag. Chapter
// 125's kill switches are the manual path; these are the automatic one.
const TRIGGERS = {
  reduced:        'breaker half-open, or global spend >80% of ceiling',
  retrievalOnly:  'breaker open, or model error rate >30% over 2 min',
  readOnlyCorpus: 'vector index unavailable',
}

Retrieval-only is the level worth building

// It is not a fallback message. It is the retrieval stack, rendered.
export async function retrievalOnly(question: string, ctx: Ctx): Promise<Degraded> {
  const chunks = await retrieve(question, ctx)      // same filter, same permissions
  return {
    mode: 'retrieval-only',
    notice: 'The assistant cannot write an answer right now. These are the passages it would have used.',
    passages: chunks.slice(0, 5).map((c) => ({
      heading: c.headingPath.join(' > '),
      document: c.documentTitle,
      excerpt: c.content,
      href: deepLink(c),
      updatedAt: c.updatedAt,
    })),
  }
}

Three properties make it genuinely useful rather than a consolation prize:

// 1. It is permission-correct. The chapter 119 filter is in the query,
//    so degraded mode cannot leak what full mode would not.
//
// 2. It carries the heading path and the deep link — chapter 62's
//    findings about what makes a citation usable apply unchanged.
//
// 3. It is honest about what happened, naming the limitation rather
//    than producing a vaguer answer and hoping.

Atlas measured it during a 38-minute provider outage: 61% of users who hit retrieval-only found what they needed from the passages, and the escalation rate during the window was lower than a normal Monday. The degraded mode was, for simple lookups, close to as good — which says something uncomfortable and useful about how many questions were ever generation problems.

Honest degraded beats slow correct

// The experiment, run deliberately after the outage. During simulated
// provider slowness, half of users waited and half were degraded at 6 s.

	Waited (p50 14 s)	Degraded at 6 s
Found what they needed	78%	71%
Rated the experience acceptable	42%	81%
Escalated to a human	19%	16%

Seven points worse on the outcome and thirty-nine points better on acceptability. The reason is visible in the free-text comments: a user who waits fourteen seconds and gets a good answer has still spent fourteen seconds not knowing whether anything was happening, and concludes the system is unreliable.

A degraded answer that explains itself is treated as the system working. A slow correct answer is treated as the system failing.

Partial degradation inside a run

// The whole system is rarely down. Usually one dependency is, and the
// run should continue without it rather than fail.
const OPTIONAL = {
  memory:     'skip recall; the answer is slightly less tailored',
  rerank:     'use vector order; recall@12 0.94 -> 0.91',
  queryRewrite: 'use the raw question; recall 0.94 -> 0.88 on pronouns',
  citationValidation: '⚠️ NOT optional — chapter 122 is a safety control',
}

// Each optional dependency has a timeout well below the run budget and
// a stated fallback. The failure is recorded, not surfaced.
const memories = await withTimeout(recall(ctx), 300).catch(() => {
  metrics.increment('degraded.memory_skipped')
  return []
})

The last line of the table is the important one. Degradation must never remove a safety control — if citation validation cannot run, the correct behaviour is to fail the response, not to serve it unvalidated. That distinction is what separates degradation from quietly turning off the checks under load, which is when they were most needed.

Telling the user

// Three rules, learned by getting each wrong first.
const NOTICE_RULES = [
  'Say what is unavailable, not that "an error occurred".',
  'Say what they are getting instead.',
  'Do not estimate a recovery time you do not know.',
]

<Banner tone="warning">
  The assistant cannot write answers at the moment. You can still search
  the documents below, and everything you can normally see is here.
</Banner>

The second clause of that banner — everything you can normally see is here — answers the question users actually ask during a degradation, which is whether the results are incomplete. Without it, people assume degraded means partial, and re-run the same search repeatedly.

Testing degradation

// Untested degradation is a second, worse system that nobody has run.
// Atlas injects failures in staging weekly and once a quarter in
// production during a quiet window, announced.
const DRILLS = [
  { inject: 'model provider 100% 503', expect: 'retrieval-only within 2 min' },
  { inject: 'vector index unavailable', expect: 'read-only-corpus, no stack traces' },
  { inject: 'memory store timeout', expect: 'runs complete, metric increments' },
  { inject: 'tool backend 100% 500', expect: 'answers without the tool, says so' },
  { inject: 'provider 8 s latency', expect: 'degrade at 6 s, not a timeout at 25' },
]

The quarterly production drill has found two things staging did not: a degraded banner that rendered behind a modal, and a retrieval-only path that bypassed the audit log entirely — which meant the most interesting forty minutes of the year would have had no record.

Key points

Most of the system does not need the model. The deterministic prefix is a permission-correct search engine over the company's documents.
Four levels: full, reduced, retrieval-only, read-only-corpus, with automatic triggers and manual kill switches.
Retrieval-only is the level worth building. It is the retrieval stack rendered, with heading paths, deep links and freshness stamps.
During a 38-minute outage, 61% of users found what they needed from passages alone, and escalations were below a normal Monday.
Degrading at 6 s beat waiting 14 s: seven points worse on outcome, thirty-nine points better on acceptability.
Degrade optional dependencies inside a run — memory, reranking, query rewriting — with tight timeouts and recorded fallbacks.
Never degrade a safety control. If citation validation cannot run, fail the response.
Say what is unavailable, say what they get instead, and tell them the results are complete. Do not estimate a recovery time.
Drill it. Atlas's production drill found a banner hidden behind a modal and a degraded path that bypassed the audit log.

Next: deployment, environments and promoting a prompt change — getting changes out without being the cause of the next degradation.