Chapter 134 · Production

Rate limits, retries and backoff against a provider

The dependency you cannot scale, fix or page. Which errors are retryable, the retry that makes an incident worse, quota partitioning, and the circuit breaker that has to know about money.

9 min read

Chapter 123 limited what users can spend. This is the other direction: the provider limiting you, and the fact that the most important dependency in the system is one you cannot scale, cannot fix and cannot page anyone about.

Classify before retrying

Chapter 77 made the general argument. The provider-specific version is worth spelling out because the status codes are misleading.

Response	Retryable	Note
429 rate limit	Yes	Honour `retry-after` if present
429 quota exhausted	No	Same status, different meaning. Check the body
500 / 502 / 503	Yes	Usually transient
529 overloaded	Yes	Provider-specific; back off harder
Timeout, no response	Carefully	The request may have completed and been billed
400 invalid request	No	A bug. Retrying hides it
401 / 403	No	A credential problem. Page someone
422 content filtered	No	A product decision, not a failure

// The two 429s sharing a status code is the trap. One clears in seconds;
// the other clears at the start of the next billing period, and
// retrying it burns the retry budget of every other request in flight.
function is429Retryable(res: Response, body: any): boolean {
  const type = body?.error?.type ?? ''
  if (/quota|billing|credit/i.test(type)) return false
  return true
}

// The timeout row is the one that costs money. A request that times out
// client-side may have completed server-side, and a naive retry pays
// twice. Idempotency keys — chapter 73's mechanism, applied to the
// provider rather than to your own tools — make it one charge.
const res = await fetch(url, {
  headers: { 'idempotency-key': `${runId}:${stepIndex}:${attempt === 0 ? 0 : 0}` },
})
// ⚠️ the key is the same across attempts of the SAME logical call, which
// is the whole point. Incrementing it per attempt defeats it.

Backoff that does not make it worse

// Exponential with full jitter. The jitter is not a refinement: without
// it, every client that failed at t retries at t+1s together, and the
// provider's recovery window is exactly when the thundering herd lands.
function delay(attempt: number, retryAfter?: number): number {
  if (retryAfter) return retryAfter * 1000 + Math.random() * 500
  const cap = Math.min(30_000, 500 * 2 ** attempt)
  return Math.random() * cap            // full jitter, not base + jitter
}

// And the budget, which is the part usually missing. Retries must be
// bounded by the RUN's remaining time, not by an attempt count — a run
// with 3 seconds left should not start a 16-second backoff.
async function callWithRetry(req: Req, run: Run): Promise<Res> {
  for (let attempt = 0; attempt < 5; attempt++) {
    try { return await call(req, { timeout: Math.min(20_000, run.remaining()) }) }
    catch (e) {
      if (!retryable(e)) throw e
      const d = delay(attempt, retryAfterOf(e))
      if (d > run.remaining() - 2_000) throw new OutOfTime(e)   // chapter 133's deadline
      await sleep(d)
    }
  }
  throw new RetriesExhausted()
}

Atlas's p99 had a 19-second component that was entirely retry backoff inside runs that then failed anyway. Bounding retries by the remaining deadline removed it and changed nothing else: the runs that would have succeeded on attempt four had time for attempt four.

Partitioning the quota

// The quota is a shared resource across every caller in the system, and
// the ones that are not users should never be able to starve the ones
// that are. Chapter 123 introduced this; here is the enforcement.
const POOLS: Record<Pool, { share: number; queue: Queue }> = {
  interactive: { share: 0.70, queue: new Queue({ maxWait: 2_000 }) },
  background:  { share: 0.20, queue: new Queue({ maxWait: 120_000 }) },
  evaluation:  { share: 0.10, queue: new Queue({ maxWait: 600_000 }) },
}

// Two properties. Background and evaluation WAIT rather than fail,
// because they have no user attached and a slow eval run is fine. And
// interactive can borrow unused capacity from the others, so the
// partition costs nothing at low load.
const effectiveShare = (p: Pool) => POOLS[p].share + (p === 'interactive' ? idleShareOf(['background','evaluation']) : 0)

Atlas's incident before this existed: a re-embedding job after a corpus migration consumed the quota and users saw rate-limit errors for forty minutes. The partition is twenty minutes of work and it converts that from an outage into a slow background job.

The circuit breaker

// Standard pattern, with one agent-specific addition.
interface BreakerState {
  failures: number
  openedAt?: Date
  // ⚠️ The addition: spend in the window. A provider returning 200s with
  // degraded output still costs money, and a breaker watching only
  // errors will not open on "every request now takes 9 steps".
  spendInWindow: number
}

function shouldOpen(s: BreakerState): boolean {
  return s.failures > 20 || s.spendInWindow > 3 * TRAILING_MEDIAN
}

// Half-open with a real request, not a health ping. The provider's
// status endpoint and the provider's inference endpoint fail
// independently, and Atlas has seen green status with 100% 529s.
async function probe(): Promise<boolean> {
  const r = await generate(CANARY_QUESTION, CANARY_CONTEXT, { timeout: 5_000 })
  return SCHEMA.safeParse(r).success
}

The spend trigger is the one worth copying. It catches the failure mode that is not an error at all: a model version rolled forward, runs got longer, and the bill tripled while availability stayed at 100%. Chapter 123 named that as telemetry; here it is wired to an actuator.

A second provider

// Worth the complexity only when the answer to this is yes:
//   "Is an hour of unavailability worse than permanently carrying two
//    prompts, two evaluations and two sets of behaviour?"
// For Atlas, no. For a customer-facing product, often yes.

// The cost of doing it properly, which is what gets underestimated:
const SECONDARY_COST = [
  'A second prompt. Chapter 54 does not make prompts portable, it makes them survivable.',
  'A second calibration of the judge, or a judge that is neither provider.',
  'A second red-team run. Chapter 116 measured 31% on one model.',
  'Two sets of eval baselines, tracked separately.',
  'Behaviour differences users notice immediately on failover.',
]

Atlas's answer was the degraded mode in chapter 135 instead: retrieval without generation, which needs no model at all and is honest about what it is.

What to watch

const WATCH = [
  { m: 'retry rate', normal: '<1%', means: 'above 5%, the pool sizing is wrong or the provider is degraded' },
  { m: 'retries that exhausted', normal: '~0', means: 'any sustained value is user-visible failure' },
  { m: 'time spent in backoff, as a share of p99', normal: '<10%', means: 'retries are eating the deadline' },
  { m: 'breaker state changes', normal: '0/week', means: 'each one is an incident to read' },
  { m: 'provider p95 vs our p95', normal: 'tracks', means: 'divergence is ours, not theirs' },
]

The last is the one that settles the argument that starts every provider-related incident. Plotting the provider's own reported latency against yours on the same chart answers "is it them or us" in two seconds, and without it the question takes twenty minutes and a screenshot of a status page.

Key points

Two 429s share a status code: rate limited clears in seconds, quota exhausted clears next billing period. Check the body before retrying.
A client-side timeout may have completed and been billed. Send the same idempotency key across attempts of one logical call.
Use full jitter, not base-plus-jitter. Without it every failed client retries together exactly during the recovery window.
Bound retries by the run's remaining deadline, not an attempt count. Atlas's p99 had a 19-second backoff component inside runs that failed anyway.
Partition the provider quota: interactive 70%, background 20%, evaluation 10%, with the non-interactive pools waiting rather than failing and interactive borrowing idle capacity.
That partition converts "a re-embedding job took the product down for forty minutes" into a slow background job.
Give the circuit breaker a spend trigger as well as an error trigger. A provider returning 200s with longer runs is a failure that no error rate shows.
Probe with a real request. Provider status pages and provider inference fail independently.
A second provider costs two prompts, two calibrations, two red-team runs and two baselines. For an internal assistant, degraded mode is usually the better answer.
Plot the provider's reported latency against yours. It answers "is it them or us" in two seconds.

Next: graceful degradation when the model is down — what Atlas built instead of a second provider.