Chapter 60 · The model layer

Token budgets and context assembly

Everything wants space in the context and the budget is fixed. A priority-based assembler, the caching boundary that halves the bill, and failing loudly when retrieval gets squeezed out.

11 min read

Chapter 52 ended with an assertion that context size had to be budgeted and a stub of the assembler. This chapter builds it, because by section 6 there will be tool definitions competing for the same space and by section 8 there will be memory.

The problem is not that context is scarce — it is that every component's author believes their content is essential, and without an explicit budget the resolution is whoever appends last.

The claimants

Claimant	Typical	Grows with	Chapter
System prompt	610	Rules added	53
Examples	250	Edge cases found	55
Retrieved chunks	2,520	Window size	52
Conversation history	0–4,000	Session length	Section 8
Tool definitions	0–1,800	Tools added	Section 6
Long-term memory	0–800	Time	Section 8
Question	20	—	—

Unbudgeted, a long session with the full tool surface reaches about 10,000 tokens — four times what chapter 52 measured as the knee, with correctness falling past eight chunks.

The budget, as data

// src/generate/budget.ts
// Priorities, not just sizes. When something has to go, the priority says
// what — and the decision is reviewable because it is written down.
export const CONTEXT_BUDGET = {
  total: 6_000,

  allocations: [
    { name: 'system',    min: 610,   max: 610,   priority: 0 },  // never trimmed
    { name: 'question',  min: 20,    max: 200,   priority: 0 },
    { name: 'retrieved', min: 1_200, max: 3_000, priority: 1 },  // the answer
    { name: 'tools',     min: 0,     max: 1_200, priority: 2 },
    { name: 'history',   min: 0,     max: 1_500, priority: 3 },  // trimmed first
    { name: 'examples',  min: 250,   max: 250,   priority: 1 },
    { name: 'memory',    min: 0,     max: 400,   priority: 4 },
  ],
} as const

Two things that make this work rather than being decoration.

min is a floor, not a suggestion. If retrieved chunks cannot get 1,200 tokens, the request fails rather than proceeding with two chunks — because an answer from two chunks is worse than an honest failure and is indistinguishable from a normal answer.

Priority 0 means never trimmed. The system prompt and the question are not negotiable, and making that explicit stops a clever assembler from truncating the rules to fit a long history.

The assembler

export function assembleContext(parts: Record<string, string>): Assembled {
  const specs = [...CONTEXT_BUDGET.allocations].sort((a, b) => a.priority - b.priority)
  const sized = new Map<string, number>()
  let remaining = CONTEXT_BUDGET.total

  // Pass 1: everyone gets their minimum, in priority order.
  for (const spec of specs) {
    const want = Math.min(countTokens(parts[spec.name] ?? ''), spec.max)
    const give = Math.min(want, spec.min)
    if (give > remaining) {
      // ⚠️ A floor could not be met. Fail rather than degrade silently.
      throw new ContextBudgetError(
        `cannot allocate ${spec.min} tokens for '${spec.name}': ${remaining} left`,
      )
    }
    sized.set(spec.name, give)
    remaining -= give
  }

  // Pass 2: distribute what is left, still in priority order.
  for (const spec of specs) {
    const want = Math.min(countTokens(parts[spec.name] ?? ''), spec.max)
    const extra = Math.min(want - sized.get(spec.name)!, remaining)
    if (extra > 0) { sized.set(spec.name, sized.get(spec.name)! + extra); remaining -= extra }
  }

  return {
    text: render(parts, sized),
    sizes: Object.fromEntries(sized),
    trimmed: specs.filter((s) => countTokens(parts[s.name] ?? '') > sized.get(s.name)!)
                  .map((s) => s.name),
    unused: remaining,
  }
}

Two passes rather than one is what prevents a high-priority claimant that happens to be small from blocking a low-priority one that fits.

Trimming, done per claimant

Generic truncation is wrong for every claimant. Each knows how to shrink itself.

// Retrieved chunks: drop whole chunks from the bottom, never truncate one.
// Half a chunk is a passage missing its qualifying clause — chapter 30's
// failure, manufactured on purpose.
export function trimRetrieved(chunks: Retrieved[], budget: number): Retrieved[] {
  const kept: Retrieved[] = []
  let used = 0
  for (const chunk of chunks) {
    if (used + chunk.tokens > budget) break
    kept.push(chunk); used += chunk.tokens
  }
  return kept
}

// History: drop whole turns from the oldest, and keep the first turn —
// it usually carries the topic the follow-ups depend on.
export function trimHistory(turns: Turn[], budget: number): Turn[] {
  if (turns.length === 0) return turns
  const first = turns[0]
  const recent: Turn[] = []
  let used = countTokens(renderTurn(first))

  for (const turn of [...turns.slice(1)].reverse()) {
    const size = countTokens(renderTurn(turn))
    if (used + size > budget) break
    recent.unshift(turn); used += size
  }
  // ⚠️ A gap marker, so the model knows turns are missing rather than
  // assuming the conversation began where the context does.
  return recent.length < turns.length - 1
    ? [first, GAP_MARKER, ...recent]
    : [first, ...recent]
}

// Tool definitions: drop whole tools by relevance, never truncate a
// description. A half-described tool gets called wrongly. Section 6.

The gap marker is a small detail with a real effect: without it, a model given turns 1 and 7–9 treats turn 7 as the second thing that happened and resolves pronouns against the wrong antecedent.

The caching boundary

The largest cost lever in this chapter, and it is a layout decision.

// Prompt caching works on a stable PREFIX. Everything before the first
// varying byte is cacheable; everything after is not.
const prompt = [
  SYSTEM_PROMPT,      // 610, identical every request   ┐
  EXAMPLES,           // 250, identical every request   │ cacheable: 1,860
  TOOL_DEFINITIONS,   // 1,000, identical per user role ┘
  // ---- cache boundary ----
  history,            // varies
  sources,            // varies
  question,           // varies
].join('\n\n')

// ❌ One misplaced line destroys it. A timestamp at the top means the
//    prefix differs every request and nothing is ever cached.
const prompt = `Current date: ${new Date().toISOString()}\n\n${SYSTEM_PROMPT}...`

// ✅ If the date is needed, put it after the cacheable prefix, or round it
//    to the day so it is stable for 24 hours.
const today = new Date().toISOString().slice(0, 10)

Atlas's measurement: ordering for caching cut input cost by 31% at 800 questions a day, with no change to any quality metric. The date bug above was found by a cache-hit-rate alert and had been costing that 31% for eleven days.

// Which is why the hit rate is monitored rather than assumed.
logger.info('generation.usage', {
  inputTokens: usage.input,
  cachedTokens: usage.cacheRead ?? 0,
  cacheHitRate: (usage.cacheRead ?? 0) / usage.input,
})

-- Alert on a drop. A prefix change is a deploy, and it is invisible
-- otherwise because nothing breaks.
select date_trunc('hour', answered_at) as hour,
       round(avg(cached_tokens::numeric / nullif(input_tokens, 0)), 3) as hit_rate
  from answer_log
 where answered_at > now() - interval '48 hours'
 group by 1 order by 1 desc limit 6;

Counting tokens without calling the model

// Use the model's own tokeniser. A character-count estimate is wrong by
// 10-25% depending on content, and code and identifiers are the worst case
// — which is most of what this corpus contains.
import { countTokens } from '@anthropic-ai/tokenizer'

// ⚠️ Cache per chunk at ingest. Counting 50 candidates on every request is
// measurable latency for a number that never changes.
// The `tokens` column in chapter 8's schema exists for this.

Reporting the budget

// Log the composition on every request. It is six numbers and it makes
// "why did this answer cost so much?" answerable.
logger.info('context.assembled', {
  ...assembled.sizes, total: sum(assembled.sizes), trimmed: assembled.trimmed,
})

-- And the weekly view that catches drift, which is the whole point of
-- having a budget rather than a limit.
select date_trunc('day', answered_at) as day,
       round(avg(ctx_system))    as system,
       round(avg(ctx_retrieved)) as retrieved,
       round(avg(ctx_history))   as history,
       round(avg(ctx_tools))     as tools,
       round(avg(ctx_total))     as total
  from answer_log
 where answered_at > now() - interval '30 days'
 group by 1 order by 1;

A history column climbing week over week is the accidental growth from chapter 52, visible before it reaches the point where retrieval gets trimmed.

The failure mode: the request that was squeezed

A rep has a long session — eighteen turns, several about one account. They ask a policy question.

History claims 1,500 tokens. Tools claim 1,200. The question is long. Retrieved chunks get 1,900 instead of 3,000, which is four chunks instead of six.

The answer is produced from four chunks and is plausible. Correctness on four chunks is 0.89 against 0.91 — so about one answer in fifty is wrong specifically because the session was long, and nothing in the answer, the logs or the quality dashboard attributes it to that.

// 1. Make it visible per answer, not just in aggregate.
if (assembled.trimmed.includes('retrieved')) {
  logger.warn('context.retrieval_trimmed', {
    questionId, got: assembled.sizes.retrieved, wanted: CONTEXT_BUDGET.allocations
      .find((a) => a.name === 'retrieved')!.max,
    historyTokens: assembled.sizes.history,
  })
}

// 2. Trim history harder before touching retrieval. Priority 3 against
//    priority 1 already says this; the assertion is that it worked.
test('a long session does not reduce retrieved chunks', () => {
  const assembled = assembleContext({
    system: SYSTEM, examples: EXAMPLES, tools: TOOLS,
    history: longHistory(40),           // deliberately oversized
    retrieved: sixChunks(),
    question: 'a question',
  })
  expect(assembled.sizes.retrieved).toBeGreaterThanOrEqual(2_520)
  expect(assembled.trimmed).toContain('history')
  expect(assembled.trimmed).not.toContain('retrieved')
})

// 3. Summarise history rather than dropping it, once it exceeds its
//    allocation. Section 8's job, and the budget is what reveals the need.

A budget that only reports totals tells you the system is fine on average. The question that matters is which requests were squeezed, and that is a per-request flag, not an average.

Key points

Without an explicit budget, context is allocated by whoever appends last.
Encode priorities alongside sizes. min is a floor that fails the request, not a suggestion.
Two-pass allocation: minimums in priority order, then the remainder.
Trim per claimant. Drop whole chunks, whole turns and whole tools — never truncate one.
Keep the first conversation turn and insert a gap marker, or pronouns resolve against the wrong antecedent.
Order the prompt for caching. It cut Atlas's input cost by 31%, and one timestamp at the top destroyed it for eleven days.
Monitor the cache hit rate and alert on a drop. A prefix change breaks nothing visible.
Count tokens with the model's tokeniser, cached at ingest.
Log the per-request composition and flag any request where retrieval was trimmed. Averages hide the squeezed requests entirely.

Next: what to put in context and what to leave out — spending the budget well.