Chapter 130 · Production

Cost: where the money actually goes

A full breakdown before any optimisation. The 7% of questions that are 48% of the bill, the costs that are not inference, and the three changes that took Atlas's cost per answer down 61%.

7 min read

Cost conversations about agents start with the token price and stop there. The token price is one line of a bill with six lines on it, and the line people optimise first is usually not the largest.

So: measure, then decide. The measurement is a day's work and it has consistently redirected the optimisation.

The full bill

Atlas, one month, 5,600 questions.

Line	Monthly	Per answer	Share
Generation tokens	$287	$0.0512	61%
Embedding (queries)	$9	$0.0016	2%
Embedding (ingestion)	$64	—	14%
Vector store compute	$48	—	10%
Eval suite	$210	—	—
Application hosting	$41	—	9%
Observability	$18	—	4%
Serving total	$467	$0.083
With evals	$677

Two things people get wrong, visible immediately.

The eval suite is 31% of total spend. It is not a serving cost and it is real money, and it is the first thing a cost review proposes cutting. Chapter 114 made the argument against; the number is what makes the argument necessary.

Ingestion embedding is 14%. It is invisible in a per-answer figure and it is the line that grows fastest, because chapter 138 is about a corpus that keeps growing whether or not anyone asks questions.

Where generation tokens actually go

// The 61%, decomposed. This is the breakdown that changes decisions.
const TOKENS_PER_ANSWER = {
  systemPrompt: 1_240,        // identical every request
  toolDefinitions: 980,       // identical per role
  fewShotExamples: 620,       // identical every request
  retrievedContext: 3_100,    // varies
  conversationState: 90,      // chapter 92, small by design
  recalledMemory: 110,
  question: 24,
  output: 284,
}
// input 6,164 · output 284

// 2,840 tokens — 46% of the input — is byte-identical on every single
// request. That is the single largest optimisable block and it is not
// the context, which is where attention usually goes.

Prompt caching makes that block nearly free, and it requires exactly one thing: the identical prefix must come first and must not be disturbed. Chapter 60 laid out the ordering; the cost consequence is measured below.

The 7% that is 48% of the bill

-- Spend is never uniform. Grouping by cost decile is the query that
-- finds the money.
select ntile(10) over (order by cost_usd) as decile,
       count(*), round(sum(cost_usd), 2) as spend,
       round(100.0 * sum(cost_usd) / sum(sum(cost_usd)) over (), 1) as pct,
       round(avg(steps), 1) as avg_steps
  from audit_run where at > now() - interval '30 days'
 group by 1 order by 1 desc;

Decile	Questions	Spend	Share	Avg steps
10th	560	$138	29.6%	7.2
9th	560	$86	18.4%	4.1
8th	560	$51	10.9%	2.8
1st–7th	3,920	$192	41.1%	1.3

The top two deciles — 20% of questions — are 48% of the bill, and the distinguishing feature is steps, not context size. A multi-step run re-sends the whole accumulated context on every step, so a seven-step run costs roughly seven times a one-step run with the same context.

// Which reframes the optimisation. "Reduce context size" helps every
// question a little. "Reduce steps on the expensive decile" helps the
// 30% of spend that is one decile, and chapter 83 already found the
// lever: a loop whose trace shows one dominant path is a workflow
// nobody has written yet.

Atlas's past-due investigation, promoted from a loop to a fixed workflow, was 71% cheaper on its own. That one change was 9% of total spend.

Three changes, measured

// 1. Prompt caching. One ordering change; the prefix was already stable.
//    input tokens billed: 6,164 -> 3,324 effective
//    cost per answer:     $0.0512 -> $0.0317   (-38%)

// 2. Routing the simple decile to a cheaper model. Chapter 131 is the
//    whole subject; the cost effect alone:
//    cost per answer:     $0.0317 -> $0.0241   (-24%)

// 3. Promoting the three dominant loop paths to workflows.
//    cost per answer:     $0.0241 -> $0.0199   (-17%)

Cumulative: $0.0512 to $0.0199, down 61%, with quality measured flat on the eval suite at every step. Measuring quality at each step is what made the sequence safe — the routing change in particular is one that can quietly cost accuracy.

What was not worth doing

// Three optimisations that were measured and rejected.

// 1. Shortening the system prompt. 1,240 tokens, and with caching it
//    costs almost nothing. The quality risk is real and the saving is
//    $0.0004 per answer.

// 2. Reducing retrieved chunks from 12 to 8. Saves 1,000 tokens, and
//    cost recall@12 from 0.94 to 0.89. Chapter 110 then showed
//    groundedness falling with it. A false economy with a quality bill.

// 3. A self-hosted embedding model. Saves $73/month on ingestion and
//    query embeddings, and costs an inference service to operate,
//    monitor and upgrade. At this scale, no.

The third is the general shape worth naming: an optimisation that converts a bill into an operational burden is only worth it when the bill is large enough to pay for the burden. At $73 a month it is not close.

Cost per unit of value

// The number to take to anyone asking whether this is worth running.
const UNIT = {
  costPerAnswer: 0.0199,
  answersPerMonth: 5_600,
  monthlyServing: 112,            // plus evals and hosting
  // The comparison that matters:
  opsTeamMinutesSaved: 6.5,       // median, measured by chapter 113's escalation work
  loadedCostPerMinute: 1.15,
}
// ~$7.48 of time per answer against $0.02 of inference.

That ratio is why the cost optimisation above, while worth doing, was never the important work. A 61% saving on inference is $170 a month. One week of the operations team's time is more than the assistant costs in a year, and chapter 113's escalation metric is what measures the other side.

Optimise cost because waste is a signal of something structurally wrong, not because the bill is large. On an internal assistant it almost never is.

Key points

Measure all six lines before optimising. Generation is 61%, ingestion embedding 14%, vector compute 10%, hosting 9%.
The eval suite was 31% of total spend. It is the first thing a cost review proposes cutting, which is why the number has to be on the table with the argument.
46% of input tokens are byte-identical on every request. That block, not the context, is the largest optimisable thing.
Spend is concentrated: 20% of questions are 48% of the bill, and the distinguishing feature is steps, not context size.
A multi-step run re-sends accumulated context each step, so seven steps costs roughly seven times one.
Three changes took cost per answer down 61%: prompt caching (-38%), routing (-24%), promoting dominant loop paths to workflows (-17%). Quality was measured flat at each step.
Rejected: shortening the system prompt (saves $0.0004 with caching), fewer chunks (recall 0.94 to 0.89), self-hosting embeddings (converts $73/month into an operational burden).
At $0.02 of inference against roughly $7.50 of staff time saved per answer, cost optimisation is a waste signal rather than a budget problem.

Next: model routing — the 24%, and how to take it without losing accuracy.