Chapter 115 · Guardrails and security

The threat model for an agent with your data

Who the adversary actually is, what they want, and the four properties that make an agent a different security problem from the application it sits beside.

9 min read

Security sections in tutorials usually open with a list of attacks. That ordering teaches the attacks and not the reasoning, and the reasoning is what transfers to the attack nobody has named yet.

So: what is actually different about this system, who is attacking it, and what does a loss look like.

Four properties that change the problem

// None of these is true of the CRM sitting next to it.
const PROPERTIES = [
  'It reads untrusted text and then acts on text.',
  'It holds a credential that is broader than any single user.',
  'Its control flow is decided at runtime by a model.',
  'Its failures are fluent, and therefore credible.',
] as const

It reads untrusted text and then acts on text. Conventional applications separate code from data by construction: a SQL parameter cannot become a SQL verb. A model receives instructions and content in the same channel, in the same format, with no mechanism that distinguishes them. That is the root of most of this section, and chapter 118 argues it cannot be patched away.

It holds a broad credential. The assistant can reach the document store, the orders database and the ticketing API. No individual user can reach all three. That concentration is the whole value of the thing and it is also chapter 120's problem.

Its control flow is decided at runtime. Section 7 was about giving up the call graph. A security review cannot read the code and enumerate the paths, because the paths are chosen by a model given text an attacker may have written.

Its failures are fluent. A corrupted API response looks corrupted. A manipulated answer looks like an answer, and chapter 110 measured how well-camouflaged invention is even without an adversary.

Who the adversary is

Naming them matters because the controls differ.

Adversary	Capability	Wants	Likelihood
Curious insider	Full user access, knows the business	Data outside their remit — salaries, margins, another region's accounts	High
Careless insider	Same	Nothing; causes harm by accident	Very high
External, via content	Can get a document into the corpus	Exfiltration, or a wrong commitment made to them	Medium
Compromised account	One user's full session	Whatever that user could reach, amplified	Low, severe
Malicious insider	Full user access, intent	Bulk extraction before leaving	Low, severe

For a 200-person distributor the top two rows are where almost all of the real risk is, and neither involves an attack in the usual sense. The curious insider types "what is the margin on the Northline account" and finds out whether the system will tell them. The careless one pastes a customer email containing a supplier's pricing into a question.

The third row is the one that is under-estimated, because getting a document into the corpus is easy. Anyone who can email a supplier document to someone in purchasing can get text in front of the model. Chapter 117 is about that path.

What a loss looks like

// Ranked by what it actually costs, which is not the same as ranked by
// how exciting the attack is.
const LOSSES = [
  { id: 'wrong-commitment', what: 'A rep quotes a price or term the company will not honour',
    cost: 'A contract dispute. The most likely bad outcome by a wide margin.' },
  { id: 'cross-user-disclosure', what: 'One person sees data scoped to another',
    cost: 'An HR matter, or a regulatory one. Chapter 121.' },
  { id: 'external-disclosure', what: 'Company data reaches someone outside',
    cost: 'Contractual, and sometimes statutory.' },
  { id: 'unauthorised-action', what: 'A credit, a cancellation, a ticket nobody approved',
    cost: 'Direct and reversible, usually. Chapter 71 made it reversible on purpose.' },
  { id: 'availability', what: 'It is down, or too slow to use',
    cost: 'Lowest. People go back to email.' },
]

Putting wrong-commitment first is deliberate and it is where most teams under-invest. It needs no attacker at all — chapter 110's 1.4% invention rate produces it on its own — and the controls for it are the groundedness and refusal work from section 9 rather than anything in this section.

Most of the harm an internal assistant does is caused by nobody. The security work is what stops the remaining cases from being worse than the accidents.

Trust boundaries, drawn explicitly

The single most important box is the green one, and the single most important claim on the diagram is that model output is untrusted. It was produced by a process that read attacker-controlled text. Every control that matters in this section sits between that red box and the world.

// Which gives the rule the rest of the section implements.
//
//   The model decides WHAT to attempt.
//   The policy layer decides whether it is ALLOWED.
//   The policy layer is ordinary code, it sees the user's identity, and
//   nothing the model emits can change its mind.

Everything the model produces — a tool name, an argument, a document id, an answer — is a request to the policy layer, evaluated against the asking user's rights. Chapter 119 is that idea applied to data access and it is the highest-value control in the section.

What is explicitly out of scope

A threat model that defends against everything defends against nothing.

// Atlas's stated non-goals, agreed and written down.
const OUT_OF_SCOPE = [
  'A user memorising what they legitimately saw. Access control, not an agent problem.',
  'Screenshots and copy-paste of legitimate answers.',
  'A compromised cloud provider or model provider.',
  'Model weight extraction. There are no weights here; it is a hosted API.',
  'Denial of service by a determined insider. Rate limits make it annoying, not impossible.',
]

Writing the non-goals down is what makes the review in chapter 126 finishable. Without them every review reopens the same unanswerable questions.

The controls, and where each one lives

Control	Layer	Chapter
Permission filter in the query	Data	119
Acting as the user, not as the service	Identity	120
Per-user scoping of memory and caches	Data	121
Typed, enumerated tool arguments	Policy	69
Confirmation before a write	Policy	72
Egress allowlist	Policy	122
Output scanning	Response	122
Rate limits and anomaly detection	Edge	123
Audit log	Everywhere	124

Note what is absent: there is no row for "instruct the model not to be manipulated". Prompt instructions are a quality measure, not a control, and chapter 118 gives the measurement behind that claim.

Key points

Four properties make this different: untrusted text drives behaviour, the credential is broader than any user, control flow is chosen at runtime, and failures are fluent.
The adversaries that matter at a 200-person company are the curious insider and the careless one. Neither is attacking in the usual sense.
Getting a document into a corpus is easy. Anyone who can email a supplier document to purchasing can put text in front of the model.
The likeliest loss is a wrong commitment to a customer, and it needs no attacker. Its controls are section 9's, not this section's.
Draw the trust boundaries. Model output is untrusted, because it was produced by a process that read attacker-controlled text.
The model decides what to attempt; a policy layer in ordinary code decides what is allowed, and nothing the model emits changes its mind.
Write the non-goals down, or the security review never finishes.
There is no control called "tell the model not to fall for it".

Next: prompt injection, demonstrated against Atlas — the mechanism, on a system that has all the obvious defences.