Chapter 126 · Guardrails and security

The security review before you let anyone else use it

A review that finishes, run against a written threat model. Thirty-one checks in six groups, the four that are non-negotiable, and how to answer the questions a reviewer will actually ask.

8 min read

A security review of an agent goes badly in a predictable way: the reviewer asks whether the model can be tricked, the team says yes, and the conversation has nowhere to go. Both sides are right and neither has a decision procedure.

The fix is chapter 115's written threat model with its stated non-goals, and a review that asks about controls rather than about the model. The question is never can it be manipulated — chapter 116 measured 31% and the answer is yes. The question is what a manipulated model can reach.

Four that are non-negotiable

If any of these is missing, the review stops and the rest is not worth doing.

const BLOCKING = [
  'The permission filter is in the query, enforced by the datastore.',   // ch 119
  'Every tool receives the caller identity, and it is used.',            // ch 120
  'No write executes without an explicit confirmation path.',            // ch 72
  'Egress is allowlisted at the HTTP client, including rendered images.', // ch 122
]

Each is binary, each is demonstrable in five minutes, and each closes a whole class rather than an instance. A system with all four and nothing else is in better shape than one with an elaborate input filter and any of them missing.

The review, in six groups

Identity and access

const IDENTITY = [
  'Permission filter in the query, not post-filtered, not in the prompt.',
  'Groups fetched fresh per request, never from a session cache.',
  'Row-level security on, with transaction-scoped settings.',
  'Caller identity on every tool signature, enforced by the type.',
  'Identity re-validated on resume, not read from the stored run.',
  'Service credentials audited for privileges the assistant never uses.',
  'Existence-disclosure policy decided and written down.',
]

Data boundaries

const BOUNDARIES = [
  'Every cache key carries the user and a hash of their groups.',
  'Memory recall pre-filters by user in the WHERE clause.',
  'Traces are redacted for sharing and synthesised for fixtures.',
  'Errors collapse not-found, not-permitted and deleted to one response.',
  'Subagents hold no memory-write capability and carry caller identity.',
  'Aggregates require a minimum visible cohort and state their scope.',
  'A two-user leak test runs on every build, with a control assertion.',
]

Ingestion

const INGESTION = [
  'Every ingestion path enumerated, with its trust tier.',
  'Chunks not visible when rendered are dropped.',
  'External sources screened for instruction-shaped text, quarantined for review.',
  'External content can never be the sole basis for an action.',
  'Canary documents in the test corpus, across every extractable region.',
]

Actions

const ACTIONS = [
  'Tool surface enumerated; nothing unlisted is reachable.',
  'Arguments typed and range-checked, enumerations rather than free text.',
  'Writes idempotent, with keys whose components are immutable.',
  'Irreversible actions behind confirmation, with the hazard count reported.',
  'Per-run and per-user spend ceilings, plus a global one.',
]

Observability

const OBSERVABILITY = [
  'Audit log append-only, hash-chained, with the head shipped off-system.',
  'Per-document access index, distinguishing retrieved from cited.',
  'Reads of the audit log are themselves audited.',
  'Retention set per table, not one policy over "logs".',
  'Alerts on out-of-profile access, spend anomalies and injection flags.',
]

Response

const RESPONSE = [
  'Four kill switches, exercised quarterly, with a symptom-to-switch runbook.',
  'Blast-radius queries written in advance.',
  'Notification template that names specific answers.',
  'A red-team suite in the eval harness, run on every model and prompt change.',
]

Thirty-one checks. Atlas's first pass failed nine, of which two were blocking — identity was not threaded through the resumption path, and rendered markdown images were not allowlisted.

Answering the questions a reviewer asks

Four questions come up every time, and the useful answer to each is a number rather than a reassurance.

"Can it be jailbroken?"

Yes. 31% of 120 red-team attempts manipulated the model. 5% reached an effect, all information disclosure within the caller's own permissions. No attempt crossed a user boundary or executed a write, because those paths do not consult the model. Here is the suite; it runs on every model change.

"What if the model hallucinates a customer's data?"

It cannot produce a document it was not given, and it is only given documents the caller may see. Invention of figures is a real and separate risk, measured at 1.4%, detected by the unsourced-specifics check on every answer and reviewed. That is a correctness problem with a number, not an access-control problem.

"Can one user see another's data?"

Eight paths were enumerated and each has a control. A two-user test covering all of them runs on every build with a control assertion. Here is the test and the three regressions it has caught.

"What happens when it goes wrong?"

Four kill switches, exercised quarterly, 10-second propagation. Blast-radius queries written in advance against a per-document access index. One real incident: contained in 6 minutes, scoped in 20, 9 users notified with their specific answers named.

// The pattern in all four: name the measurement, name the control, offer
// the artifact. A reviewer cannot evaluate "we take security seriously"
// and can evaluate "31%, here is the suite".

What the review will not settle

// Say these out loud rather than letting them be discovered.
const RESIDUAL = [
  'The model can be manipulated. The controls assume it will be.',
  'Paraphrased and inferential disclosure are not detectable at the output.',
  'Aggregates leak in ways no per-document check sees.',
  'An input filter at this base rate has ~13% precision; it is a tripwire.',
  'A compromised account reaches everything that account reaches.',
]

Presenting the residual risks is what gets a review signed. A team claiming no residual risk is a team that has not measured, and an experienced reviewer knows it.

Who should run it

// Three passes, in order, each catching what the previous cannot.
const PASSES = {
  self:     'the checklist above, by the team. Catches the omissions.',
  internal: 'someone outside the team, with the threat model in hand.',
  external: 'a red team against production-like data, once, before wide rollout.',
}

The external pass is worth its cost once. Atlas's found one thing the other two missed, and it was not a model problem: a debug endpoint left reachable that returned a full trace for any run id, which is chapter 121's fifth path arriving through an ordinary web vulnerability. The lesson generalises — an agent is still an application, and the boring attack surface does not stop existing because the interesting one arrived.

Key points

Review controls, not the model. "Can it be tricked" is answered — yes, 31% — and tells you nothing about exposure.
Four blocking checks: permission filter in the query, caller identity on every tool, confirmation before writes, allowlisted egress including rendered images.
Thirty-one checks in six groups. Atlas failed nine on the first pass, two of them blocking.
The two blockers were identity dropped on the resumption path and markdown images not allowlisted — both invisible from the correct synchronous code.
Answer a reviewer with a number, a control and an artifact. "31%, here is the suite" is evaluable; "we take security seriously" is not.
State the residual risks. A team claiming none has not measured.
Three passes: self, internal, external. Atlas's external pass found a debug endpoint returning any run's trace — an ordinary web vulnerability on an agent's attack surface.

That closes section 10. The system now has a threat model, controls that hold when the model does not, a record of what it did, and a procedure for when it goes wrong.

Next, and the start of section 11: production. Everything so far has been about building the thing correctly. Section 11 is about running it — deployment, caching, cost at scale, latency, monitoring, on-call, versioning the corpus and the prompt together, and the operational work that decides whether an assistant is still trusted a year after launch.