The demo went well. The agent read a sample inbox, drafted sensible replies, updated the right records and saved a team hours. The sponsor asked for a production pilot, and the request went to security. Then the questions started. What can it access? The CRM, through the integration. Can it delete records? It should not. How would we stop it? We would turn off the server, probably. The reviewer sent a long vendor-risk questionnaire built for SaaS products. The team answered half of it. Three months later the pilot is still pending and the sponsor has moved the budget elsewhere.
From the inside, the security team looks like the bottleneck. From the reviewer's side, the team asked for permission to run something nobody could describe. The fix is not a faster reviewer. It is a project team that arrives with the answers already written down, in the shape the reviewer needs.
What the reviewer is actually deciding
A security review of an AI pilot is a risk decision, not a pass or fail exam. Classic security programs formalize this as an authorization: NIST's Risk Management Framework describes control selection, implementation and assessment, system authorizations and continuous monitoring, and it establishes responsibility and accountability for the controls in place [3]. You do not need the full federal process, but the shape is right. Someone has to look at what the system can do, what controls exist, what risk remains, and who accepts it.
For an agent, the reviewer is weighing four questions. Is the blast radius bounded, meaning the worst outcome of the agent's actions is known and limited by something other than the model's good behavior? Can we see what it does, with enough detail to reconstruct an incident? Can we stop it quickly and undo what it did? And who accepts what is left? If any of the four is unanswered, the safe choice for the reviewer is to wait, and waiting is what you observe.
Why AI pilots in particular get stuck
- Authority is described in the prompt. The team says the agent only reads records because the system prompt says so, while the credential behind it can write and delete. Reviewers have learned to treat prompt instructions as intentions, not controls.
- Credentials are borrowed. The demo ran on the builder's own token or a shared integration account, so nobody can say what the agent's identity is or how to revoke it alone.
- Inputs are not mapped. The agent reads email, documents or web pages written by outsiders, and nobody has written down that this makes it a target for prompt injection.
- The tool list keeps moving. A pilot that gains two tools a week cannot be reviewed once; the reviewer needs to know what triggers a re-review.
- The demo used synthetic data. Production means real customer records, a model provider receiving them, and contractual questions the demo never raised.
- There is no rollback. The team has thought about turning the agent off but not about reversing what it already did.
OWASP's guidance on excessive agency names the root causes reviewers worry about: excessive functionality, excessive permissions and excessive autonomy [2]. Its newer Top 10 for Agentic Applications, published in December 2025, is a peer-reviewed list of the most critical risks for autonomous and agentic systems, and some reviewers now ask teams to map their pilot against it [4]. Neither document is a checklist you can pass, but both tell you which questions are coming.
The review kit: six artifacts in one folder
The kit is short by design. Each artifact answers one reviewer question, fits on one or two pages, and is produced by the project team before the review meeting. If you have run a threat-modeling session for the agent, the first two artifacts already exist.
Each artifact answers one question the reviewer will otherwise ask in a meeting. The kit is complete when every tile has a page and every page names an owner.
| Artifact | Question it answers | What it contains | Produced by | Done when |
|---|---|---|---|---|
| Data flow | What goes in and out, and who writes it? | One diagram: every input with its author, every system the agent calls, what is sent to the model provider, data classes involved | Builder | Every input is labeled trusted or untrusted by who wrote it, not where it is stored |
| Authority inventory | What can it actually do? | Each tool, its credential, the intended scope and the scope the credential really has | Builder with each system owner | No action is limited only by the prompt, or each such case is listed as unenforced |
| Consequence tiers | What is the worst single action? | Every action ranked from read to reversible change, irreversible change, outbound, and money or admin | Builder and security | Each tier states whether it runs, asks a person or is blocked during the pilot |
| Controls | What stops the bad outcomes? | Each top risk mapped to a control outside the model, with where it is enforced | Operator | Every control has been exercised at least once in a test |
| Evidence | How will we know what it did? | Log fields per action, a sample of real transcripts, test results, and the monitoring plan | Operator | Any action from the test run can be traced to the input that caused it |
| Rollback | How is it stopped and undone? | The pause mechanism, who can use it, and how each consequential action is reversed | Operator and system owners | The pause has been drilled and timed, with the result recorded |
What makes each artifact credible
The authority inventory carries the most weight. Write the intended scope and the actual scope side by side; reviewers trust a team that shows them the gap more than one that claims there is none. The controls page should follow OWASP's two most practical recommendations: limit downstream permissions to the minimum necessary, and require a person to approve high-impact actions before they happen [2]. Name where each approval happens and who answers it.
Evidence and rollback map to the NIST AI RMF. MAP 3.5 asks that processes for human oversight be defined, assessed and documented; MANAGE 4.1 asks for post-deployment monitoring plans that include incident response, recovery and change management; and MANAGE 2.4 asks for assigned responsibilities and mechanisms to disengage or deactivate a system whose outcomes are inconsistent with intended use [1]. A reviewer who sees those three handled by named people has most of what they need.
The questions reviewers actually ask
Reviewers vary, but the same questions come up in nearly every AI pilot review. Prepare a strong answer for each. A weak answer is not always wrong; it is unverifiable, and an unverifiable answer is what produces the next round of questions.
| Question | Weak answer | Strong answer |
|---|---|---|
| What can the agent access? | The CRM and email, through the integration | Its own CRM identity with read on cases and write on two fields; email send restricted to the case contact; scopes in the authority inventory |
| Can it delete or change data permanently? | It is told not to | The credential has no delete permission; irreversible changes are not in the pilot's tool list |
| Who can put text in front of it? | Our customers, via email | Anyone who can email the claims address, plus customers through attachment text; both are treated as untrusted input |
| What happens if it is manipulated? | The model is good at refusing | The worst reachable action is a drafted reply that a person sends; the trace is in the data-flow page |
| What data goes to the model provider? | Only what is needed | Case text and policy number; no payment data; the provider terms on retention and training are attached |
| How do we stop it? | Turn off the server | One pause stops all runs; the operator drilled it and it took four minutes; anyone on the pilot can pull it |
| How do we undo what it did? | We would look at the logs | Every write is logged with the triggering message; field changes are reverted from history by the system owner |
| What makes you come back to us? | Major changes | Any new tool, credential, input channel, data class or model; listed in the pilot charter |
Scope the pilot so it can be approved
A production pilot is not production at small scale. It is a bounded experiment with real data, chosen so that the residual risk is small enough for a business owner to accept with confidence. Most of the scoping work is deciding what the pilot will not do yet.
- Named users and a fixed volume, such as one team's queue, rather than a percentage of all traffic.
- The narrowest data classes that still prove the value, with payment data and special-category data left out unless they are the point.
- Consequential actions ask a person during the pilot, even if the long-term plan is for some of them to run unattended.
- A fixed duration with exit criteria agreed in advance: what success looks like, what failure looks like, and what triggers an early stop.
- A named risk owner in the business who signs the residual risk, separate from the security reviewer who advises on it.
A worked example: the claims-intake assistant at Tamsin Mutual
Consider Tamsin Mutual, a fictional regional insurer. A claims operations team built an assistant that reads incoming claim emails, looks up the policy, drafts a status reply and creates a task for an adjuster. The demo impressed leadership, and the pilot request had been in security review for eleven weeks. The reviewer's open questions were all variants of what it can touch and how it is stopped.
The team built the kit in a week. The data flow showed that claim emails and their attachments are written by claimants and anyone who knows the address, so the assistant reads untrusted input on every run. The authority inventory found that the claims-system integration user could edit and close claims, not just read them, and that the email tool could send to any address. Neither was needed. The consequence tiers placed sending a reply as outbound, so the team changed the design: the assistant drafts, and an adjuster sends.
The controls page then became short. A new identity for the assistant with read on claims and create on tasks. No send permission at all. Payment and medical attachments excluded from the pilot by routing rule. Evidence was a log entry per tool call carrying the email ID that triggered it. The rollback page described one pause and noted that tasks could be deleted by the adjuster team lead. The reviewer approved a six-week pilot for one regional claims team with three conditions, and the risk owner, the head of claims operations, signed it.
A two-week path to an approved production pilot
Two weeks is realistic when the pilot is scoped tightly and the reviewer is involved from the first day rather than the last. Book the review meeting on day one for day ten; a fixed date focuses everyone.
Scope and the first two artifacts come first, fixes land in the middle, and the last days produce evidence and a timed pause drill. The reviewer sees drafts on day three, so the meeting on day ten confirms rather than discovers.
- Day 1: kickoff with the builder, operator, a security reviewer and the risk owner. Agree the pilot scope, users, data classes, duration and exit criteria in a one-page charter.
- Days 2-3: produce the data flow and authority inventory, and share both drafts with the reviewer. Expect the inventory to surface at least one credential that can do more than intended.
- Days 4-5: rank actions into consequence tiers and decide what runs, asks or is blocked in the pilot. Write the controls page against the top risks.
- Days 6-8: make the fixes, which are usually a dedicated identity for the agent, narrower scopes, removing a tool, and adding approval on outbound actions.
- Day 9: run tests, including injected instructions in an input, a direct tool call that exceeds a limit, and a trace from a logged action back to its input. Attach the results to the evidence page.
- Day 10: drill the pause and record how long it took, then hold the review. The usual outcome is approval with conditions, each condition with an owner and a date.
After approval: keep the pilot approved
Approval is a snapshot. The RMF treats authorization as ongoing, backed by continuous monitoring [3], and an agent pilot needs the same idea in lighter form. List re-review triggers in the charter: a new tool, a new credential, a new input channel, a new data class or a model change. Any of these sends the changed artifact, not the whole kit, back to the reviewer. Hold a short exit review at the end of the pilot that compares the exit criteria with what happened and decides the scope of production.
What goes wrong in practice
- The kit is written to pass rather than to inform. A reviewer who finds an undisclosed scope in the first test stops trusting the rest of the folder.
- The vendor questionnaire is answered instead of the kit being built. Questionnaires cover the vendor; the kit covers your agent, your credentials and your data.
- Approvals are added to every action to speed up sign-off, and the pilot users approve everything without reading. Approve only what is consequential.
- The pilot quietly grows. New users and tools arrive without the triggers firing, and the approval no longer describes the system.
What to do this quarter
- Pick the pilot that has waited longest in review and build its kit with the six artifacts in the table above.
- Agree a standard kit template with your security team so every future pilot arrives in the same shape.
- Require a dedicated identity per agent before any pilot touches production data.
- Make the pause drill and the trace-from-action test mandatory evidence for every pilot approval.
- Write re-review triggers into each pilot charter and check them at every change to the agent.
Where a runtime gateway helps
Several artifacts in the kit are easier to produce when the agent's traffic passes through a runtime gateway. Intertrace sits inline between the agent and its models, tools and MCP servers, reads what each call does from the tool name and its arguments, and assigns a consequence tier, so outbound messages, payments and admin changes ask a person. New agents start in a learning period that records what they use while consequential actions ask, which gives the authority inventory real data; every decision is recorded with the rule, layer and evidence behind it, which supplies the trace the evidence page needs; and the enforcer fails closed when it cannot decide. Scoping credentials and writing the charter remain the team's work.
