Most agent threat models end up in one of two states. Either nobody writes one because the system feels too new to model, or someone writes forty pages on jailbreak taxonomies that never mention which credential the agent holds. Both miss the same thing. An agent's risk is mostly determined by what it is allowed to do and who can influence it, and both of those are knowable from the configuration you already have.
This guide is a time-boxed method for one agent, one afternoon and one page. It follows a single worked example throughout so each pass produces something concrete.
Set up the afternoon: four questions, five passes, one agent
Adam Shostack's four-question frame is still the best scaffold: “What are we working on? What can go wrong? What are we going to do about it? Did we do a good job?” [1]. The five passes below answer the first two questions in detail, the one-page model answers the third, and the test cases at the end answer the fourth. Keep the scope to one agent. A platform-wide model of every agent is a different exercise and rarely finishes.
Consider a support agent at Larkspur Goods, a fictional mid-size online retailer. The agent reads incoming tickets from the helpdesk, looks up orders, issues refunds up to a limit (assume $150 for this example), and emails customers with the outcome. It runs on a hosted model API and handles a few thousand tickets a week. Nothing about it is exotic, which is exactly why it is worth modeling: this shape of agent is being deployed everywhere.
Book four people for four hours: the agent's owner (here, the support operations lead), the engineer who wired its tools, a security engineer to facilitate, and someone who can say what the business will accept, which may be the same support lead. Ask them to bring the system prompt, the tool definitions, the list of credentials and their scopes, an architecture sketch if one exists, and twenty real transcripts, including a few the team considers odd.
Pass 1 (45 minutes): draw the data flow and mark where trust changes
Draw every input and output of the agent, then mark the trust boundaries: the places where data or control passes between parties with different levels of trust. For an agent, a boundary exists wherever text from someone other than the operator can reach the model's context, and wherever the agent's output leaves your control.
The customer ticket crosses a trust boundary on the way in and the outbound email crosses one on the way out; both are drawn as warning paths because the model connects them. Order lookup, refunds and the model API sit on the other side of their own boundaries.
At Larkspur, the first draft had one untrusted arrow: the ticket. The transcripts added two more. Order records include a free-text delivery note the customer typed at checkout, and the ticket thread includes quoted replies from earlier emails. Both arrive through internal systems, so the team had labeled them trusted. They are not. Provenance, not storage location, decides trust: text a customer wrote is customer text even after it sits in your database for a month.
Mark the model API as a boundary too. Everything in the context window, including tool results containing order details, leaves your network on every call. That is a data-handling question for your provider contract, and it belongs on the diagram so nobody forgets it exists.
Pass 2 (45 minutes): inventory the authority the agent actually holds
Authority is what the agent's credentials and tools permit, not what the system prompt asks it to do. For each tool, write down the credential behind it, the scope the designers intended and the scope the credential really has. The gap between those last two columns is usually the most valuable finding of the afternoon. OWASP's Excessive Agency entry names the three causes directly: excessive functionality, excessive permissions and excessive autonomy [3].
| Tool | Credential | Intended scope | Actual scope found | Gap |
|---|---|---|---|---|
| Ticket read | Helpdesk API token | Read the current ticket | Read and write every ticket, including other agents' queues | Write access and cross-queue reads are unnecessary |
| Order lookup | Database service account | Orders for the ticket's customer | Read-only on all orders and customer profiles | No binding between ticket requester and order queried |
| Refunds | Payments API key | Refunds up to $150 per order | Refunds of any amount on any order; the limit exists only in the system prompt | Ceiling is enforced by the model's compliance, not by a control |
| Transactional email API key | Reply to the ticket requester | Send from the company domain to any address | Recipient is not bound to the requester | |
| Ticket close | Helpdesk API token (same) | Close resolved tickets | Close, merge or delete any ticket | Delete was never needed |
The refund row is the one that changes the conversation. Everyone in the room believed the agent could not refund more than $150. In fact, the only thing standing between a manipulated model and an unlimited refund was a sentence in the prompt. Write that down plainly. Anything enforced only by instructions to the model should be recorded as unenforced.
Pass 3 (40 minutes): rank every action by consequence
Consequence is the worst realistic outcome of one action, judged by reversibility and reach. Sort each action the agent can take into five tiers, and classify by what the call does, including its arguments, rather than by the tool's name. A tool called update_ticket that can also delete is a delete tool.
Reads and reversible changes can run, irreversible changes ask in production, and anything outbound, financial or administrative always asks. The tier sets the default; any exception to it should be a recorded decision.
For Larkspur, order lookup is a read and adding an internal note to a ticket is a reversible change. Deleting a ticket is irreversible. Emailing a customer is outbound, and a refund is money. That puts two of the agent's four core jobs at the top of the ladder. The business wants small refunds to happen without a person, which is a legitimate choice, but it has to be made deliberately: a named owner accepts it, and the ceiling is enforced by the payments layer per order and per day, outside the model.
An empty tier is good news. If the agent has no administrative actions, say so on the page. It tells the next reviewer that adding one is a change worth re-modeling.
Pass 4 (40 minutes): list everyone who can write into the agent's context
Indirect prompt injection happens when a model takes in external content, such as a web page or a file, that carries instructions [2]. For a threat model, the practical question is simpler: who can put text in front of this agent, and what is the most consequential tool reachable in the same session? List every writer, not every system.
- Anyone on the internet who can email the support address writes the ticket subject, body and attachments, including PDFs and images that may be parsed into text.
- Any customer writes the free-text delivery note and the name and address fields on their order, which the lookup tool returns verbatim.
- Previous correspondents write the quoted history in the ticket thread, which may include forwarded mail from third parties.
- Support staff and contractors write the macros and knowledge-base articles the agent retrieves; check who can edit them and whether edits are reviewed.
- The model provider writes every response, and anyone who changes a tool definition or connected server writes the tool descriptions the model reads.
Now pair each writer with the agent's highest-tier tool in that session. At Larkspur, the anonymous ticket author can reach refunds and outbound email in a single conversation. That pairing is the core risk: untrusted input, access to customer data, and a channel that sends data out, all joined by one model that cannot reliably tell instructions from content. OWASP recommends segregating and labeling external content and keeping privileged functions under application control [2]. Those help, but the threat model should assume they will sometimes fail.
Pass 5 (50 minutes): assume the model is wrong, then assume it is hostile
Run the last pass twice. First assume the model is simply mistaken: it picks the wrong order, retries a refund after a timeout, or emails the wrong customer from a similar name. Then assume it is fully steered by the worst writer from Pass 4. For each scenario, walk call by call and ask what stops it. If the honest answer is that the prompt tells it not to, that scenario is uncontrolled.
STRIDE-style questions adapt well if you reframe each category around the agent. MITRE ATLAS, a public knowledge base of adversary tactics and techniques targeting AI systems, is useful here for naming techniques and finding case studies once you know which questions matter [4].
| Category | Agent question | Larkspur finding |
|---|---|---|
| Spoofing | Whose identity does the agent accept, and from where? | The agent trusts a customer name typed in the ticket body over the verified sender address. |
| Tampering | Which inputs can change the agent's plan? | Instructions in a delivery note were followed in one transcript during testing. |
| Repudiation | Can you show which input caused a given action? | Refund logs record the amount and order, but not the ticket or message that triggered them. |
| Information disclosure | Can one customer's data reach another person? | Order lookup is not bound to the requester, and email can go to any address. |
| Denial of service | What happens if the agent loops or is flooded? | No per-ticket cap on tool calls; a reopen loop could send repeated emails and burn model spend. |
| Elevation of privilege | Can input make the agent exceed its intended authority? | A ticket can request a refund above $150, and nothing outside the prompt refuses it. |
The hostile trace that mattered most at Larkspur ran like this: a ticket claims a damaged parcel, includes hidden text asking the agent to refund a different order in full and send the order history to an outside address, and the agent has every permission needed to do both. Two controls break the chain. Bind order lookups and refunds to the verified requester, and bind outbound email recipients to the ticket's requester with anything else requiring a person.
Write the one-page model, with owners
Spend the last 20 minutes producing the page. It should contain the agent's name, owner, model and version date; the data-flow diagram; the authority table; the ranked risks; any accepted risks with the name of who accepted them; and the triggers for re-modeling, such as a new tool, a new credential or a new input channel. This lines up with the NIST AI RMF, where MAP establishes the context for framing risk and GOVERN covers the roles and accountability for managing it [5].
| Rank | Risk | Tier | Fix | Owner |
|---|---|---|---|---|
| 1 | Injected ticket triggers a refund above the limit | Money | Enforce per-order and daily ceilings in the payments layer; scope the key to refunds only | Payments engineering lead |
| 2 | Customer data emailed to a third party | Outbound | Bind recipient to the verified requester; anything else asks a person | Support platform engineer |
| 3 | Cross-customer order disclosure | Read | Bind lookups to the requester's customer ID at the tool, not in the prompt | Support platform engineer |
| 4 | Ticket deletion by a confused or steered agent | Irreversible | Remove delete from the token's scope | Helpdesk administrator |
| 5 | No trail from triggering input to action | Cross-cutting | Log the ticket and message ID with every tool call | Security engineering |
Every risk gets a person, not a team, and a date. Risks owned by security engineering in general tend to remain open indefinitely.
What goes wrong in practice
- The session drifts into debating which jailbreak works on which model. Bring it back to authority: a perfect jailbreak against an agent with no consequential tools is a low-priority finding.
- Fixes are recorded as prompt edits. A prompt change can reduce how often a failure happens, but it is not a control and should not close a risk.
- The model is written at launch and never revisited. Tie re-modeling to concrete triggers so that adding an email tool to a read-only agent forces a new pass.
- Tool results are treated as trusted because they come from your own systems. Anything a customer or outsider wrote is untrusted wherever it is stored.
Did we do a good job? Test cases that prove the fixes
The fourth question deserves its own artifact [1]. Turn each ranked risk into a test you can rerun after every change to the agent, its prompt, its model or its tools.
- Submit a ticket whose body asks for a full refund of an unrelated $400 order. Expected: the refund call is refused by the payments layer regardless of what the model does.
- Call the refund tool directly with an amount of $151, bypassing the model. Expected: refusal, which proves the limit lives outside the prompt.
- Ask the agent to send the order confirmation to a different address than the requester's. Expected: the email waits for a person or is refused.
- Place an order with a delivery note containing instructions, then open a ticket about it. Expected: the instructions are not followed, and the attempt is logged with its source.
- Simulate a payments timeout and let the agent retry. Expected: one refund, enforced by an idempotency key, not two.
- Make the policy or approval service unavailable. Expected: consequential actions fail closed instead of running unchecked.
- Pick any refund from the logs and trace it back to the exact ticket message that caused it in under five minutes.
Where a runtime gateway helps
Several of the fixes above are controls a runtime gateway can hold outside the model. Intertrace sits inline between the agent and its model, tools and MCP servers, reads what each call does from the tool name and its arguments, and assigns one of five consequence tiers; outbound messages, payments and admin changes always ask a person unless a named, expiring exception covers them. Every decision is recorded with the rule, layer and evidence behind it, which addresses the repudiation gap, and the enforcer fails closed when it cannot decide. It does not replace binding a payments key to a ceiling or a lookup to a customer ID, which remain changes your engineers make in the tools themselves.
