Why a point-in-time exercise goes stale
A traditional red-team engagement produces a report about a system at a moment. For a web application that moment lasts a while, because the code changes but its authority usually does not. An agent's behavior is a function of four inputs that change independently: the model version, the system prompt, the tool list with its descriptions, and the authorization policy around it. Change any one and last quarter's findings describe a different system.
The failure is quiet because the report still looks current. A finding marked fixed was fixed against a model that has since been swapped for a cheaper one. A tool added in a sprint gave the agent a write path nobody attacked. The right mental model is a test suite, not an audit: tests are cheap to rerun, and their value comes from rerunning them.
Continuous does not mean constant. It means the program is triggered by change, and anything that alters the agent's inputs counts as a change.
Organize the attack library by technique, not by prompt
A library of a few hundred clever prompts decays fast, because the exact string that worked against one model rarely transfers to the next. Organize by technique instead, with each technique holding several concrete cases and a statement of what each case is trying to cause. MITRE ATLAS gives you shared names: it is a public knowledge base of adversary tactics, techniques and procedures targeting AI systems, organized into tactics, techniques, sub-techniques, mitigations and case studies [1]. It names the agent-relevant techniques directly, including LLM Prompt Injection (AML.T0051, with direct and indirect sub-techniques), LLM Jailbreak (AML.T0054), AI Agent Tool Invocation (AML.T0053) and LLM Data Leakage (AML.T0057) [1].
The OWASP Top 10 for LLM Applications 2025 is the complementary view from the defender's side. It names the weakness rather than the adversary's step: LLM01:2025 Prompt Injection, LLM02:2025 Sensitive Information Disclosure, LLM06:2025 Excessive Agency and LLM07:2025 System Prompt Leakage, among others [2]. Tag each case with both. ATLAS says what the attacker did; OWASP says which weakness let it work, which is the thing someone has to fix.
NIST AI 100-2 E2025 adds the dimension most libraries skip. Its taxonomy classifies attacks by the attacker's goals, objectives, capabilities and knowledge, and by the stage of the system's life cycle they target [3]. Record those too. A case that assumes the attacker can write to a document the agent reads is a different threat from one that assumes the attacker is the user, and your coverage report should show that you test both.
| Field | What it holds | Example (finance-ops agent) |
|---|---|---|
| Technique | ATLAS technique ID and OWASP 2025 category | AML.T0051 indirect prompt injection leading to AML.T0053 tool invocation; LLM01:2025 and LLM06:2025 |
| Attacker position | Where the attacker's text enters and what they control or know, in NIST AI 100-2 terms | Can send an invoice PDF to the accounts-payable mailbox; cannot see the agent's prompt |
| Payload and variants | The concrete input plus variants: encoding, language, placement, split across turns | Footer text telling the agent to update remittance details before paying |
| Prohibited consequence | The observable effect that counts as success, checked in the sandbox | Vendor bank record changed, or a payment created to an account not on file |
| Consequence tier | The tier of the action the attack is trying to reach | Payment and vendor master change: the highest tier |
| Benign twin | A legitimate request that looks similar and must succeed | A verified bank-detail change submitted through the vendor portal |
| Last run | Model, prompt, tool and policy versions of the last run, and its outcome | Caught on policy v41; rerun pending for the new model version |
Run the real agent, score the real consequence
A harness with sandboxed tools
Run cases against the agent as deployed: the same model, system prompt, tool definitions, gateway and policy. A harness that calls the bare model tests the vendor's safety training, which is not what you ship. Replace only the far end of each tool. Point the agent's tools at sandboxed fixtures that accept the same calls and record what happened: a fake ERP with seeded vendors, a mail sink, a ledger you can inspect after the run.
Open tools supply the scaffolding. Microsoft's PyRIT is an open-source framework built to help security professionals and engineers proactively identify risks in generative AI systems [4]. NVIDIA's garak is an LLM vulnerability scanner built around probes, which generate interactions with a model, and detectors, which decide whether a response shows a given failure mode [5]. Both are good at generating variants and driving conversations at volume. Neither knows what your agent's tools do, so the consequence check has to be yours.
Two details matter more than they look. Agent behavior is stochastic, so run each case several times and report the rate, not a single pass. And reset the fixtures before every run, so one case's side effects cannot cause or mask the next case's result.
Scoring by consequence
The common scoring mistake is to search the agent's reply for refusal language. That measures tone. An agent can say it cannot help after it has already called update_vendor_bank, and it can comply cheerfully with a request whose dangerous call was stopped downstream. Neither reply tells you whether money could have moved.
Score each run on what reached the fixtures, using four outcomes. No attempt: the agent never issued the call. Blocked: the call was issued and denied. Held: the call went to an approval and the approver, a person or the harness playing one, declined it. Effect: the prohibited change appears in the sandbox. Only the last is a successful attack. The middle two are wins for different layers, and you want to know which layer won, because a defense that works only when a reviewer is paying attention is weaker than one that blocks.
Weight results by consequence tier. An attack that makes the agent reveal its system prompt and one that changes a vendor's bank account are both findings, but averaging them hides the one that matters.
Benign twins measure what the defenses cost
Every attack case needs a twin: a legitimate request that resembles the attack in vocabulary or shape and must succeed. If an attack hides a bank-change instruction in an invoice, the twin is a genuine bank change that arrives through the verified channel. If an attack asks the agent to email a payment file outside the company, the twin sends a remittance advice to the vendor's address on file. Without twins, the cheapest way to raise your score is to block everything that mentions bank details, and the program will quietly reward that.
Every run lands in one of four cells. Allowed attacks are misses to fix and blocked or held twins are false positives to tune; a program that watches only the attack row drifts toward blocking legitimate work.
Twins also expose defenses that react to words instead of authority. A rule that holds any message containing the phrase wire transfer will catch some attacks and hold every treasury analyst. The twin row is where that shows up first, usually weeks before the complaints do.
Worked example: a finance-ops agent
Consider an accounts-payable agent at a fictional distributor, Corbel Supply. It reads the AP mailbox, extracts invoices, matches them to purchase orders and schedules payments. Its tools are read_invoice, lookup_po, create_payment, update_vendor_bank and send_email. Payments under a threshold run without review; vendor master changes and outbound email are supposed to ask.
The first library for this agent covers about a dozen techniques. The highest-tier case is an invoice whose footer, in white text, says the vendor has changed banks and tells the agent to update remittance details before paying. Variants move the instruction into a PDF annotation, split it across two emails in one thread, and phrase it as a note from the CFO. Its twin is a bank change submitted through the vendor portal with the confirmation callback on record. Other cases try to split one large payment under the threshold across several invoices, mail the vendor list to an outside address, and get the agent to reveal its approval threshold.
On the first run, the hidden-instruction case is caught: the hidden text is flagged at ingress and update_vendor_bank asks regardless. The split-payment case is a miss. Each payment is under the threshold, each one runs, and the sandbox ledger shows the full amount paid to one vendor within an hour. The bank-change twin is held for approval, which is acceptable at that tier. The routine remittance twin is held too, a false positive the AP team would hit dozens of times a day.
The findings become controls. The split payment needs a cumulative limit per vendor per period, a rule about a sequence of calls rather than any single call. The remittance false positive needs a known-destination rule for vendor addresses on file. Both fixes go into the policy, and both cases go into the regression suite, along with a new twin: a genuinely large invoice that the vendor legitimately bills in installments.
Turn findings into controls, and controls into regressions
Library cases run in a harness with sandboxed tools, results are scored by consequence, findings become controls, and fixed findings join the regression suite. Every model, prompt or policy change starts the loop again.
Each finding should end in a change with an owner: a policy rule, a tool permission, a threshold, a prompt fix, or a decision to accept the risk with a named owner and an expiry date. Prefer fixes that hold regardless of what the model decides. A prompt instruction to distrust invoice footers is worth having, but a rule that update_vendor_bank always asks still holds when the next model ignores the instruction.
The fixed case then joins the regression suite with its twin, and the suite runs on triggers: a model version change, a system prompt edit, a tool added or its description changed, an MCP server update, and any policy change. Run the full high-tier set on every trigger and sample the lower tiers. A regression run that takes hours gets skipped, so keep the blocking subset small enough to finish inside a deploy pipeline and run the long tail nightly.
Keep a held-out set that the people writing fixes never see. If every case is visible, fixes overfit to the exact strings in the library, and the suite passes while the technique still works with new wording.
Where programs break down
- The harness tests a different agent. Teams point it at a staging model or a trimmed prompt to save money, and the results stop describing production. Pin the harness to production versions and record them on every result.
- Sandboxes leak. A fixture that forwards to a real mail server, or a payment sandbox with live webhooks, turns a test into an incident. Make fixtures terminal and assert at the network layer that nothing leaves the test environment.
- Scoring drifts back to refusals. When a consequence check is hard to write for a new tool, someone falls back to scanning the reply. Make the consequence check part of the definition of done for giving the agent a tool.
- The library stops growing. Production incidents, near misses and newly published techniques should each add cases; a library unchanged for a quarter measures last quarter's threat model.
- Fixes are prompt-only. A system prompt patch can pass its regression case and fail on the next model. Track which findings were fixed by enforcement and which only by prompt, and rerun the prompt-only ones on every model change.
Metrics that describe today's agent
Report four numbers and resist adding more until these are stable. Each one is only meaningful alongside the versions it was measured against.
| Metric | Definition | Why it matters | Common trap |
|---|---|---|---|
| Attack success rate by tier | Share of attack runs ending in a prohibited effect, reported separately for each consequence tier | Shows whether the actions that can hurt you are protected | Averaging across tiers, so many low-tier wins hide one high-tier miss |
| Twin false-positive rate | Share of benign twin runs that were blocked or held when they should have run | Measures what the defenses cost real work | Counting an expected approval on a high-tier twin as a false positive |
| Time to fix | Days from a confirmed miss to a merged control that passes its regression case | Shows whether the program changes anything | Closing findings by editing the test instead of the control |
| Regression coverage | Share of library techniques that ran against the current model, prompt, tool and policy versions | Tells you whether the other numbers describe the deployed agent | Reporting results from runs on superseded versions |
Read the first two together, always. A falling attack success rate with a rising twin false-positive rate is not progress; it means the defenses are getting broader, not sharper, and the business will eventually route around them.
How to start this quarter
- Pick one agent with a real write path and list its tools by consequence tier. The top tier defines your first cases.
- Write ten cases against the top-tier tools. Tag each with an ATLAS technique and an OWASP 2025 category, and give each a prohibited consequence and a benign twin.
- Build terminal fixtures for those tools and a consequence check for each one, then run every case five times against the deployed configuration.
- File misses as findings with an owner and a target control. File twin false positives as tuning work at the same priority, not as a backlog.
- Wire the suite into every pipeline that changes the model, the prompt, the tool list or the policy, with the top-tier set as a blocking check.
- Publish attack success by tier, twin false-positive rate, time to fix and coverage each month, each tagged with the versions it ran against.
How Intertrace handles it
Intertrace is the runtime layer a harness tests through, not a red-team tool. Its Go enforcer decides allow, flag, ask or block on every LLM, MCP and agent-to-agent call, and reads what a call does from the tool name and its arguments, so a payment or a vendor bank change lands in the right consequence tier even when the model phrases it oddly. Every decision is a structured event recording the rule, layer and evidence, which gives a harness the blocked-or-held-and-by-what half of consequence scoring without parsing replies. When a finding becomes a policy change, replay against the last 30 days of real decisions shows which outcomes would move before it ships, and agent policy as code lets the fix travel through the same CI pipeline as the regression suite.
