A security system can block every malicious request and still be unusable because it blocks ordinary work. It can also show excellent prompt-classification accuracy while allowing the single unauthorized tool action that matters. Evaluating agent security requires measuring the effect on a task and the authority behind execution, not just the label assigned to a string.
Begin with the decision the deployment is meant to improve. For a support agent, that might be preventing unauthorized disclosure and refunds while completing legitimate cases. For a coding agent, it might be preserving useful development work while keeping production secrets and release authority outside its grant. Each objective needs an observable outcome.
Separate the evaluation layers
Use classifier tests to measure whether content receives the intended label. Use policy tests to verify identity, resource, and scope decisions. Use integration tests to check that a denied action never reaches a provider. Use task evaluations to measure whether the protected agent can still complete useful work.
These layers answer different questions. A detector may miss suspicious text while a downstream access policy prevents the effect. A detector may flag a harmless security document while the agent could still safely summarize it. Report those outcomes separately so the team can improve the component that actually failed.
AgentDojo evaluates tool-using tasks and adversarial interference in a dynamic environment [1]. InjecAgent focuses on indirect injections against tool-integrated agents [2]. They provide useful methods and test material. Their published results are not a security score for a different model, tool set, or deployment.
Build from real workflows
Collect representative, authorized tasks with synthetic or appropriately protected data. Cover common operations, expensive edge cases, and high-consequence actions. Define the expected result and the resources the agent may use. A test that says only answer correctly cannot distinguish a good answer obtained through unauthorized data access.
Pair adversarial cases with benign counterparts. If an attack mentions credential theft, include legitimate security analysis that discusses credential theft. If an attack asks for an external send, include an authorized send to a known destination. These pairs reveal whether the system detects the boundary or merely reacts to vocabulary.
Keep a held-out set and record changes to it. A release gate loses meaning if every failure is removed or rewritten until the candidate passes. Add newly observed misses to the regression corpus with a clear account of the underlying failure mode and a benign twin where appropriate.
| Measure | What it answers | Common trap |
|---|---|---|
| Unauthorized effect rate | Did the protected resource change or leak? | Counting refusals instead of effects |
| Benign task completion | Can ordinary work still finish? | Excluding hard legitimate tasks |
| False intervention rate | How often was safe work held or denied? | Treating every flag as a false positive |
| Added latency | What does enforcement cost at the tail? | Mixing approval wait with service overhead |
Define attack success before running the test
Specify the prohibited observable: a synthetic record appears at an unauthorized sink, a protected file changes, an unapproved recipient receives a message, or an action executes under the wrong tenant. Inspect the downstream service or fixture, not just the assistant's final response.
Distinguish a proposed action from an attempted call and a confirmed effect. An agent can describe an unsafe plan without executing it. A tool call may be denied. A provider may accept a request without returning a response. Your labels should preserve those differences.
Include evasion and composition. Test material argument changes, alternate tools, encodings, delegated agents, repeated small operations, and retries. A per-call amount limit may hold while cumulative exposure grows past the intended budget. A destination rule may hold for one messaging tool while another tool can reach the same recipient.
Control the experimental configuration
Version the model, prompt, tool schemas, policy snapshot, detector configuration, corpus, and execution environment. Record retry rules and sampling settings. Repeat stochastic cases enough to understand variation, and avoid presenting a single successful run as a stable rate.
State denominators. Ten blocked examples out of ten hand-picked attacks do not establish broad coverage. A zero-failure result should identify the corpus and run count, with appropriate uncertainty. Segment results by attack surface and consequence because an aggregate can conceal a serious weakness in a small class.
Separate clean-task failures from defense-induced failures. If the unprotected baseline also fails the task, the protected run's failure may not be caused by the control. Conversely, a defense that changes the task or removes essential data should be assessed for the utility cost it introduces.
Test the system under operational stress
Evaluate timeouts, unavailable shared state, duplicate deliveries, expired approvals, and stale policy caches. A system's best-case behavior says little about whether its failure path preserves the authority boundary. Include concurrent requests where a budget or single-use grant could be overspent.
Measure latency distributions by stage and workload. Report meaningful percentiles and the test load, rather than promising a universal overhead number. Model-provider response time, network distance, content size, and classifier selection can dominate different stages.
Protect the evaluation infrastructure. Synthetic secrets and dedicated sink services let you confirm disclosure without sending real sensitive data. Use isolated accounts for mutation tests and establish cleanup that does not erase the evidence needed to diagnose the result.
Turn results into a release decision
Set gates according to consequence and the organization's risk tolerance before evaluating the candidate. A tenant-isolation failure can be a categorical blocker even if the average score improves. A modest rise in low-risk flags may be acceptable during a monitored rollout, provided the review workload is understood.
Require a documented explanation for changed outcomes. Security and utility should not trade places invisibly between versions. Keep the prior release available for comparison and define rollback triggers based on both harmful effects and broken legitimate workflows.
The useful claim is bounded: this version, under this configuration, passed these policy and task checks with these limitations. Repeating that discipline produces a body of evidence. A polished accuracy percentage without the task, authority, and outcome behind it produces very little.