Authorization changes are production changes
When an agent's tool list, approval rules or thresholds change, the next thousand calls behave differently. A rule meant to add an approval for large refunds that accidentally matches every refund stalls the support queue. An exception meant for one onboarding task that matches any vendor update lets an injected instruction rewrite bank details without a person seeing it. Both are outages: one of availability, the other of control.
Google's SRE team reports that in its experience a majority of incidents are triggered by binary or configuration pushes [1]. Authorization policy is configuration with unusually high stakes, and in many organizations it changes through an admin console with no review, no test and no record of what it replaced.
NIST SP 800-53 already describes what good looks like for configuration in general. Control CM-3 asks organizations to review proposed configuration-controlled changes and approve or disapprove them with explicit consideration of security and privacy impact, document the decisions, retain records and monitor change activity; enhancement CM-3(2) asks that changes be tested, validated and documented before implementation is finalized [2]. The rest of this paper is a way to meet that bar for agent authorization without a weekly change board.
The release pipeline, stage by stage
A change moves from pull request through rule tests, a 30-day replay that produces a decision diff, monitor mode, staged enforcement by agent, and an audit link from each decision to its policy version. Rollback from staged enforcement returns to the pull request, not to a console edit.
Each stage answers a different question. Review asks whether the intent is right. Rule tests ask whether the rules do what the author meant on the cases the author thought of. Replay asks what the rules do on the cases nobody thought of, which are the cases already in your logs. Monitor mode asks what happens on live traffic with no consequences. Staged enforcement limits the blast radius of anything the earlier stages missed, and the audit link makes every later decision traceable to the change.
DORA's research is a useful check on how heavy this should be. It found that change approval works best as peer review during development backed by automation, that heavyweight approval by external bodies hurts delivery performance, and that there was no evidence a more formal external review lowered change failure rates [3]. The pipeline is built to make peer review sufficient by handing the reviewer evidence, not to add a committee.
Policy in git, with tests on every change
None of this works until policy is text: the organization's baseline, a file per agent or role, and named exceptions, all in a repository with required review. Generated policy is fine if the generator's output is what gets committed, because the reviewer has to see the effective rules, not the script that produces them.
Write rule tests the way Open Policy Agent expects them. In Rego, tests are ordinary rules whose names start with test_, the opa test command discovers and runs them, it can report coverage showing which rule bodies were exercised, and the with keyword replaces input data and functions with mocks [4]. Whatever your policy language, keep those properties: tests live beside the rules, coverage exposes rules no test touches, and inputs can be mocked so each test pins one specific call.
Test the boundaries, not the middle. For a refund threshold, test the amount just below it, exactly at it and just above it. For a destination rule, test the address on file, a lookalike domain and a new address. Add one test for each incident or red-team finding that produced a rule, so that deleting the rule fails CI with the reason in the test name.
Why tighten-only layering keeps review tractable
A reviewer can only judge a change quickly if they can reason about it without reloading the whole policy. Layering makes that possible when the inner layers can only narrow. Put a floor at the organization layer: what is always blocked, what always asks, and the data-protection minimums. Let each agent's policy tighten that floor for the agent's job. Allow loosening only through exceptions that are named, scoped to one agent and set to expire.
Under that rule most changes are monotonic. An edit to an agent layer can only make that agent stricter, so the reviewer's question becomes whether it breaks legitimate work, which replay answers. A change to the organization floor touches every agent, so it gets the widest replay. The only paths that loosen are exceptions and floor changes, both rare and both visible in the diff. Without the rule, any agent edit might quietly override a baseline, and every review turns into a global analysis.
Record the deciding rule and layer on every decision. When a call is held, the event should say it was held by the organization's payment rule rather than by the agent's own list, so whoever reviews a later change knows which layer they are touching.
Replay produces the decision diff
Tests cover the cases you wrote; replay covers the cases you had. Take the proposed policy, evaluate it over the inputs of the last 30 days of real decisions for the affected agents, and compare each new outcome with the one that actually happened. The result is a decision diff: how many calls keep the same outcome, how many become stricter and how many become looser, each with examples a reviewer can open.
Read the direction of each change, not just the count. Allow → ask and ask → block are tightenings whose cost is friction, so the question is whether the volume is acceptable. Ask → allow, block → ask and block → allow are loosenings whose cost is risk, so each one needs a reviewer to confirm it was intended. A good pipeline fails the check on any unacknowledged loosening, however small.
Replay has limits worth stating plainly. It only sees traffic that happened, so it says nothing about a tool the agent has never called. It sees inputs as they were logged, so if your logs redact arguments, argument-based rules replay against the redacted form. And it cannot predict how agents and the people who run them will adapt to a new ask. Treat it as the strongest evidence you have, not as proof.
| Change in outcome | Direction | Main risk | Review action |
|---|---|---|---|
| Unchanged | None | None from this change | Sample a few to confirm the replay ran against the intended agents and window |
| Allow → ask | Tighter | Approval volume and stalled work | Estimate approver load per day and check whether a scoped pre-approval could absorb routine cases |
| Ask → block | Tighter | Legitimate work left with no path | Confirm each blocked case has another route or was never legitimate |
| Allow → block | Tighter | Silent breakage of a working flow | Notify the agent's owner and stage the change by agent |
| Block → ask | Looser | Calls that used to stop now reach a person | Confirm approvers know what to refuse |
| Ask → allow or block → allow | Looser | Consequential calls run with nobody involved | Open every case, require a second reviewer, and record a named reason in the change |
Worked example: tightening a procurement agent
Consider a procurement agent at a fictional freight company, Brackwater Freight. It reads purchase requests, checks budgets, creates purchase orders and maintains vendor records. One pull request makes three changes. Purchase orders over $25,000 now ask. Direct calls to the payments API are blocked, since payments are supposed to go through the ERP. And, to unblock an onboarding project, an exception lets the agent update vendor records without asking.
The rule tests pass. The replay over the agent's last 30 days of decisions produces this diff.
Illustrative counts from replaying 30 days of decisions: 18,412 unchanged, 236 allow → ask and 41 ask → block from the intended tightenings, and 17 ask → allow loosenings flagged as needing extra review.
Of 18,706 replayed decisions (illustrative counts), 18,412 are unchanged. The 236 that move from allow to ask are almost all purchase orders above the new threshold, and the 41 that move from ask to block are the direct payment calls. The 236 is a staffing question for the approvers, roughly eight extra asks a day, not a security one.
The 17 ask → allow results are the exception, and opening them shows the problem. It was written for vendor updates on any record, and several of the 17 are bank-detail changes on existing vendors that a person had previously reviewed. The fix narrows the exception to new vendor records created by the onboarding job and gives it a 14-day expiry. The replay rerun shows zero loosenings.
Without replay, this change would have been reviewed as a tightening, because two of its three parts were. The loosening sat in the part that looked administrative, which is where loosenings usually sit.
Monitor, stage, roll back, and keep the audit link
Replay tells you what would have happened last month. Monitor mode tells you what happens this week: the new policy evaluates live calls and records its decision next to the enforced one without acting on it. Run it through at least one full weekly cycle of the agent's work, and compare the live diff with the replay diff. If they disagree, either the agent's traffic has shifted or the replay inputs were incomplete, and you want to know which before enforcing.
Then enforce in stages. Start with one agent or with non-production environments, then widen. The SRE workbook defines canarying as a partial and time-limited deployment of a change together with its evaluation, with later stages exposing a larger population once earlier stages pass [1]. For authorization the population is agents rather than hosts, and the evaluation signals are ask volume, new blocks on calls that used to run, and reports from each agent's owner.
Rollback should mean redeploying the previous policy version from git: one command, no hand edits. Two cautions apply. Rolling back a tightening is a loosening, so take a quick look at what it reopens. And approvals or pre-approvals granted under the new version may not mean the same thing under the old one.
Link every decision to a policy version
This is the stage that pays off during an incident. Every enforced decision should carry the identifier of the policy version that produced it, alongside the rule and layer. Then the question of why a payment ran on the 12th has an answer that names a commit, a pull request and its reviewer, and the question of which decisions a bad change affected becomes a query instead of an investigation. It is also what CM-3 asks for when it calls for documented change decisions, retained records and monitoring of change activity [2]. Record the source commit rather than a build number, so the link survives rebuilds.
What to do this quarter
- Export every agent's current effective policy into a repository and require review for changes. Make the console read-only, or have it open pull requests instead of editing in place.
- Write rule tests for each consequential tool: thresholds at their boundaries, destinations on file and their lookalikes, and one test per past incident or red-team finding.
- Add a replay step to CI that posts the decision diff on the pull request and fails on any loosening that a second reviewer has not acknowledged.
- Run each new policy in monitor mode for at least one weekly cycle of the agent's work, and compare the live diff with the replay before enforcing.
- Enforce by agent or by environment, widening only when ask volume and new blocks match what the replay predicted.
- Stamp each decision with its policy version and source commit, and keep rollback to a single redeploy from git.
How Intertrace handles it
Intertrace treats agent policy as code: an agent's policy can be exported, edited and applied with the itr CLI, so changes can flow from CI through ordinary code review. Proposed access and policy changes can be replayed against the last 30 days of real decisions to show which outcomes would change before they ship. Layering runs organization → agent → exception, an agent can tighten but never loosen, and exceptions are named, scoped to one agent and expire after seven days by default and 90 at most. Each decision records the rule and layer that made it, which is the hook for tracing outcomes back to a change; monitor-then-enforce staging and commit stamps are practices you build around it.
