A policy service that works only when every dependency is healthy is not a complete enforcement design. Production traffic encounters timeouts, cache misses, network partitions, provider errors, and restarts. Those conditions can decide whether a dangerous action runs just as surely as the policy rules do.
The right failure behavior depends on the consequence and the stage of execution. A system may be able to serve an already authorized public document from a bounded cache while refusing a new privileged change. It should not improvise that distinction during an incident. Define it before deployment, test it, and expose it to the operator.
Separate decision failure from execution uncertainty
If authorization fails before the provider is called, the system can report that execution did not begin. If the provider accepted the request but the response was lost, the outcome is unknown. Treating both conditions as a generic retryable error creates duplicate sends, refunds, and state changes.
Represent the lifecycle explicitly: requested, evaluated, approved if required, dispatched, and confirmed or uncertain. A decision record should not claim that a blocked request executed. A provider timeout should not claim that no effect occurred. These distinctions matter for recovery, customer communication, and audit evidence.
Place the policy check close enough to execution that material context can still be validated. When a queue or human approval introduces delay, recheck expiry, identity, resource, and applicable policy before dispatch. Approval at submission time cannot account for a connection revoked while the job waited.
Define a failure policy by consequence
Identify operations whose failure to run is recoverable and operations whose unauthorized execution is unacceptable. Reads of public cached content, internal drafts, sensitive exports, production mutations, and privilege changes need different treatment. The classification must come from trusted operation and resource context.
For consequential actions, a missing authorization result should stop fresh execution. Return a structured reason, a retry or review path, and a correlation identifier. Avoid falling back to a direct provider client because the gateway is unavailable. That fallback would remove the control precisely when it cannot make a decision.
OWASP's authorization guidance includes safe failure handling and permission checks on each request [1]. Translating that principle into a production service requires defining timeouts, cache rules, queue behavior, and operational ownership rather than relying on a boolean fail-closed setting alone.
| Condition | Safe default for consequential work | Operator evidence |
|---|---|---|
| Policy timeout | Do not dispatch | Stage, deadline, decision ID |
| Stale authorization | Re-evaluate or hold | Policy revision and expiry |
| Provider response lost | Reconcile before retry | Provider request or idempotency key |
| Revoked connection | Stop queued and fresh work | Revocation time and affected jobs |
Bound the usefulness of cached decisions
Cache keys need all attributes that influence the decision: tenant, principal, connection, action, material arguments, environment, and relevant policy revision. A cache keyed only by tool name can reuse a harmless read decision for a different resource or a more consequential operation.
Give cached grants explicit expiry and invalidation behavior. A short lifetime reduces the stale-authority window but does not guarantee immediate revocation. Where immediate revocation is required, use a dependable online check or an invalidation mechanism whose delivery and failure behavior are understood.
Do not cache an approval as an unbounded allow. Preserve its constraints, remaining budget, and consumption state. Concurrent requests must not each spend the same supposedly single-use authorization. Shared counters and replay prevention are correctness requirements, not optional analytics.
Make retries respect external effects
Classify operations by whether retries are safe, whether the provider supports idempotency, and whether the system can query the outcome. A unique idempotency key should identify one intended operation, not every request made by the agent. Reusing it across distinct actions can suppress legitimate work; changing it on every retry can duplicate effects.
Persist dispatch intent before sending when the workflow requires recovery after a crash. Record enough information to reconcile an ambiguous result without storing secrets. A worker restart should resume the operation state machine rather than ask the model to guess what happened from a partial transcript.
Exactly-once execution is not a property an authorization token can guarantee across arbitrary external APIs. A local single-use check can prevent repeated dispatch through that service, while provider idempotency and reconciliation handle the downstream uncertainty. Document the limits where a provider offers neither.
Understand the streaming boundary
For streamed model output, bytes already delivered to the client cannot be recalled. Detection that triggers late in the stream may stop further disclosure while leaving the earlier prefix visible. An interface should not imply complete prevention merely because it terminated the connection.
Choose buffering and inspection according to the data and use case. More buffering may improve pre-delivery inspection but adds latency and can change the user experience. A tool execution must wait for complete, validated arguments; do not execute a partial streamed call because the prefix looks safe.
Measure where the time goes: identity validation, policy resolution, classifier work, approval wait, dispatch, and provider response. Separate service overhead from human waiting and model generation. A single end-to-end average hides the stage that threatens the reliability budget.
Exercise the recovery path
Test policy-service unavailability, expired grants, dropped invalidations, duplicate queue deliveries, slow provider responses, worker restarts, and loss of shared state. Check the actual resource outcome and the evidence trail. Verify that the dashboard can distinguish a policy denial from an infrastructure hold.
Give operators a bounded recovery mechanism. They may need to retry after reauthorization, reconcile an uncertain provider action, or cancel stale work. An emergency override should identify the affected scope, expiry, responsible person, and controls that remain active.
A reliable enforcement layer makes its uncertainty visible and refuses to convert missing information into broad authority. That is how availability engineering and security engineering become one operational contract rather than competing slogans.