Fail-closed ExtAuthz on a production agentgateway

Envoy-style ExtAuthz is only as strict as the failure mode. If the filter treats RPC errors as allow, you have built a circuit breaker that opens the front door. Our Go enforcer is the opposite: unauthenticated, unauthorized, or unreachable checks deny. That includes promptGuard and ExtMCP inspect, not only the bearer-token check. The user-visible class is 403. The code paths for deny and for error must not share a fallthrough that forwards to the model.

Where this runs

The product data plane is Fly agentgateway at https://intertrace.fly.dev, pinned v1.3.1, started with Intertrace-compiled --file YAML. Clients present Bearer itr_rt_*. All planes are live: LLM /v1/*, MCP /mcp, Inference, Service /service/* with a per-tenant registry, and A2A /a2a with deep message/send inspect. The Intertrace Go process sits beside the gateway and serves ExtAuthz, promptGuard, ExtMCP, and POST /v1/actions/authorize. Railway gateway-v2 is not this path. Do not tune fail-closed behavior on the legacy proxy and call it production.

Agentgateway owns connectivity: routing, streaming, protocol translation, rate limits at the edge. Intertrace owns security decisions. That split (ADR 0005) is easy to draw and easy to violate. The violation looks like “if the enforcer is down, just proxy so the demo stays green.” The correct sentence is: if the enforcer is down, the gateway is a 403 factory, and that is an availability incident you caused on purpose because the alternative is a confidentiality incident you cannot unwind.

What ExtAuthz actually decides

HTTP ExtAuthz receives client request attributes as headers plus an optional body (includeRequestBody). Allow is HTTP 200 with decision and permit response headers. Deny is HTTP 403. Transient authorizer failures also deny—mirroring agentgateway failureMode default Deny. The handler refuses to run if the authorizer is not configured. That is a startup concern, not a runtime maybe.

1. Authenticate the runtime key. ExtAuthz calls POST /api/runtime/verify-key. Tenancy is key-derived. A missing, malformed, revoked, or expired itr_rt_* is INVALID_API_KEY or sibling denials. There is no anonymous LLM path.
2. Map the request into POST /v1/actions/authorize when the plane requires a decision (tools, high-impact Service writes, A2A message/send, and other execution boundaries). Do not reimplement policy in the ExtAuthz adapter.
3. On MCP, ExtMCP CheckRequest inspects list/call/resources/prompts traffic. tools/call maps into authorize and must consume an execution permit before Pass.
4. promptGuard runs on LLM bodies. Protected assets always ForceAI (ADR 0004). Classifier timeout on those paths is deny, not skip.
5. Backend-phase ExtAuthz for Service and A2A verifies and consumes the permit. Missing verifier configuration is PERMIT_VERIFIER_UNAVAILABLE, which is a deny.

Body-too-large, body-required-but-missing, MCP header/body method disagreement, shape limits, and tenant RPM limits are denials. They are not “let it through and log.” JSON-RPC application errors can still be HTTP 200 with jsonrpc.error; policy deny stays 403. Mixing those status codes is how a client smoke test goes green while the call was never authorized.

Failure modes that look like availability

verify-key unreachable

Every model call depends on mapping itr_rt_* to an org. If POST /api/runtime/verify-key is down, cold requests deny. Cached identity for already-primed keys may exist in the enforcer; it must be short-lived and must not outlive revocation. The control-plane-down runbook is explicit: high-impact paths stay fail-closed. A cache that converts a revoke into a five-minute open proxy is not a cache. It is a bug with a TTL.

Authorizer panic, timeout, or empty config

writeDeny(AUTHORIZER_UNAVAILABLE) exists because a nil authorizer used to be the kind of thing a refactor could leave behind. Timeouts must map to deny with a reason code (including MCP_CLASSIFIER_TIMEOUT and A2A_CLASSIFIER_TIMEOUT where those planes require a rich path). Empty allow because “the RPC returned nothing” is the classic ExtAuthz footgun. Treat unknown decision enums as deny. Do not default.

Dashboard ingest is slow

Evidence reporter failures must never change ExtAuthz allow/deny. Events go to gateway_events via SecurityEventEnvelope on an asynchronous path. If you await ingest before returning 200 to agentgateway, you have coupled TTFT to Supabase. If you skip deny because ingest failed, you have coupled security to a queue. The correct coupling is: decide locally, emit in the background, page if emit lags, never invert the decision because telemetry hurt.

Debug allow-all

Local development wants INTERTRACE_DASHBOARD_URL unset and a local-dev key. That bypass must not ship in the Fly image. A verify-key stub that returns valid: true for a fixed org is how a staging cluster becomes a cross-tenant incident. Feature flags that fail open when unset will be unset in the one region you forgot. Default deny. Name the allow. Prove it in smoke tests that expect 403 when the enforcer is killed.

ADR 0004 is part of fail-closed

Fail-closed is not only “RPC error equals deny.” It is also “we will not grow a regex library until every paraphrase is listed.” CI freezes pattern counts in gateway-go/internal/classifier/regex.go and internal/riskgate/gate.go. New misses go to intent-router prototypes, classifier eval, tenant OPA, or ForceAI on protected assets. Benign AI-skip is opt-in (INTERTRACE_INTENT_ROUTER_ALLOW_SKIP) and default off. It never applies to protected assets.

A regex miss that you “fix” by adding a pattern is a fail-open of a different kind: you encode last week’s attack and ship a diff that looks like security. A classifier timeout that you “fix” by skipping AI on LOW is fail-open on the class of traffic attackers will mimic. Protected assets always take the AI path. That costs latency. It is cheaper than a regex file that cannot be reviewed.

What must be up

• verify-key: dashboard API that maps itr_rt_* to org, project, agent, asset, and status. Internal secret or ADR 0007 workload token with audience intertrace.gateway.verify-key and capability verify-key.
• Enforcer: gRPC/HTTP services for ExtAuthz, promptGuard, ExtMCP inspect, authorize, permit verify. Health checks that actually call Check, not only listen.
• Policy artifacts: signed packs and OPA bundles the enforcer loads. A CMS page is not a bundle. A last-known bundle may serve primed keys; missing bundles on high-impact cold paths deny.
• Permit nonce store, if you claim replay protection across machines: Redis (or equivalent) attached. Process-local jti is honest only for a single process.
• Fly metrics and logs. /metrics without ExtAuthz latency histograms is decoration.

How we live with it

Fail-closed without SLOs is cruelty. ExtAuthz is on the user-visible path. verify-key and authorize must stay small: prefix lookup, hash compare, policy evaluation, optional classifier with a budget. We scrape Fly metrics, keep the hot path measured in milliseconds, and emit gateway events asynchronously so a slow dashboard cannot stall the user. Semantic inspect is the contested budget. TLS and token checks are not where you “save” time by skipping.

Deploy ordering is part of the contract. Never roll an enforcer that cannot answer Check. Never point agentgateway at a verify-key URL that still boots into allow-all. Never cut a region’s Redis out from under permit jti and call replay protection done. Pin agentgateway v1.3.1 in deploy/agentgateway/; compile YAML in lib/agentgateway/. Pins beat “whatever latest the image pulled on Tuesday.”

Identity split still applies

Clerk authenticates humans. Convex holds entitlements. Supabase holds telemetry under org_members RLS. ExtAuthz does not consult Convex to allow a model call, and it does not consult Clerk. A runtime key is the principal. Mixing those stores on the hot path is how you turn a billing outage into an open proxy or a login outage into a total LLM outage. Entitlements can gate dashboard hubs. They must not fail-open the data plane, and they must not be a synchronous ExtAuthz dependency unless you are willing to 403 the world when Convex blips.

Reason codes and operator UX

Operators need to tell a policy deny from an outage. Reason codes exist for that: INVALID_API_KEY, AUTHORIZER_UNAVAILABLE, PERMIT_REQUIRED, PERMIT_REPLAY, MCP_HEADER_MISMATCH, MCP_CLASSIFIER_TIMEOUT, RATE_LIMITED, and the rest. User-facing copy can stay humane (“blocked by policy” versus “runtime unavailable”) without collapsing those codes in logs. Collapsing them is how you page the wrong team.

The dashboard Traffic hub shows gateway events for the org derived from the key, not from a filter the analyst remembered to apply. Findings are optional product objects; we do not auto-open an incident for every 403. Fail-closed will generate volume. Volume is not an argument for fail-open. It is an argument for better baselines, ForceAI on the right assets, and not growing regex.

Tests that prove the failure mode

• Kill the enforcer process. Agentgateway must not forward /v1 or /mcp. If it does, failureMode is wrong.
• Break verify-key (5xx or timeout). Cold requests deny. Cached primed keys expire. Revoke during cache window must still deny within the documented bound.
• Return an empty ExtAuthz body. Deny.
• Omit includeRequestBody on a path that requires a body. BODY_REQUIRED deny, not authorize-empty-arguments.
• MCP Mcp-Method disagrees with JSON-RPC method. MCP_HEADER_MISMATCH deny.
• ALLOW_MONITOR without a permit. Executor PERMIT_REQUIRED, no upstream call.
• Permit replay. PERMIT_REPLAY, no second upstream call.

Unit tests on the Go handler are necessary and not sufficient. The integration that matters is agentgateway’s failureMode plus the enforcer’s writeDeny. Those are two codebases. A green Go test with a stub gateway will not catch “the YAML set failureMode allow in one bind.” Read the compiled --file. Smoke: npm run smoke:fly-agentgateway-full after a Fly deploy. If smoke only sends a happy-path completion, it is not a fail-closed test.

Planes share the failure mode, not the classifier

LLM, MCP, A2A, Service, and Inference all hit ExtAuthz. They do not all hit the same semantic model. Service health and discovery should not wait on promptGuard. MCP tools/call should not skip authorize because chat used a compact classifier. A2A message/send has its own pre-screen and timeout reason codes. The invariant is the failure mode: if the check that plane requires is unavailable, deny. If the check that plane does not require is unavailable, do not invent a skip that also skips authentication.

YAML binds still split LLM and MCP listeners. Collapsing them onto one port without attaching both to a gateways object is a miscompile, not a simplification. Fail-closed will then look like “MCP is broken” when the bind never received ExtMCP. Read the compiled --file from lib/agentgateway/. Pins in deploy/agentgateway/ are how you stop a silent upgrade from changing failureMode. Smoke after npm run deploy:fly-agentgateway-full must include a killed-enforcer case, a bad itr_rt_* case, and a tools/call without permit case—not only a chat completion that returned 200.

Admission and shape limits

Per-tenant RPM and request-shape caps (token counts, message depth, tool cardinality) are denials. They exist so a looping agent cannot turn fail-closed into a self-DoS of verify-key and the classifier. RATE_LIMITED is not fail-open later with a retry that bypasses ExtAuthz. BODY_TOO_LARGE is not “truncate and classify the prefix”; truncation is an attacker-controlled window. If you must cap, deny the oversize request and make the client chunk. Silent truncation is a classification bug with a content-length.

On-call: 403 storms versus open proxies

When ExtAuthz denials spike, the runbook is: distinguish INVALID_API_KEY (customer rotation), AUTHORIZER_UNAVAILABLE (your page), RATE_LIMITED (admission), and policy BLOCK (working as intended). The worst response is flipping failureMode to allow to “restore the demo.” That converts an availability incident into a tenancy and data-exfil incident with no envelope to reconstruct what left. Prefer degrading product UX (dashboard banner, retry copy) while the kernel stays closed. Clerk or Convex being down should not change this: humans cannot sign in; agents still need itr_rt_* verified.

Containment (pause, kill, session terminate) is local and fail-closed. Acknowledgement to the control plane is telemetry. If ack is down, enforcement remains. Mixing those is how a dashboard outage becomes a resume-all. org_members still gates who can click containment in the UI; the enforcer gates whether the workload’s key still works. Two different principals, two different failure domains.

BYOK credential resolution, when enabled, is another ExtAuthz-adjacent dependency. If the vault lookup fails, do not forward the customer request to a shared platform provider key “so it works.” That is tenancy failure dressed as reliability. Fail closed or fail to a named degraded mode that still does not cross org credentials. ADR 0007 scoped tokens (credential:resolve audience) exist so a stolen ingest secret is not also a vault secret. Until that migration is complete, INTERTRACE_INTERNAL_SECRET remains a compatibility credential with a large blast radius—treat its leak as fail-closed for internal routes, not as a reason to disable ExtAuthz.

What we will not do

We will not fail open for “benign” paths because a product manager saw a 403 in a demo. Benign skip is an explicit opt-in on the intent router and still never applies to protected assets. We will not treat Railway gateway-v2 as a fallback when Fly ExtAuthz is unhappy. We will not add a header that means skip-authz. We will not grow regex.go to paper over a timeout we should have budgeted.

We will keep ExtAuthz small, observable, and deny-by-default. The cost is operational discipline: verify-key SLOs, deploy ordering, pin files, and smokes that expect 403. The benefit is that an unavailable enforcer is an outage, not a breach. That is the only honest accounting for a security kernel sitting in front of other people’s models. Product people (including the author, on tired nights) will ask for fail-open so a board demo never 403s. The answer is a staging key, a healthy enforcer, and a demo that includes a deny. If the deny cannot be shown, the kernel is not in the path.