Key-derived tenancy for AI runtimes

Multi-tenant AI gateways fail in a predictable way. The proxy trusts an X-Org-Id header, a JWT claim the application minted itself, or a path prefix that happens to contain a UUID. Attackers then become another tenant. The failure is not exotic cryptography. It is a control-plane mistake: treating tenancy as a request parameter instead of a consequence of authentication.

Intertrace runtime keys are opaque, prefixed itr_rt_*, and mapped server-side. Production keys start with itr_rt_live_; non-production keys start with itr_rt_test_. The client never needs to know the UUID of the organization. The dashboard does, via membership, not via the same bearer token the agent uses. That split is the whole design.

The tenant is not a header

A header is a suggestion. A path segment is a suggestion. A JSON field named org_id is a suggestion. Any of those can be copied from a screenshot, guessed from an invite URL, or replayed from a log line that should never have been printed. The only durable binding is: this secret maps to this row, and that row already has an org_id. If the secret is wrong, there is no tenant. If the secret is revoked, there is no tenant. If the secret is expired, there is no tenant.

On the production data plane that mapping happens at ExtAuthz. Agentgateway on Fly (https://intertrace.fly.dev, pinned v1.3.1) receives Bearer itr_rt_*. The Intertrace Go enforcer does not trust the client to name its org. It calls POST /api/runtime/verify-key. The control plane looks up the key by a short prefix, hash-matches the secret, checks revocation and expiry, and returns org_id, project_id, agent identity, and asset_id. Subsequent authorize, inspect, and event emission inherit that context. They do not re-parse a client-supplied tenant field as authority.

Human keys and machine keys

A Clerk session proves a human is in an organization. A runtime key proves an agent or SDK is allowed to speak to the data plane. Using one credential for both is convenient and wrong. Humans revoke sessions; agents rotate keys; incident response needs to disable a workload without locking the CISO out of the dashboard. Mixing those lifecycles produces either an agent that cannot be killed without an outage of the investigation UI, or a human who cannot sign in because someone rotated a shared secret that was doing two jobs.

The product identity stack is therefore three stores, not one. Clerk owns session and organization membership for humans. Convex holds plan and feature flags. Supabase remains the security telemetry store with row-level security. The JWT bridge from Clerk into Supabase RLS is unglamorous plumbing. It is also the difference between a multi-tenant product and a shared spreadsheet with a login screen. Runtime keys never enter that human path. They are minted against runtime_api_keys, hashed, and presented only to the Fly edge.

Membership has one source of truth

Dashboard RLS keys off org_members. That table is the membership source of truth for telemetry, findings, evidence, and settings. Mission Control historically had organization_members. Writing to only one table is how you get “I invited them but they cannot see findings.” Triggers keep the two in sync. Clerk org membership is how a human arrives; it does not replace those tables. If you only write to Clerk, the database cannot enforce tenant isolation on events.

This matters for runtime tenancy because the org_id derived from a key must be the same org_id that RLS uses when a human later opens Traffic. If those identifiers diverge—one UUID from Clerk, another from an onboarding script, a third from a hardcoded local gateway—you will spend a week explaining why the demo “worked” and production shows an empty estate. Key-derived tenancy is only as good as the uniqueness of org_id across mint, verify, ingest, and query.

What verify-key actually returns

POST /api/runtime/verify-key is an internal control-plane call. The gateway authenticates to it with the internal secret (and, under ADR 0007, with a workload token scoped to audience intertrace.gateway.verify-key and capability verify-key). The body carries the raw runtime key. Optional project_id and agent_id, when present, must match the key’s binding. When omitted, identity is key-derived from the runtime_api_keys row. That second mode is the multi-tenant edge path: the client presents a bearer token and nothing else that could select a tenant.

• Format check first. Keys that do not start with itr_rt_live_ or itr_rt_test_ are invalid. Do not hash-compare garbage; fail closed with INVALID_API_KEY.
• Lookup is by a 16-character prefix, then bcrypt compare against key_hash. Prefix collision is expected; uniqueness is in the hash match, not in the first bytes of the secret.
• Revoked keys return REVOKED_API_KEY, not a generic 404. Expired keys return EXPIRED_API_KEY. The distinction is for operators, not for attackers: both are denials at ExtAuthz.
• A matching row still has to resolve an asset in ai_assets under that org_id and project_id. A key that points at a deleted agent is not a tenant. It is AGENT_NOT_FOUND.

On success the response includes org_id, project_id, agent_id (the agent_key string the runtime uses), asset_id (the ai_assets UUID), key_id, environment, and any expected_tools declared on the asset’s security_policy. ExtAuthz stamps that context onto the request. Authorize, promptGuard, ExtMCP, and event emission read the stamp. They do not re-derive tenancy from a second header the client might have added “for debugging.”

Last-used is telemetry, not authorization

verify-key updates last_used_at on the key row. That column is for estate views and rotation hygiene. It is not a lease. A key does not become more authorized because it was used recently, and it does not become less authorized because ingest of last_used_at failed. Authorization is the hash match plus revocation plus expiry plus asset binding. Side effects on the control plane must never be on the allow path as a liveness check that can fail open.

What the client is allowed to know

SDKs need a base URL and a bearer token. For production that is https://intertrace.fly.dev/v1 and Bearer itr_rt_*. They may also send OpenAI-compatible fields, MCP JSON-RPC, or A2A envelopes. They should not send org_id as an authorization input. If a client includes an org UUID for correlation in a trace header, the enforcer may log it as a claim to check for mismatch; it must not use it to select a tenant record. ADR 0007 makes the same rule for gateway workload identity: a handler rejects a request when a tenant in the URL or body differs from the verified identity. It does not use an unverified claim to select a row.

The dashboard is the opposite shape. A human authenticates with Clerk, lands in an org they belong to, and queries Supabase under RLS. They see keys as objects they can mint, name, bind to an asset, rotate, and revoke. They never see the raw secret after the mint response. They never paste a runtime key into a browser cookie. The investigation UX and the data plane share org_id. They do not share credentials.

Failure modes we design against

Key leak

A leaked itr_rt_* is scoped to one org, one project, and usually one agent binding. That is still a serious incident. It is not a platform-wide incident. There is no shared “Intertrace production key” that every customer’s traffic uses. Rotation is a control-plane write: revoke the old row, mint a new secret, update the workload. Traffic for that agent 403s until the new key is deployed. That is the correct outage. A leak of a header-derived design is worse: the attacker forges tenant B while continuing to use a valid-looking token for tenant A.

Key confusion

verify-key is the only promotion from bearer string to tenant. Logs must not print secrets. Access logs may print key_id and org_id after verification. Prefixes in metrics are acceptable; full tokens are not. Local gateways that still hardcode an org for “easier debugging” are a production footgun if that binary is ever pointed at a real dashboard. Fail-closed ExtAuthz does not help you if the verify-key URL is stubbed to always return valid: true for a fixed org.

Dashboard impersonation

The Supabase service role is for ingestion, admin jobs, and verify-key’s own lookup. It is never for a browser bundle. A compromised dashboard session should be able to read the caller’s org under RLS, not every org. Runtime keys cannot be used as Clerk stand-ins: they do not mint a human JWT, they do not pass Convex entitlement checks, and they do not appear in org_members. Conversely, a Clerk session cannot call the Fly LLM plane as if it were an agent. Those are different principals on purpose.

Cross-tenant event writes

Gateway events and findings are written with the org_id from verified context. A bug that takes org_id from the JSON body of an event is a tenancy bug even if ExtAuthz was correct on the request path. Ingest routes must stamp the verified tenant and reject body fields that disagree. SecurityEventEnvelope is the contract. Prefer its tenant fields after verification, not before.

• Key leak: scoped to one org, rotatable, visible in Traffic as the calling identity—not a shared platform key.
• Key confusion: verify-key is the only promotion from bearer string to tenant. Logs must not print secrets.
• Dashboard impersonation: service role is for ingestion and admin jobs, never for a browser bundle.
• Mismatch: a client-supplied org_id that disagrees with the key is a deny, not a merge.

Binding keys to assets, not to hope

An unbound key is an accident waiting for a confused-deputy. Intertrace runtime keys bind to a project and, in the edge path, to an asset. Inventory assets can be runtime-enabled without type=agent; the verifier still resolves an ai_assets row under the key’s org. expected_tools on the asset’s security_policy is not decoration. It is a baseline the MCP and authorize paths can compare against when a tool list suddenly grows a networked server the user never approved in that session.

Protected assets always ForceAI under ADR 0004. That policy is an asset property, not a client hint. If tenancy were header-derived, an attacker could present tenant A’s key and tenant B’s “this is not a protected asset” flag. Key-derived tenancy makes the asset row a server-side fact. The classifier freeze on regex.go and riskgate/gate.go is a separate decision; it assumes the asset identity is already true.

What we refuse to treat as a product path

Railway gateway-v2 and the Node deployables may still serve older bst_* traffic. They are not the product path. Do not plan tenancy work against them. Do not add a second verify-key dialect so a legacy proxy can keep trusting X-Org-Id. Clients that matter set base_url to https://intertrace.fly.dev/v1 and present itr_rt_*. If a prototype still injects org id because “the local gateway needed it,” that prototype is not documentation. It is debt.

Preloop is the control-plane foundation for discovery, agent lifecycle, and governance authoring patterns. It is not the customer UI, not the identity provider, and not the authorize source of truth. Customers never see a Preloop console. They also never present a Preloop JWT to the Fly edge as a substitute for itr_rt_*. If a Preloop process is run, it federates into Clerk plus org_members. It does not mint runtime tenants.

Operational practice

1. Mint keys in the dashboard against a named asset. Copy the secret once. Store it in the workload’s secret manager, not in a git-tracked .env that five contractors share.
2. Revoke on incident, on offboarding, and on “we are not sure who still has this.” Revocation is a row update. It should be faster than rotating a cloud IAM role.
3. Prefer one key per agent per environment. Sharing a live key across staging and production is how staging logs become a production leak.
4. Treat verify-key latency as user-visible. Cache verified key identity in the enforcer with a short TTL if you must, but never cache an allow for a revoked key across a revoke. Fail closed if verify-key is unreachable.
5. Watch for keys that never call last_used_at and for keys that call from unexpected ASNs. Estate is a projection of observed traffic plus registered assets, not a CMDB of hoped-for architecture.

Fail-closed ExtAuthz makes verify-key a dependency of every model call. That is the point. A security kernel that forwards to the model because the control plane timed out is not a kernel. The operational work is making verify-key fast, highly available, and boring: small payload, prefix lookup, hash compare, asset join, JSON out. Event emission to the dashboard is asynchronous. A slow findings insert must not stall TTFT. A slow verify-key must stall the request. Those are different queues.

Testing tenancy without lying

Integration tests that set org_id in a header and skip verify-key will pass forever and protect nothing. The tests that matter present two keys, two orgs, and assert that events, findings, and RLS queries never cross. They revoke a key and assert 403. They send a well-formed org UUID that does not match the key and assert deny. They run the dashboard queries as a Clerk-bridged user in org A and confirm org B’s gateway_events are invisible. Service-role tests are necessary for verify-key itself; they are insufficient for proving the product is isolated.

Local development can use itr_rt_test_* keys against a local enforcer. It should not use a hardcoded org in the gateway “just this once.” Every once becomes a screenshot in Slack and then a customer configuration. If you need a local-dev bypass, name it, default it off, and refuse to compile it into the Fly image.

Why this is the kernel, not a feature

Authorize, ExtMCP, execution permits, and evidence-computed readiness all assume the request already has a true tenant. If that assumption is a header, every later control is theater. A permit bound to tenant A is worthless if the caller can present it on a request that ExtAuthz labeled tenant B because the client said so. RLS is worthless if ingest wrote the row under a body field. Classifier ForceAI is worthless if the attacker chooses a non-protected asset id.

Key-derived tenancy is therefore not a billing convenience. It is the first allow/deny. Everything else—policy packs, OPA, promptGuard, MCP inspect, A2A message/send, Service registry—runs after the key has named the org. The client’s job is to keep the secret. The gateway’s job is to refuse to guess.