Keeping inspect inside a 50ms conversation

Sub-50ms is a class of interactive hops, not a promise that every MCP tool round-trip including the vendor model will land under fifty. The number that matters is gateway-added time: TLS, authenticate, ExtAuthz, inspect, and release. Upstream model generation and upstream MCP server work dwarf that—and must not be confused with it in marketing, in incident review, or in a slide that puts a laptop curl next to a production SLO. If we cannot say which clock we started, we are not talking about a budget. We are talking about a feeling.

This post is the honest version. Fifty milliseconds is the conversation we want inspect to stay inside on a warm path: connection reused, key verification cache-friendly, lexical floor cheap, judge either skipped under an explicit opt-in we mostly keep off or completed quickly. It is not the p95 envelope we page on. The Go enforcer publishes ProxyGatewayAddedBudget at 1500ms as the documented gateway-added budget before upstream (precheck, policy, detectors). Mission Control mirrors GATEWAY_ADDED_P95_BUDGET_MS = 1500. Those numbers coexist. Collapsing them into one slogan is how you either lie about production or give up on a tight inspect path.

What the clocks are

Gateway-added time starts when the request hits the Fly edge and ends when we are ready to call upstream—or ready to reject without calling it. It includes TLS setup when you did not reuse a connection, ExtAuthz including POST /api/runtime/verify-key, policy lookup, lexical fast-gate, optional intent routing, optional AI judge, MCP argument checks, and authorize/permit minting. It does not include the model’s first token, the MCP server’s tool body, or the customer’s own retries. It does not include dashboard ingest.

TTFT as the user sees it is gateway-added plus upstream. For chat completions, upstream dominates. A gateway that adds 40ms to a 1200ms completion is inside a 50ms conversation and still a small fraction of wait. A gateway that adds 40ms to a 15ms MCP tools/list is the whole conversation. That is why we budget inspect as an interactive hop rather than as a percentage of LLM time. Percentage-of-model is how people excuse an unbounded judge.

• TLS and connection reuse: boring, measurable, worth fixing before blaming the classifier. Cold TLS can spend the entire 50ms conversation by itself.
• ExtAuthz and verify-key: must stay small or fail-closed becomes an outage generator. This is a control-plane round trip with INTERTRACE_INTERNAL_SECRET, prefix lookup, and hash compare.
• Inspect: the contested budget—semantic classification, OPA, MCP argument checks, A2A message/send deep inspect.
• Upstream: model or MCP or HTTP service; excluded from gateway-added latency.
• Emit: SecurityEventEnvelope toward ingest; asynchronous. A slow finding write must not stall release.

Measure Fly, not localhost

Production is https://intertrace.fly.dev, agentgateway pin v1.3.1, keys itr_rt_*. Metrics live on the Fly scrape path (https://intertrace.fly.dev/metrics unless overridden). Histograms for gateway-added time and authorize duration are what we look at. p50 and p95 of those histograms are the conversation. A curl against gateway-connected.js on a laptop measures your laptop, your local OpenAI key, and whatever you did not enable. It is a useful development check. It is not a production SLO, and we will not quote it as one.

Mission Control’s event-derived snapshot reads gateway_added_latency_ms (falling back to latency_gateway_ms) from gateway_events and computes a p95 against the 1500ms budget. That is a second clock: persisted per-event fields, sampled from recent rows, not the Prometheus histogram. They should tell a similar story. If they diverge, ingest enrichment or clock skew is the bug, not a reason to prefer the number that looks greener.

Where the milliseconds actually go

TLS

Clients that open a new TLS session per chat turn will never see a 50ms inspect conversation, because they spent it on handshakes. HTTP/2 or connection reuse is part of the budget, not an optimization for later. This is unglamorous and shows up immediately on Fly metrics when a new SDK is configured naively. Fix the client before opening a classifier issue.

ExtAuthz and verify-key

Every request that needs a tenant must prove a key. ExtAuthz calls POST /api/runtime/verify-key. The handler rejects missing internal secrets, invalid prefixes, revoked and expired keys, and scope mismatches. Hash compare is calibrated so that minting cost does not imply a 250ms verify on the hot path if we can help it; api-keys/create comments on bcrypt cost for that reason. If verify-key is slow, fail-closed means the user sees 403s or timeouts, not a mysteriously open model. That is correct and also an outage if we sleep on it.

Caching verified keys is a loaded design. A cache that outlives revocation is a tenancy bug. A cache that never hits makes 50ms impossible once the control plane is a few tens of milliseconds away. The honest operational stance: measure verify latency (Mission Control also tracks verification p95 against a 5000ms hybrid-verification budget, which is a different call). Keep the ExtAuthz path hot. Do not fail open on a cache miss.

Inspect

Lexical fast-gate is the sub-millisecond to low-millisecond floor: CRITICAL patterns, obvious exploits. ADR 0004 froze growth of that library. Semantic inspect is the remainder. Protected assets always ForceAI; they will not take a LOW_REGEX shortcut to save milliseconds. Benign skip of the judge is INTERTRACE_INTENT_ROUTER_ALLOW_SKIP, default off. Turning it on recovers latency by betting that the intent router’s benign class is not attacker-shaped. We do not turn it on to make a slide.

OPA and tenant packs add CPU and, if remotely fetched, tail latency. Policy should be local to the enforcer at request time. A pack that requires a network fetch on the hot path is not a pack; it is an outage amplifier. MCP list/call and A2A message/send inspect are first-class: tool-return bodies are untrusted context. Oversized tool JSON is a common p95 story. Argument byte caps belong upstream of the judge.

Timeouts are policy

When inspect exceeds budget, we do not fail open on catastrophic classes. We fail closed, or we fall back to a stricter lexical subset that is still a deny-capable gate. Silent pass-through is how a 50ms slide becomes a bypass. The Go proxy records fail-closed on guard timeout with an explicit tag rather than forwarding uninspected tokens. Stream buffer caps that would flush uninspected output fail closed. High-impact missing agent context fails closed. Low-risk optional detectors may monitor or skip only where ADR 0002 already allowed fail_open_monitor for that class.

Fail-closed without a latency SLO is cruelty: you will 403 whenever the judge is slow, and then someone will disable the judge. Fail-open without a class distinction is a kernel that is not a kernel. The compromise is categorical. CRITICAL and protected-asset paths do not silently proceed. Greetings-class traffic is where skip, if ever enabled, earns its keep. Fusion (ADR 0009) may FLAG instead of 403 on disagreement; FLAG is still a decision with evidence, not a pass-through.

1. Lexical CRITICAL match: block, cheap, fail-closed on timeout of this stage if you cannot complete it.
2. Judge timeout on protected asset or high/critical class: block or explicit fail-closed error, never forward.
3. Judge timeout on LOW with skip disabled: fail closed or degrade to a stricter lexical gate, still deny-capable.
4. Judge timeout on LOW with skip enabled: you already accepted a hole; do not widen it by forwarding uninspected HIGH.

Emit is async or it is not emit

SecurityEventEnvelope writes to the dashboard must not sit on the user-visible path. Ingest validates, writes gateway_events, may create findings, may attach finding_evidence. That work includes Postgres, RLS-adjacent service-role writes, and sometimes posture recalculation. If we await it before returning upstream bytes, every dashboard hiccup is TTFT. Dual-write to an outbox and after() scheduling exist so the hot path can finish. If events are delayed, Traffic is stale. If events block the proxy, chat is down. We pick stale over down, then we alert on stale.

Findings and incidents are not latency features. Creating a finding is ingest-side. Creating an incident is manual in the product contract; it must never be on the gateway’s return path. A pager integration that requires a synchronous incident insert in ExtAuthz has confused detection with case management.

The 50ms conversation versus the 1500ms envelope

Why publish both? Because they answer different questions. Fifty milliseconds is the design conversation for inspect on an interactive, warm, non-ForceAI path: what we want the judge and ExtAuthz to feel like when we are not also doing DEEP_GUARD authorize with a compact classifier. Fifteen hundred milliseconds is the documented p95 budget for gateway-added time including the ugly tails: cold verify, ForceAI on a protected payroll agent, oversized MCP arguments, regional hiccups. AuthorizeFastPathBudget (500ms) and AuthorizeDeepGuardBudget (2s) further split authorize. VerificationCallBudget (5s) is the dashboard hybrid verification round-trip, which is not gateway-added time at all.

If a post quoted 50ms p95 for all production traffic including ForceAI and cold TLS, it would be false on Fly. If a post quoted 1500ms as the inspect target, we would stop trying to keep ExtAuthz and the lexical floor tight. The engineering move is to plot p50 and p95 of gateway-added time, annotate ForceAI versus not, and refuse to average them with model generation. The marketing move we will not make is implying that Claude’s 2s completion happened in 50ms because it went through Intertrace.

Fail-closed makes latency a security bug

Product people, including tired engineers, want fail-open so demos never 403. The Go enforcer does not. That couples SLO work to security work. A verify-key p95 that crawls toward seconds will page as availability and is also a denial of service we inflicted on ourselves. A judge that always takes 800ms will push p95 toward the 1500ms envelope and will tempt someone to flip INTERTRACE_INTENT_ROUTER_ALLOW_SKIP globally. The correct pressure-release is eval-backed skip on non-protected traffic, cheaper judges, connection reuse, and not inspecting 2MB tool payloads as if they were a sentence.

Railway gateway-v2 and Node bastion-gateway-deploy are not where we prove this. They are not the product path. Localhost curls are not where we prove this. Fly /metrics after a real smoke is. If the Node proxy is faster because it skipped ExtAuthz, that is not a latency win. That is a missing kernel.

A single request, labeled

Walk one warm /v1 chat completion. Connection already open: TLS ~0 on this turn. ExtAuthz plus verify-key: a few milliseconds if the control plane is close and the prefix lookup hits. Lexical floor: sub-millisecond. Intent router prototypes: cheap. Judge: the contested tens of milliseconds, forced if the asset is protected, skipped only if the operator opted into INTERTRACE_INTENT_ROUTER_ALLOW_SKIP and the route allows it. Permit minting if this turn also authorizes a tool. Then upstream generation, which may be a second or twenty, and is not in the 50ms conversation. Emit overlaps the return. If any of those inspect stages times out on a catastrophic class, the user gets a failure, not a completion that we failed to look at.

Streaming does not change the clocks. You still inspect inbound before you subscribe to upstream tokens. You still must not flush uninspected outbound buffers when the cap is hit. TTFT includes gateway-added plus the model’s first token. Reporting TTFT as if it were inspect is how a slow model makes a fast gateway look broken, and how a slow gateway hides inside a slow model. Split the traces. Fly /metrics is the gateway half. The vendor’s usage object is the other half. gateway_events can store both if ingest enrichment is correct; it cannot invent a 50ms model.

Practical checklist

• Scrape Fly /metrics. Look at gateway-added histograms, p50 and p95. Do not screenshot a local curl.
• Break out TLS, ExtAuthz/verify-key, inspect, upstream, emit. If you cannot break them out, you cannot budget them.
• Confirm emit is async. Confirm catastrophic timeouts fail closed. Confirm protected assets still ForceAI under load.
• Treat skip flags as latency features with a threat model, default off.
• Cap tool JSON. Most inspect tails are payload size, not model philosophy.
• Never compare against upstream completion time in the same sentence without labeling both clocks.

What we will not claim

We will not claim that every request through https://intertrace.fly.dev adds less than 50ms at p95. We will not claim that MCP round-trips including tool execution are 50ms. We will not claim that ForceAI on protected assets is free. We will not claim that a regex freeze makes inspect faster; it makes inspect honest, which can be slower on LOW. We will not claim that dashboard ingest latency is the user’s problem when we accidentally put it on the hot path—that one is ours.

We will claim that gateway-added time is a defined clock, that 50ms is the inspect conversation we design the warm path toward, that 1500ms is the documented p95 envelope before upstream, that events emit asynchronously, and that when inspect cannot finish in time on a catastrophic class we fail closed instead of becoming a very fast hole. Measure it on Fly. Label the clocks. Keep Postgres off the user’s critical path. That is the entire discipline.

The classifier freeze in ADR 0004 is part of this budget. Adding regex to make LOW cheaper is how inspect becomes a phrase library again. Keeping ForceAI on protected assets is how we refuse to buy p50 with a hole. INTERTRACE_INTENT_ROUTER_ALLOW_SKIP remains default off because the 50ms conversation is not worth more than a fail-closed kernel. If we cannot inspect in time, we do not pretend we did.