Intertrace is a unified AI operating platform: a security kernel, a posture and investigation UX, and a product API. Customer apps send AI traffic through a data plane. A Next.js app ingests telemetry, stores it under org isolation, and surfaces findings, assets, gateway events, reports, and settings. The tagline is protect AI with AI. The construction was not a tagline. This is the record of the problems that actually burned calendar and credibility—not a victory lap, and not a company origin story. I am writing it as the engineer who had to live with the seams.
I keep a private list of decisions I would rather not defend in a design review. Almost none of them are about model quality. They are about which proxy is allowed to grow features, whether a client-supplied header can name a tenant, which database is allowed to hold a finding, what happens when ExtAuthz times out, whether a missed jailbreak is allowed to become a regular expression, which membership table is true, whether a status dropdown is an audit trail, whether a YAML pack is a certificate, which hostname a screenshot is allowed to show, and whether composing Preloop means shipping a second console. Each of those questions has a wrong answer that ships faster.
I am not going to pretend the kernel arrived fully formed. I shipped prototypes because I needed to learn event shapes and how an SDK actually points at a base_url. I trusted an org header because the demo looked clean. I wanted one database because operational overhead is real. I wanted fail-open on tired nights because a 403 in a recorded walkthrough feels like a product failure. I wanted to add a pattern because a reviewer can see a pattern. The work was noticing those urges, writing them down as constraints, and then discovering every code path that still violated them. Constraints without archaeology are slides.
Preloop is the control-plane foundation for discovery, agent lifecycle, and governance authoring patterns. Agentgateway is the runtime. Customers see one product spine. They never see a Preloop console or an upstream agentgateway admin UI. I had to say that in review more times than I compiled YAML.
1. The first gateway is rarely the product path
I shipped Node and Railway prototypes because they were fast to iterate. gateway-connected.js, bastion-gateway-deploy, gateway-go in provider/proxy mode—those taught me event shapes, classifier latency, and how customers actually set base_url and a bearer token. They also created a gravitational well. Every new feature wanted to land on the thing that was already receiving bst_* traffic. Product managers, including the voice in my own head, argued from existing customers. Existing customers are a fact. They are not a topology.
The product path is Fly agentgateway, pinned at v1.3.1, --file YAML compiled in-repo, ExtAuthz into the Go enforcer, clients on https://intertrace.fly.dev with itr_rt_* keys. Planes live: /v1 LLM, /mcp, Inference, /service/*, /a2a. Deploy is the Fly full config; smoke is the Fly full smoke. Railway gateway-v2 is legacy compatibility. Planning cutover work against it is how you spend a quarter polishing a dead end. Saying that internally, repeatedly, was harder than writing the YAML compiler. The compiler is finite. The argument restarts every time someone files a bug on the proxy that still has traffic.
The operational rule I ended up writing down is boring on purpose. No new providers, planes, policy features, customer examples, or onboarding flows target a legacy surface. Product code must not add a requirement that Fly can fail over to Railway. Fly rollback means restoring a known-good Fly release. Compatibility changes need an actual remaining client or a security issue. Until an EOL gate is approved, retained means freeze, not a second product path. I still catch myself opening the wrong repo directory when a latency question arrives. Muscle memory is part of the challenge.
What the prototypes were good for, I will not rewrite out of embarrassment. They were how I learned that dashboard ingest on the hot path makes your SLO into your Postgres. They were how I learned that OpenAI-compatible clients will send tools you did not document. They were how I learned that MCP is not “another HTTP route.” Starving them of new detection was the discipline. Keeping them runnable for diagnosis was the courtesy. Treating them as the roadmap was the failure mode I had to keep killing.
2. Tenancy that cannot be spoofed in a header
Early drafts trusted an org id the client helpfully sent. That is a demo, not a kernel. I knew it while I was typing it. The header made local fixtures easy and Postman collections readable. It also meant that anyone who could reach the proxy could become another tenant if they knew or guessed a UUID. Production tenancy is key-derived. ExtAuthz calls POST /api/runtime/verify-key. The key maps to an organization. Every subsequent event, finding, and RLS query is bound to that org. If verify-key fails, the request does not proceed. Caller-supplied org headers never override it.
The challenge was not the happy path. The happy path is ten lines and a diagram. The challenge was every temporary debug header, every local gateway that still had a hardcoded org, every dashboard query that forgot RLS and looked fine in a service-role session, every log line that printed enough of a key to be useful to an attacker, every test that stubbed verify-key to always allow so the rest of the suite could stay green. I found one “just for integration tests” allow that had leaked into a deploy profile. Fail-closed is a property of the path you actually ship, not of the path you draw.
Human keys and machine keys are not the same object. A Clerk session proves a human is in an org. A runtime key proves an agent or SDK is allowed to speak to the data plane. Using one credential for both is convenient and wrong. I wanted it anyway, once, because onboarding copy is shorter if there is one secret. Incident response disagrees. You need to disable a workload without locking an operator out of the dashboard. You need to rotate itr_rt_* without revoking every browser session. You need Traffic to show the calling identity as a key, not as a person who happened to be logged in when a cron ran.
3. Identity, entitlements, and telemetry are three stores
I wanted one database. The market and the threat model disagreed. Clerk owns session and organization membership for humans on the platform host. Convex holds plan and feature flags. Supabase remains the security telemetry store with RLS. Mixing those concerns produces the worst of all worlds: a findings table that cannot be queried under tenant policy, or an identity system that starts looking like a SIEM, or a billing plane that now has prompt payloads. The JWT bridge from Clerk into Supabase RLS is unglamorous plumbing. It is also the difference between a multi-tenant product and a shared spreadsheet with a login screen.
The practical bugs were all boundary crossings. A hub that checked Convex for a plan flag and then queried Supabase with the service role “because the user is already entitled.” An entitlement that people treated as membership. A Mission Control view that wrote organization_members and forgot org_members. A webhook that created a Clerk org and assumed Postgres had caught up. Each crossing looks like a small glue function. Each one is a tenancy bug with a UI. I started refusing glue functions that did not name which plane they were allowed to trust.
Convex entitlements answer “may this org use this hub?” Supabase answers “what did this org’s traffic do?” Clerk answers “who is this human, in which org?” Collapsing those questions feels efficient until the first audit. I collapsed them on paper more than once and had to unroll the drawing.
Sign-in lives at /sign-in on platform.intertrace.ai. That sounds like a hosting footnote. It is not. Marketing on the apex will absorb every product path if you let it. Cookies, Clerk redirect URLs, and “open the dashboard” buttons all want a single origin. I spent more hours on host discipline than I expected to spend on classifiers. The classifiers were at least interesting. Redirect URLs are interesting only when they are wrong in production.
4. Fail closed when the interesting part is down
Product people, including me on tired nights, want fail-open so demos never 403. A security kernel that forwards to the model because ExtAuthz timed out is not a kernel. It is a reverse proxy with a linter attached. The Go enforcer is fail-closed on ExtAuthz, promptGuard, ExtMCP, and authorize plus permits. Unauthenticated, unauthorized, or unreachable checks deny. Deny and error share a user-visible 403 class. They must not share a code path that falls through.
The challenge is operational. You must make the authz path fast and observable, or you will be blamed for outages you correctly caused. Latency work and fail-closed policy are the same workstream. I learned to scrape Fly /metrics, keep verify-key small, and emit gateway events asynchronously so a slow dashboard ingest cannot stall the user. If SecurityEventEnvelope writes sit on the user-visible path, your interactive SLO is now your database. That is not a kernel design. That is an accident.
Deploy ordering is the remaining pain. Never roll an enforcer that cannot answer. Never point agentgateway at a verify-key URL that still boots into allow-all for debugging. I did that once in a local profile and found the same pattern suggested for a staging overlay “so QA can work.” QA can work with a deny they understand. QA cannot work with a silent open proxy that teaches the rest of the company the kernel is optional. Fail-closed without SLOs is cruelty. Fail-open with a comment that says temporary is how you ship the comment to production.
5. ADR 0004 — stop growing the regex
Every missed jailbreak produces a message that begins “just add a pattern.” I have written that message. I have reviewed that message. I have merged that message. Then the next paraphrase missed, and the file grew, and false positives trained operators to ignore the gateway, and the library overfit our own red-team phrasing. ADR 0004 freezes regex.go and riskgate/gate.go pattern counts in CI. New misses go to intent-router prototypes, classifier eval and feedback, tenant OPA, or ForceAI on protected assets. Benign skip is opt-in via INTERTRACE_INTENT_ROUTER_ALLOW_SKIP and default off. Protected assets never take it.
The social challenge was larger than the technical one. Reviewers want a diff that looks like security. A frozen file looks like neglect. It is the opposite. A MustCompile of yesterday’s wording is how you get a green CI and a production miss that is one synonym away. Detection quality moves to evaluated classifiers and policy you can cite. Fast-gate remains a hard floor for literal secrets and clear CRITICAL exploits. It is not the product. I had to put the freeze in CI because I did not trust myself to win the argument every Friday afternoon.
ForceAI on protected assets is the other half. Once Estate has named something protected, pass-through LOW_REGEX must not apply. Skip on “obviously benign” traffic recovers latency and creates an attacker-shaped hole: look like benign. The flag stays off unless an operator accepts that trade. I still get asked to turn it on globally to make a latency slide prettier. The slide is not the kernel. The kernel is what happens to a protected asset when someone is optimizing for the slide.
6. Membership has one source of truth
Dashboard RLS keys off org_members. Mission Control historically had organization_members. Writing to only one table is how you get “I invited them but they cannot see findings.” Triggers keep them in sync. The challenge was discovering every code path that inserted a row just this once during onboarding, every script that used the service role to patch a member, every test fixture that created a Clerk user and assumed Postgres had a row. Membership bugs do not look like security bugs until the wrong person sees a finding or the right person cannot.
Clerk org membership is how a human arrives. It does not replace org_members. If you only write to Clerk, the database cannot enforce tenant isolation on telemetry. If you only write to Postgres, the human cannot sign in. If you write to both without a trigger, you will drift. I treated the trigger as a backstop and the application as the contract, the same pattern I later used for finding status. Backstops without contracts still leave a week of bad rows. Contracts without backstops still leave a route that forgot the helper.
7. Findings need an audit story, not a status dropdown
Changing finding status without history is a toy. Status changes append to finding_status_history via trigger, and application code must pass status_change_source through withFindingChangeSource(). Evidence lives in finding_evidence. The challenge was teaching every UI mutation and API route to pass a source instead of quietly patching a column. Auditors do not care that the dashboard looked clean. They care who marked it resolved, from where, and with what attachment. I care too, because I have been the person asked to reconstruct a close from Slack.
Incidents were the adjacent trap. Auto-creating an incident for every finding trains teams to ignore both queues. I wanted the graph to look busy. Busy is not an operating model. The product exposes manual create. Correlation stays in Traffic, Graph, and Work. You cannot claim you incidented every detection unless you actually did, by hand or by a policy you can show. I would rather show a quiet incident list and a complete finding history than a pager that everyone has muted.
Gateway ingest has to feed this story without inventing it. POST /api/gateway/event validates, writes gateway_events, may create findings, updates posture. Envelope fields are preferred when extending. If the event shape drifts from the mapper, you get findings that cannot be joined to a decision, which means the audit trail starts with a shrug. Coordinating gateway payload builders, the route, and lib/ mappers is tedious. Tedious is the correct texture for evidence.
8. Policy packs are not a certificate
Industry-solution pages wanted to say compliant. I do not. Readiness is computed from evidence. Catalog wording is approved. Packs and framework mappings are inputs. The challenge is commercial: buyers ask for a badge. Selling a computed score with a claim boundary is slower and more honest. I would rather lose a slide than mint a false certificate. I have sat in reviews where attaching a pack ID was treated as 100% ready. That trains customers to lie to themselves. It also trains engineering to skip evaluation suites because the heatmap is already green.
The SSOT for this is lib/industry-solutions/: catalog, domain evaluation suites, evidence lifecycle, approved claim wording, computed readiness. There is no manual maturity override. Expiry and revocation can demote a claim. Docs must not freelance stronger language. I treat a marketing adjective near a pack name as a bug. Packing YAML is configuration. Configuration is not an attestation, not a certification, and not a government authorization. If that sentence is too long for a landing page, the landing page is asking for the wrong sentence.
9. One product spine, four hosts
Marketing lives on intertrace.ai. Auth and dashboard live on platform.intertrace.ai. Mission Control is mission.intertrace.ai. Docs are docs.intertrace.ai. Middleware 308s product paths off the apex. The challenge was not DNS. DNS is finite. The challenge was muscle memory. Internal links, Clerk redirect URLs, screenshot capture, environment examples, and “just use localhost” all wanted to collapse hosts. Customers should never see Preloop or agentgateway as separate products. Hosts exist to separate marketing, product auth, mission operations, and documentation. They are not an invitation to fork the UX.
I have merged PRs that were correct on platform and wrong on the apex, correct in the dashboard and wrong in Mission Control, correct in a local screenshot and wrong in the docs host. Each one looked like a small href. The spine is Connect → Estate → Traffic → Work → Graph → Policies. Guardian feeds Work; it is not a parallel product. Labs run against the live Fly edge, not a leftover Railway path. When a host or a hub drifts, operators invent a second way to do the same job. Two ways is how audit questions get two answers.
10. Composing Preloop without a second console
ADR 0005 says reuse, integrate, extend, contribute upstream. Preloop is the control-plane foundation for discovery, onboarding, agent lifecycle, and governance authoring patterns. It is wrapped behind lib/platform/preloop/. It is not the customer UI, not the identity IdP, and not the authorize SSOT. Adapter-only is the baseline. A VERSION pin is not a deployed control-plane cluster. Remote mode requires INTERTRACE_PRELOOP_BASE_URL and an operator who owns it. I had to keep repeating that because a pin in a repo looks like a service in a slide.
The challenge was social and technical at once. Upstream has a mature console. Upstream has a model gateway and an MCP firewall. Running those as a second data plane would have given me screenshots in a week and a dual-runtime incident later. I refused the screenshots. Mapping jobs onto the spine took longer: discovery onto Estate, session models onto Traffic, authoring inputs compiled into authorize and OPA, approvals onto existing notify/decide paths. Customers should not learn the word Preloop to operate Intertrace. Engineers should learn it to wrap the right patterns and to avoid wrapping the wrong ones.
Authorize stays ours. POST /v1/actions/authorize plus permits is canonical. Preloop CEL/YAML access rules, when reused, compile into Intertrace policy surfaces. They do not bypass the kernel. Catalog feed from agentregistry stays poll-only. I will not promote a catalog into a runtime because Estate looks empty. Empty Estate with honest gaps is better than a full Estate of YAML that never saw a key.
What I would tell myself in month one
- Name the production data plane early and starve the prototypes of new features. Fly agentgateway, itr_rt_*, v1.3.1. Railway and bst_* are not the roadmap.
- Derive tenant from the key. Never from a client-supplied org header. verify-key failure is deny.
- Keep identity, entitlements, and telemetry in the stores that match their threat models. Clerk, Convex, Supabase. Do not store findings in the billing plane.
- Fail closed on ExtAuthz, promptGuard, ExtMCP, and authorize plus permits. Make that path fast enough that you can live with the 403s you correctly serve.
- Freeze cheap detection that cannot generalize. Invest in classifiers and tenant OPA. Protected assets always ForceAI. Skip stays default off.
- Make membership and finding audit boring, triggered, and impossible to bypass from a convenient API. org_members. withFindingChangeSource(). finding_evidence.
- Do not auto-incident everything. Do not print a certificate from a pack. Readiness is evidence-computed.
- Four hosts, one spine. Compose Preloop as adapters. Never ship a second console because the demo is easier.
None of this is unique wisdom. It is the residue of building a kernel people will put in front of real model calls. The interesting bugs were almost never in the model. They were in the seams: key, org, event, finding, host, pack, plane, and the human urge to fail open so the demo stays green. I still feel that urge. The difference is that the repository now argues back: CI freeze, RLS, triggers, pins, and a product path that is allowed to 403 when the interesting part is down.
If you are composing your own kernel on someone else’s gateway, the question I would ask is not which model you classify with. It is which of these seams you have named, which ones you have frozen, and which ones you still plan to fix after the next demo. The demo will wait. The header will not become honest later by itself.