Tenant isolation when every customer shares the model

Teams building AI features into a SaaS product usually inherit a sound tenancy model from the rest of the application: every table has a tenant column, every query is scoped, row-level security catches mistakes. Then the AI feature adds components that were never part of that model. A vector index is created once for everyone. A prompt cache is shared because hit rates are better. A support agent's tool uses one API token for the whole platform. Each is a reasonable engineering shortcut, and each is a path from one customer's data to another customer's screen.

This paper walks through where cross-tenant leakage actually happens, the preconditions for each path, and the controls and tests that close them. The model's weights are the least interesting part of the problem.

To the provider, your whole platform is one customer

Model providers isolate their own customers from each other, and that isolation does not extend inside your account. OpenAI's prompt caching documentation, for example, states that caches are not shared across organizations [3]. That is a boundary between you and other provider customers. If every tenant of your product is served through one provider organization, the provider sees one customer, and anything shared at that level is shared across all of your tenants unless you divide it.

The same holds for everything you build around the model. Isolation is a property you construct component by component: the cache, the index, the memory store, the credentials your tools use, and the logs. Each needs its own answer to the question "which tenant is this, and who decided?"

Prompt caches turn latency into a read channel

Prompt caching reuses computation for a prompt prefix seen before, so a cached prompt returns faster than an uncached one. That makes response time depend on what other requests contained. Gu and colleagues built statistical audits to detect this in production APIs; they detected prompt caching in 8 of 17 providers and global cache sharing across users in 7 of them [1]. They disclosed results to affected providers in October 2024, and at least five providers made changes in response [1]. The attack relies on cache hits occurring on prefix matches between prompts [1].

The same channel exists in self-hosted serving. Wu and colleagues showed that KV-cache sharing in multi-tenant serving frameworks such as SGLang and vLLM, which reuse cache across users with identical token sequences, opens side channels that let one user reconstruct another's prompts, an attack they call PromptPeek [2].

The preconditions are specific, which is what makes the risk manageable. An attacker needs a cache shared with the victim, prefixes long enough to cache that contain tenant-specific data, the ability to submit prompts with a guessed prefix, and fine-grained latency measurement. On your platform, any tenant user is a candidate attacker against any other if your provider-side or self-hosted cache is shared across tenants and your system prompts embed tenant data near the start.

• Keep tenant data out of shared prefixes. Put the common system prompt first and tenant-specific content after it, so the cacheable shared part contains nothing secret.
• Partition the cache by tenant where you control it. In self-hosted serving, include the tenant ID in the cache key or run separate cache pools for tenants with sensitive prompts.
• Do not mistake routing hints for isolation. OpenAI documents its `prompt_cache_key` as influencing routing, without pinning requests to a machine or guaranteeing a hit [3]; treat any such parameter as a performance feature, not a boundary.
• Reduce the timing signal you expose. Coarse latency reporting and server-side streaming buffers do not close the channel, but they raise the cost of measurement.
• Watch for probing. Many requests from one tenant that share a long prefix and vary only a short suffix, with latency being read closely, is the shape of a cache-timing attack and worth an alert.

Shared vector indexes and filters the client controls

The most common real leak is retrieval. A platform stores every tenant's documents in one index and filters by a `tenant_id` metadata field. The question is where that filter value comes from. If it is read from the request body, a header the client sets, or a parameter the model fills in through a tool call, then the tenant boundary is whatever the caller says it is.

OWASP's LLM08:2025 Vector and Embedding Weaknesses names this risk directly: in multi-tenant environments where several classes of users share one vector database, context can leak between users or queries [4]. Its prevention guidance calls for fine-grained access controls, permission-aware vector stores, and strict logical and access partitioning of datasets [4].

Three implementation details decide whether the filter holds. Apply it inside the search, not after it: post-filtering a global top-k returns fewer results for small tenants and invites a "fall back to unfiltered" fix that leaks. Treat an empty filter as an error, never as "match all". And for your most sensitive tenants, prefer a separate index or namespace per tenant, so a missing filter fails to find anything instead of finding everything.

Memory, fine-tunes, credentials and logs

Conversation memory

Long-term memory features summarize past conversations and retrieve them later. If memory is keyed by user ID alone and user IDs are not globally unique, or if a "shared team memory" spans an organization boundary, one tenant's facts surface in another's sessions. Key memory by tenant and user together, and apply the same server-side filter as retrieval.

Models tuned on pooled data

Fine-tuning one model on several customers' data turns the model into a shared store with no access control at all. Anything the model memorized can be produced for any tenant that prompts well. If you tune, tune per tenant, or tune only on data every tenant has agreed may be pooled, after de-identification.

Tool credentials

An agent that calls a CRM, ticketing system or storage bucket with one platform-wide token can reach every tenant's records, and the only thing scoping it is the model choosing the right parameters. A prompt injection in one tenant's document then becomes a query against another's data. Issue per-tenant credentials, or per-tenant scoped tokens from a broker, so the downstream system enforces the boundary even when the agent is wrong.

Logs, evaluation sets and capacity

Logs and eval datasets mix tenants by default. A support engineer debugging tenant A sees tenant B's prompts in the same trace view, and an evaluation set built from production traffic gets reviewed by a vendor or replayed into another tenant's test environment. Tag every record with the tenant, restrict access by tenant, and build eval sets per tenant or from de-identified data. Capacity is an isolation property too: one tenant's runaway agent can exhaust a shared rate limit or token budget and deny service to everyone else, so enforce per-tenant limits ahead of the shared provider quota.

Derive tenancy from the credential, then carry it everywhere

Every control above depends on one fact: the system knows which tenant a request belongs to, and that knowledge comes from authentication, not from anything the caller asserts. Resolve the tenant once, at the edge, from the credential, whether that is a session token, an API key or a runtime key. Then carry it as server-side context to every component: retrieval, cache, memory, tool broker, logs.

Any tenant field in the request body, a header, or model-generated tool arguments should be ignored for authorization. If you must accept one, for example for a partner acting across several tenants, check it against the set the credential is allowed to act for and log any mismatch as a security event. The model in particular must never choose the tenant: a tool schema with a `tenant_id` parameter is an invitation for an injected instruction to change it.

Worked example: a helpdesk platform with one shared index

Consider Brightdesk, a fictional helpdesk SaaS whose AI assistant drafts replies from each customer's knowledge base and ticket history. All customers' articles live in one vector index with a `tenant_id` field. The assistant's retrieval tool takes `query` and `tenant_id` as arguments, and the orchestrator fills `tenant_id` from the request payload the web client sends.

A tester at one customer changes the payload's tenant field and receives draft replies citing another customer's internal articles. Nothing in the model misbehaved; the boundary was a client-controlled value. A second issue appears in review: the system prompt begins with the customer's name and internal escalation policy, and the platform's self-hosted serving stack shares KV cache across all requests.

1. Remove `tenant_id` from the retrieval tool's schema. The retrieval service reads the tenant from the authenticated request context and rejects calls without one.
2. Move the largest customers to per-tenant namespaces, and make the shared index refuse queries with an empty filter.
3. Reorder the system prompt so the shared instructions come first and the customer's name and policy come after, then add the tenant ID to the serving cache key.
4. Replace the platform-wide ticketing token with per-tenant tokens issued by a broker, so a wrong tool call fails at the ticketing API.
5. Seed canary articles in two test tenants and add the cross-tenant tests below to CI.

Verification is the part teams skip. Brightdesk replayed the tester's forged-payload request after the change and confirmed that it returned only the tester's own articles, then searched the previous 90 days of retrieval logs for responses whose cited documents belonged to a different tenant than the requester. That search is only possible because each log record carried both the authenticated tenant and the tenant of every retrieved document.

Isolation tests in CI: canaries that must never cross

Isolation that is not tested regresses with the next refactor. Treat cross-tenant access as a test suite that runs on every change to retrieval, caching, memory or tool code.

1. Create at least two test tenants and seed each with canary documents, memory entries, tickets and log lines containing unique random strings.
2. Authenticated as tenant A, query for B's canaries through every path: direct search, the assistant, memory recall and each tool. Any appearance fails the build.
3. Send requests with a forged tenant field in the body, headers and tool arguments, and assert that results still come only from the authenticated tenant.
4. Send a request with no tenant context to each internal service and assert that it is rejected rather than served globally.
5. Search logs, traces, exports and eval datasets visible to A for B's canaries.
6. For self-hosted serving, run a timing test: seed a long canary prefix as B, then compare latency for that prefix and a random one as A. A consistent gap means the cache crosses tenants.

Where a runtime gateway helps

Intertrace's gateway derives tenancy from the `itr_rt_*` runtime key during authentication, never from the request body, and that key-derived tenant scopes every LLM, MCP, service and agent-to-agent call it decides. It applies access rules and DLP to tool-call arguments, so a call can be held or blocked before it reaches a downstream system, and it records each decision with the rule and layer that made it. It does not partition your caches, indexes or memory stores; those boundaries live in your services, and the tests above are how you prove them.

What to do this quarter

1. List every shared component in your AI path (caches, indexes, memory, tuned models, tool credentials, logs, eval sets, rate limits) and record how each enforces tenancy today.
2. Find every place a tenant value is read from the client or the model, and replace it with server-side context from the credential.
3. Restructure prompts so tenant data follows the shared prefix, and namespace self-hosted caches by tenant.
4. Move tool access to per-tenant credentials, starting with tools that read customer records.
5. Add the canary suite to CI, and fail the build on any cross-tenant hit.