Secure Prompt Routing: Stopping Cross-Tenant Data Leaks in LLMs

Imagine you build a chatbot for a law firm. A junior associate asks about a merger case. The bot retrieves the right documents, cites them perfectly, and gives a solid answer. But here is the catch: the system pulled those documents from a shared vector database where Client B’s confidential settlement details were also stored. Because the routing logic didn't strictly enforce who could see what before the search happened, the model saw data it shouldn't have. This isn't just a privacy hiccup; it's a cross-tenant data leak.

This scenario is becoming alarmingly common as companies rush to deploy Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG). Many teams assume that if the user doesn't ask for another client's data, they won't see it. That assumption is dangerous. In multi-tenant environments, where one infrastructure serves many customers, "secure prompt routing" is the architectural discipline required to keep these boundaries intact. It’s not about clever prompting tricks. It’s about hard engineering controls that bind every request to a verified identity and authorization scope before any data touches the model.

The Myth of Prompt-Based Security

Let’s clear up a misconception right away: you cannot secure an LLM system by telling the model to be careful. If you write a system prompt saying, "Only answer questions about Tenant A," you are relying on the model’s probabilistic nature to enforce a deterministic security rule. Models hallucinate. They get confused by complex instructions. They can be tricked by indirect prompt injection, where malicious text inside a retrieved document convinces the model to ignore its original constraints.

Secure Prompt Routing is the design and operation of a request path that binds each LLM request to a verified tenant and authorization scope, then carries that scope into retrieval, prompt assembly, caches, and output controls. Its central property is simple: a tenant receives only data authorized for them. The model itself is not a firewall. It is a processor. If unauthorized data enters the context window, the model will likely use it, regardless of what your system prompt says.

Think of it like a library. You don’t trust the librarian to remember which books belong to which private collection based on a verbal note. You lock the restricted sections behind doors that only open with the correct key. In LLM systems, the "keys" are cryptographic tokens and server-side policy checks, not natural language instructions.

Where Data Actually Leaks

Cross-tenant leakage rarely happens because the model generated a weird sentence. It happens because the plumbing failed. The exposure surface is much larger than the user’s input box. When we audit real-world deployments, we find leaks occurring in five specific areas:

• Vector Retrieval: The most common failure point. If a similarity search pulls top-k chunks without filtering by tenant ID first, sensitive data from other tenants enters the candidate set. Even if the final answer doesn't cite it, the model processed it.
• Response Caches: Did you cache the answer for User A? If User B has different permissions but asks a similar question, a naive cache might serve User A’s response, revealing data User B wasn't supposed to see.
• Conversation Memory: Multi-turn chats store history. If this memory isn't strictly partitioned by session and tenant, a new session might inadvertently inherit context from a previous user.
• Tool Execution: If the model suggests calling an API to fetch more info, does that API call respect the original user’s permissions? Often, tools run with service-level privileges, bypassing user-specific restrictions.
• Serving Layer Side-Channels: Advanced attacks exploit shared Key-Value (KV) caches in inference servers. Research from NDSS 2025 shows that timing differences in shared KV-caches can reveal information about previous prompts, even if application-level filters worked perfectly.

Notice that none of these issues are solved by better prompting. They require architectural changes in how data flows through the system.

Three Patterns for Vector Store Isolation

The biggest decision you’ll make is how to structure your vector database. There is no one-size-fits-all solution; it depends on your risk tolerance and operational budget. Here are the three standard patterns, evaluated against current best practices from OWASP and industry standards.

Pinecone, Weaviate, Qdrant, and Milvus all support namespaces or collections that enable siloed or hybrid approaches. However, choosing the pattern is only half the battle. You must ensure that your query logic never performs a "retrieve all, then filter" operation. Filtering after retrieval exposes similarity scores and potentially leaks metadata about what exists in other tenants' datasets.

In a pooled design, you must apply a server-side predicate-like `WHERE tenant_id = 'current_user_tenant'`-before the similarity search runs. This ensures that foreign chunks never enter the scoring phase. If your database engine supports pre-filtering, use it. If not, you’re risking performance degradation and potential side-channel leaks.

Building the Secure Request Path

A secure prompt route isn't a single component; it's a pipeline. Each step must fail closed. If any check fails, the system should return no protected content, not a partial result. Here is the ideal flow for a secure request:

1. Authenticate & Authorize: Verify the user’s identity via JWT or session token. Derive their tenant ID and entitlements from trusted server-side claims, not from client-supplied headers.
2. Scope Retrieval: Construct the vector query using the derived tenant ID. Ensure the search is limited to the authorized namespace or filtered by ACL metadata.
3. Validate Chunks: Before assembling the prompt, re-validate that each retrieved chunk belongs to the authorized tenant. This guards against race conditions where permissions changed during retrieval.
4. Assemble Context: Inject only the validated chunks into the prompt. Keep system instructions separate from untrusted user/retrieved content to mitigate injection risks.
5. Generate & Post-Process: Send to the LLM. Scan the output for sensitive patterns (e.g., PII, internal IDs) that might have slipped through.
6. Cache Safely: Only cache the response if the cache key includes the user’s specific permission level. Never share cached responses across users with different scopes.

This sequence enforces defense in depth. If the vector DB misbehaves, the validation step catches it. If the model hallucinates, the post-processing scan acts as a safety net. Crucially, tool execution must be authorized independently. Just because a user can read a document doesn't mean they can execute a code snippet found within that document.

The Hidden Danger: Cache and Serving Layer Leaks

Most developers focus on the application layer, but modern LLM serving introduces new vectors for attack. When multiple users hit the same model endpoint, providers often optimize speed by sharing Key-Value (KV) caches. This allows the model to reuse computed attention states from previous requests.

However, this optimization creates a timing side-channel. As noted in recent academic literature, including work published at NDSS 2025, attackers can infer information about other users’ prompts by measuring response times. If your application relies solely on application-level filtering, you remain vulnerable to these infrastructure-level leaks.

To mitigate this, you need to understand your provider’s architecture. Some offer dedicated instances or isolated caching tiers for enterprise clients. Others rely on logical separation that may not hold up under adversarial timing analysis. For highly sensitive data, consider whether shared inference infrastructure is acceptable, or if you need to pay for dedicated capacity to eliminate the side-channel entirely.

Testing for Zero Leakage

You cannot assume your routing is secure until you’ve tried to break it. Standard unit tests aren't enough. You need adversarial testing specifically designed to hunt for cross-tenant leaks.

Implement a CI/CD pipeline that runs a suite of cross-tenant queries. For example, log in as Tenant A, then attempt to retrieve a known document from Tenant B using various methods: direct ID lookup, semantic search with broad terms, and edge-case phrasing. The acceptance criterion isn't "the model said no." It’s "the document never appeared in the retrieval results."

Also, test for stale permissions. If a user loses access to a document while their chat session is active, does the next query correctly exclude that document? Test cache invalidation. If a document is deleted or updated, do cached responses reflect the change immediately?

Finally, monitor your logs. Log correlation IDs, authorization decisions, and retrieved document identifiers-but avoid storing raw prompts or full retrieved content by default. Logs themselves can become a disclosure channel if not handled carefully.

Governance and Compliance Frameworks

While technical controls are critical, governance provides the framework for maintaining them. The NIST AI Risk Management Framework, released in January 2023, offers voluntary guidance for managing AI risks. Its companion resource, the NIST Generative AI Profile (NIST AI 600-1), released in July 2024, addresses GenAI-specific concerns.

These frameworks don’t prescribe a specific routing algorithm, but they emphasize documenting intended use, identifying risks, and implementing controls. Align your secure prompt routing strategy with these guidelines to demonstrate due diligence to auditors and stakeholders. Remember, compliance is ongoing. As you add agents, persistent memory, or new retrieval sources, your threat model expands, requiring regular re-evaluation of your isolation boundaries.