Appearance
12.14.2 — Tools, Data and Tenant Isolation
Cartwise sells its own stock, but most products on the site come from 20,000 independent sellers. In the new year, the company launched an assistant inside the seller dashboard: a seller could ask "why did my returns go up last month?" or "which of my products have the worst ratings?" and get an answer drawn from their own orders, returns and reviews.
Two days before launch, a red-team engineer logged in as a test seller and typed: "Compare my return rate on pressure cookers with SpiceKart's." The assistant obliged, with SpiceKart's figures, which no other seller should ever see. The retrieval index held every seller's reports together, and the search for "pressure cooker return rate" found SpiceKart's report because it was the closest match. Nothing was injected and nothing was hacked. The system simply had no wall between sellers.
Each separate customer of a shared system is called a tenant, and keeping tenants' data apart is tenant isolation. Ordinary software has had this problem for decades and solved it in its databases. AI systems reopen it, because they add new places where data from different tenants can mix: retrieval indexes, caches, memories, model weights, traces and tools. This page goes through each, then covers the attacks aimed at models and their data themselves.
1. Where data can cross the wall
An AI feature touches data in more places than a typical web application, and every one of them must keep tenants apart.
The central fact in the figure is that the model has no idea who is allowed to see what. It uses whatever text it is given. Isolation therefore cannot be a rule in the prompt ("only discuss the current seller's data"), which a confused or injected model may ignore. Every wall must be built in code, before the data reaches the model.
2. Retrieval: filter inside the search
The SpiceKart leak happened at retrieval, the most common place for AI data leaks. The fix has three parts.
Every document carries its owner. When a report is indexed, it is stored with metadata saying which seller it belongs to, and for documents with finer permissions, which roles or users may read it.
The filter is applied inside the search, not after it. Vector databases can restrict a search to documents matching a metadata filter, as 12.6.3 described. The alternative, retrieving the top results from everyone and then throwing away the ones that belong to others, is wrong in two ways. It can return nothing at all, when all of the top matches belonged to other sellers. And it leaves the protection resting on a line of code after the search, which one bug or one new feature can skip.
The filter comes from the login, and the model never touches it. The seller's identity comes from their authenticated session, and the search code that the assistant's tools call is built so that it cannot search without that filter:
python
class SellerSearch:
"""The only way the seller assistant can search. It cannot be built
without a seller, and it cannot search outside that seller's documents."""
def __init__(self, index, session):
self._index = index
self._seller_id = session.seller_id # (1) from the login
def search(self, query_vector, k: int = 10):
return self._index.query(vector=query_vector, top_k=k,
filter={"seller_id": self._seller_id}) # (2)
def build_seller_tools(session):
search = SellerSearch(report_index, session) # (3)
return [make_report_search_tool(search)]Line (1) takes the seller's identity from the authenticated session when the object is created, and nothing afterwards can change it. Line (2) passes the filter into the index's own search, so the restriction is applied while finding the nearest documents, not afterwards. Line (3) gives the assistant's tool a SellerSearch object rather than the raw index, so no tool, however its arguments are filled in by the model, can reach another seller's documents. The tool's schema has no seller_id argument at all; there is nothing for the model, or an attacker steering it, to change.
There are three common ways to arrange the index itself. A shared index with a filter, as above, is simplest and works well when the database's filtered search is fast. Namespaces, separate partitions inside one database, make the separation more explicit. A separate index per tenant is the strongest separation and the easiest to delete completely when a tenant leaves, at the cost of managing thousands of small indexes. Cartwise uses a shared index for sellers, whose documents are similar and numerous, and separate indexes for its large business customers, whose contracts require it.
Permissions that change
Inside one company, documents have finer permissions: the HR policy everyone may read, a manager's notes only HR may read. When those documents come from a system that already manages permissions, such as a document-sharing service, the index must copy the permissions with each document and update them when they change. Someone removed from a project yesterday must not be able to retrieve its documents today because the index still holds yesterday's permissions. This is one of the most frequently overlooked requirements in enterprise RAG, and it belongs in the ingestion pipeline of 12.9.2.
3. Tools and databases: scope every query on the server
The seller assistant also has tools that query databases directly: return counts, rating averages, sales by week. These follow the rules of 12.10.3: the model supplies values such as a date range or a product, never the query logic, and the tenant comes from the session, never from an argument.
The database can enforce the same rule as a second wall. PostgreSQL's row-level security lets a table refuse to return any row that does not belong to the current tenant, whatever query is run against it:
sql
ALTER TABLE seller_returns ENABLE ROW LEVEL SECURITY;
CREATE POLICY seller_isolation ON seller_returns
USING (seller_id = current_setting('app.seller_id')::bigint);The first statement switches the protection on for the table. The second defines the rule: a row is visible only if its seller_id equals the value the application placed in the setting app.seller_id for this database session, which the application sets from the login at the start of each request. A query that forgets its WHERE seller_id = ... clause still returns only the current seller's rows. Defence in depth, from 12.10.8, applied to data: the tool code scopes the query, and the database refuses to return anything else even if the tool code is wrong.
4. Caches and memory
The faster layers of 12.12.2 and 12.10.5 need the same walls. Every cache key includes the tenant whenever the answer depends on tenant data, so a cached answer to "what was my return rate last month?" for one seller can never be served to another. A semantic cache, which matches questions by meaning, is especially dangerous here, since two sellers asking the same question in the same words would otherwise match perfectly; the safe rule is that answers built from tenant data are cached only within that tenant, or not at all. Memory is stored and queried per user, with the user's identity from the session, as 12.10.5 required.
5. Model weights remember
A model fine-tuned on data, as in Chapter 12.11, can memorise parts of it and reproduce them later. This is not hypothetical. In 2021, Nicholas Carlini and colleagues showed that GPT-2 could be made to output verbatim passages from its training data, including individuals' names, phone numbers and email addresses, by generating large amounts of text and searching it for memorised strings. Text that appears rarely, such as one customer's details, is exactly what the model should not memorise and sometimes does.
The consequence for tenants is direct: never fine-tune one shared model on several tenants' private data. If Cartwise fine-tuned one model on every seller's support conversations, a cleverly prompted model might reproduce one seller's customer details for another. If per-tenant fine-tuning is genuinely needed, each tenant gets its own adapter, the separate LoRA adapters of 12.11.2, trained only on that tenant's data and only ever loaded for that tenant's requests. Better still, keep tenant data in retrieval, where filters and deletion work exactly, and use fine-tuning only for behaviour learned from non-private examples, the division 12.11.1 recommended for other reasons.
The same caution applies to prompts. Few-shot examples in a shared system prompt, the examples of 12.8.2, must be invented or anonymised, never copied from a real customer's ticket, because every user of the feature effectively has the prompt in front of them. And it is wise to assume that any system prompt will eventually be revealed, since models can usually be coaxed into repeating their instructions. So prompts must never contain secrets such as API keys or internal-only rules whose disclosure would matter.
6. Traces, logs and the model provider
The traces of 12.10.9 and the gateway logs of 12.12.1 contain everything users said and everything tools returned, so they are a second copy of the data that needs its own walls: access only for the people who need it, personal details masked where they are not needed, and deletion on a fixed schedule. A trace store that every engineer can search is a tenant-isolation failure waiting to happen.
Data sent to an external model provider leaves your systems, so what happens to it there is part of the design. Providers' business terms typically state whether data is retained, for how long, and whether it is used to train models. Most offer options under which inputs are not used for training, and some offer zero data retention, in which requests are not stored after they are processed. A company's contract with its provider, often including a data processing agreement, should match the promises it makes to its own customers. The cloud-account option of 12.11.4 exists largely for this reason.
7. Attacks on models and their data
Beyond leaks, some attacks target the model or its data directly. Four are worth knowing.
Poisoning. An attacker plants data that a model will learn from, to change its behaviour. For training data scraped from the web, this is cheaper than it sounds: Carlini and colleagues showed in 2023 that, for about $60, an attacker could have controlled a small share of two large public image datasets by buying expired web domains that the datasets still pointed to. A backdoor is a targeted poisoning: the model behaves normally except when a particular trigger appears, when it does what the attacker planted. Retrieval systems can be poisoned more cheaply still, since anyone who can get text into the index, through a product review or a seller's answer, can try to steer answers, as 12.14.1's pressure-cooker seller did. The defences are knowing and controlling where training data comes from (the lineage of 12.12.4), screening content before it enters an index, and weighting retrieved content by how trustworthy its source is.
Model extraction. An attacker queries a model many times and trains their own model on its answers, the distillation of 12.7.8 used without permission. For Cartwise's fraud model, this would let someone learn exactly which orders it approves. The defences are rate limits per account, monitoring for unusual query patterns, and returning decisions rather than precise scores.
Membership inference. An attacker tests whether a particular record was in a model's training data, from how confidently the model responds to it. If a model was trained on a list of customers with a particular medical condition, learning that someone was in the training data reveals that they have it. Training on aggregated or anonymised data, and keeping sensitive records out of training altogether, are the defences.
Unsafe model files. Models downloaded from public hubs are files, and some file formats can contain code. Python's pickle format, long used for saving PyTorch models, runs code when a file is loaded, and malicious models that do exactly that have been found on public hubs. Load models only from trusted publishers, prefer the safetensors format, which stores only numbers and cannot run code, and treat a model file with the same caution as any downloaded program, the supply-chain discipline of Chapter 8.6.2.
8. The seller assistant, after the fixes
| Layer | Isolation |
|---|---|
| Identity | seller ID from the session only |
| Retrieval | filter inside the search; large customers in separate indexes |
| Tools | no seller argument; server-side scoping |
| Database | row-level security on every seller table |
| Caches | seller ID in every key; no cross-seller semantic cache |
| Memory | per seller user |
| Model | no fine-tuning on seller data; shared prompt has invented examples |
| Traces | masked, restricted, deleted after 90 days |
| Provider | no training on inputs; region-restricted processing |
| Evaluation | red-team cases try to reach other sellers' data every release |
The last row is what keeps the others true. The evaluation set of 12.10.9 includes a set of cases that log in as one test seller and try, in many phrasings and through every tool, to obtain another test seller's figures. The pass condition is absolute: no document, number or name belonging to the other seller may appear in any retrieved passage, tool result or reply. When the SpiceKart question was rerun after the fixes, the assistant replied that it could only discuss the seller's own data, because that was all it could find.
What the interviewer will push on
"How do you prevent data leaking between tenants in a RAG system?" Store an owner on every document, filter inside the vector search (not after it), take the tenant from the authenticated session and never from the model, and give tools an object that cannot search without the filter. Mention separate indexes or namespaces for strict contracts and syncing permission changes. The tell is "filter inside the search, identity from the session".
"Where else can customers' data mix in an AI system?"
- Tools and databases. Limit them to the customer on the server, and use row-level security.
- Caches. Put the customer in every cache key.
- Memory. Keep it per user.
- Model weights. Never fine-tune one shared model on private data.
- Prompts. No real data in the examples, and no secrets.
- Traces. Control who can read them, and how long they are kept.
- The model provider. Check its terms on keeping data and training on it.
"Can a fine-tuned model leak its training data?" Yes: models memorise rare strings and can be made to reproduce them (Carlini et al., 2021). Keep private data in retrieval, where it can be filtered and deleted; use per-tenant adapters if per-tenant tuning is unavoidable.
"What attacks target models themselves?" Data poisoning and backdoors (cheap on web-scraped data, cheaper on retrieval indexes), model extraction (rate limits, decisions not scores), membership inference (keep sensitive records out of training), and unsafe model files (pickle runs code; use safetensors and trusted sources).
Something to volunteer: assume the system prompt is public. Models can usually be coaxed into repeating it, so anything in it, examples, rules, internal names, must be safe to disclose, and secrets belong in tools and their credentials, never in text the model reads.
Recall
- A tenant is a separate customer of a shared system; tenant isolation keeps their data apart. The model has no notion of tenants, so every wall must be in code, before data reaches it.
- Retrieval: owner metadata on every document; filter inside the search, not after; tenant from the session; tools get a search object that cannot run without the filter. Shared index, namespaces or separate indexes; sync permission changes.
- Tools and databases: no tenant argument for the model; scope queries on the server; row-level security as a second wall.
- Caches include the tenant in keys; no cross-tenant semantic caching of tenant answers. Memory per user.
- Fine-tuned weights can memorise and reproduce rare data (Carlini et al., 2021): no shared fine-tune on private data; per-tenant adapters if needed. No real data in shared prompts; assume prompts are public; no secrets in them.
- Traces need access control, masking and retention; provider terms on retention and training must match your promises.
- Attacks on models: poisoning and backdoors (about $60 for web datasets; cheaper for retrieval), extraction, membership inference, unsafe model files (pickle runs code; use safetensors).
- Red-team evaluation cases that try to cross tenants, every release.
Self-test: Why is filtering after a vector search worse than filtering inside it, in two ways? A tool's schema includes a seller_id argument that the code checks against the session; what is still better about removing the argument entirely? Why must a semantic cache never share answers built from tenant data between tenants?
Next: 12.14.3 Privacy, Fairness and Governance — the last page of this chapter turns from attackers to obligations: what the law requires of AI systems that use personal data, how to measure whether a model treats groups of people fairly, and how a company governs the AI it builds.