Multitenant Vector DB Isolation: Detecting Index Drift and Cross-Tenant Leaks in Production

A tenant filter is one line of code. Leave it out of one code path, and a vector database will happily return another customer’s documents.

It won’t throw an error. The results will look relevant, because they are relevant. They just belong to someone else.

That is the uncomfortable truth about multitenant vector database isolation: when it fails, nothing crashes. Latency stays flat. Dashboards stay green. The first alarm is often a customer email.

Here is a recent example. In June 2026, a user opened Milvus issue #50398. The report describes a collection on a managed Zilliz Cloud Dedicated cluster with partition key isolation switched on. A normal search without a tenant filter was rejected, as expected. Then the reporter tried hybrid_search. According to the report, the same filterless request went through and returned data from several tenants. The isolation switch was on. One query path simply didn’t honor it.

Most writing on this topic stops at prevention: API keys, private endpoints, row-level security. We covered that ground in our vector database security hardening guide. This article starts where that one ends. It is about detection: how to notice, while the system is live, that your tenant boundaries have stopped holding or that your index has quietly drifted away from what you tested.

Here is what you’ll walk away with:

• A four-type taxonomy of vector index drift, so you know what you are monitoring for
• A ten-mode taxonomy of cross-tenant leaks, so you know where to test
• Three documented incidents, and the lesson each one teaches
• A six-layer detection architecture with working code
• Two interactive calculators and a readiness scorecard you can use today

What multitenant vector database isolation actually means

Multitenant vector database isolation is the guarantee that a query made for one tenant can only retrieve, rank, cache or log vectors that belong to that tenant. Databases enforce it with separate indexes, namespaces, shards, partitions, payload filters or row-level security. The guarantee holds only if every query path enforces it.

That last sentence does the heavy lifting.

Isolation is decided in three places, and each one can fail on its own:

1. The database layer. Namespaces, tenant shards, partition keys, RLS policies.
2. The application layer. Code that picks the namespace, builds the filter, merges results and writes prompts.
3. The pipeline layer. Caches, rerankers, ingestion jobs and reindexing scripts that touch vectors without going through your normal query path.

The OWASP Top 10 for LLM Applications names this risk directly as “Vector and Embedding Weaknesses” (LLM08:2025). It warns that when different groups share a vector database, embeddings from one group can be retrieved for another group’s queries. Its recommended fix is a permission-aware vector store.

Notice what that guidance cannot tell you. It cannot tell you whether your deployment is actually permission-aware today, on every route, after last Tuesday’s release.

In short: Settings describe what should happen. Detection proves what did.

Why hardening alone leaves a detection gap

Hardening closes the doors you know about. Detection finds the ones you didn’t know existed.

The cost of that gap is well documented. IBM’s 2025 Cost of a Data Breach Report puts the global average breach cost at USD 4.44 million. As CyberScoop’s coverage of the report notes, organizations took an average of 241 days to identify and contain a breach, which IBM described as a nine-year low. IBM’s head of X-Force said faster detection is one of the most effective ways to reduce breach costs. Those figures cover breaches of every kind, not vector stores, so read them as context for why detection time matters. The same report found that 13% of organizations reported breaches involving an AI model or application, and that nearly two-thirds lack AI governance policies.

Vector databases add a twist. Three things make leaks here harder to spot than in a classic SQL system.

Similarity search rarely says “no.” A relational query for a missing row returns nothing. A nearest-neighbor query returns the nearest results it can find, even when they are the wrong ones, unless you add a minimum-similarity cut-off.

Data-plane logs are usually your job. Pinecone, for example, documents its audit logs as a record of user, service account and API actions on the management and control plane. Who queried what, with which filter, and what came back is typically something you log yourself. Other vendors differ, and the documentation changes, so check yours.

The LLM hides the evidence. A leaked chunk usually gets summarized or paraphrased in an answer. Nobody sees a raw row, so nobody sees a raw violation.

Think of it as the difference between a smoke detector that is installed and one whose battery was tested this month.

In short: Prevention reduces the odds of a leak. Detection decides how long a leak lives.

How five popular vector stores isolate tenants (and where each can fail)

Every major vendor offers a different isolation unit. That choice decides what can go wrong, and what you need to monitor.

Pinecone: namespaces

Pinecone’s multitenancy guide recommends one namespace per tenant and notes that upserts, queries and other data operations always target a single namespace. Its learn series describes namespaces as a physical partition of records in an index.

That’s strong isolation. But the database cannot know whether the namespace your service picked belongs to the caller. If a bug passes tenant_b for a tenant_a request, Pinecone sees a perfectly valid query and answers it.

Does namespace isolation guarantee no cross-tenant leakage? No. It guarantees that one operation touches one namespace. Whether that is the right namespace is your application’s job to get right and your monitoring’s job to confirm.

Weaviate: one shard per tenant

Weaviate’s multi-tenancy docs state that each tenant is stored on a separate shard, and that queries on multi-tenant collections must name the tenant. The architecture write-up from Weaviate adds that tenants can be ACTIVE, INACTIVE or OFFLOADED.

Two details matter for detection.

First, the data concepts page says cross-references between objects belonging to different tenants are not supported. Treat that as a test case rather than a comfort: write a test that tries it.

Second, backup scope depends on version and tenant state. The docs say backups of multi-tenant collections include active and inactive tenants but not offloaded ones, and that releases before v1.37.0 (with backports to v1.35.17 and v1.36.10) covered active tenants only. If your forensic plan relies on restoring a snapshot, check which tenants it contains. The docs also say tenant state changes are eventually consistent across nodes, so re-run your isolation tests after you change states.

Milvus: from databases down to partition keys

Milvus describes its Partition Key as a search optimization built on partitions: it narrows the scan to partitions that match the key in your filter. That wording matters. A feature designed for speed can double as isolation, but only when the filter is present. Milvus documents a partition key isolation option for stricter behavior, and the issue above shows why you should test that every search path honors it.

What’s the difference between namespace isolation and partition key isolation? A Pinecone-style namespace is an address: the API call names it, and the operation cannot span two. A Milvus partition key is a hash-based grouping chosen by your filter expression, so the filter does the work. That makes the filter the thing to test, on every query path.

The trade-off runs along a spectrum. Dedicated databases and collections isolate more strongly but cap how many tenants you can host. Partition keys scale to far more tenants but lean harder on correct filters. Zilliz’s guide to multi-tenancy strategies in Milvus lays out the options and describes the partition key route as the more scalable one.

Qdrant: payload partitioning

Qdrant’s multitenancy tutorial says that when you serve many independent users, a single collection with payload-based partitioning is the best practice. You index a tenant field with is_tenant=True, which also tells Qdrant to store one tenant’s points together.

The filter travels with each request. So the failure mode is plain: a request that forgets the filter, or builds it with an OR where it needed an AND, searches the whole collection.

Can Qdrant payload filtering be bypassed in production? Not by design, but by mistake, yes. The database applies whatever filter it receives. Qdrant also supports granular access control with JWTs, but its security documentation describes tokens that scope access by collection and by access level, read or read-write. Payload filters inside JWT claims were an older option. A Qdrant maintainer wrote in a GitHub discussion that they were deprecated in v1.16.0 because of problems with write operations.

Read together, those sources suggest that in a shared collection the tenant filter remains your application’s job, and that a collection-scoped token is the boundary Qdrant itself can enforce. That second point is our inference, so confirm it against your version. Either way, you still need the result checks described later in this guide.

pgvector: row-level security

With pgvector, isolation usually means Postgres row-level security. The PostgreSQL documentation is blunt about the exceptions: superusers and roles with BYPASSRLS always skip row security, and table owners normally skip it too, unless you run ALTER TABLE ... FORCE ROW LEVEL SECURITY.

Connection pooling adds a second trap. If your app stores the current tenant in a session variable and the pooler hands that session to the next request without resetting it, the next tenant inherits the previous tenant’s identity. Scoping the variable to a single transaction is the usual fix. In a PostgreSQL 16.15 sandbox test we ran, a value set with SET LOCAL was gone after COMMIT, while a plain session-level SET stayed in place for later statements. That is one version and a minimal setup, so test your own pooler.

That’s one connection-string mistake away from a leak. Connect your app as the table owner and every policy you wrote is decoration. The Crunchy Data guide to tenant RLS walks through safe setups. We go deeper on authorization models in our guide to enterprise RAG access control.

Why not give every tenant a private index?

Memory. The Curator paper on multi-tenant vector indexing notes that multi-tenancy is usually built either on one shared index or on a separate index per tenant, and proposes a research index meant to combine low memory use with fast per-tenant queries. Curator is a research design, not a feature of the products above. Our reading is that vendors land in different spots between those two options, which is why isolation varies so much from one product to the next.

Choosing an isolation unit: a decision matrix

Use this as a starting point. It is editorial guidance built on the vendor documentation cited above, not a vendor rule. Re-test it against your own tenant counts, compliance duties and budget.

In short: Every isolation model moves the trust boundary somewhere. Your job is to know where, and to watch that spot.

The four types of vector index drift

Vector index drift is the gradual gap between what your index returns and what it should return. It comes from embedding model changes, graph degradation, quantization error or stale source data. It lowers recall and relevance without raising errors, which is why uptime monitoring usually misses it.

Most articles lump all of this under “data drift.” That’s too vague to act on. In practice, drift has four distinct causes, and each one needs its own detector.

What is embedding drift in RAG? It’s a shift in the vectors your pipeline produces, usually after a model update or a change in preprocessing. Old and new vectors stop living in the same space, so similarity scores between them lose meaning.

Type 1: embedding model drift

Embedding providers retire and replace models. A vector made by one version is not comparable to a vector made by another, even when the model name looks the same. Mix them in one index and you get a slow split: new documents match new queries well, old documents quietly fade.

The fix starts with a boring habit: write the model name and version into every record’s metadata. Then you can ask the question that matters, “what share of this tenant’s index was made by the old model?” The Redis engineering blog on vector database challenges makes the same point from the operations side: embedding drift lowers search quality without warning, because nothing throws an error.

Type 2: graph index degradation

HNSW builds a layered graph of neighbors. Build-time settings like M and ef_construction set its quality. Heavy delete-and-reinsert traffic can leave some points unreachable. In one experiment on the SIFT dataset, reported in a paper on real-time updates to HNSW, repeated deletion and reinsertion left about 3% to 4% of points unreachable after roughly 3,000 iterations. Recall fell by about 3%, and raising the search-time ef parameter did not recover it.

That is one paper’s setup, not a statement about any vendor’s implementation, but it shows why you measure rather than assume. The Qdrant tutorial on measuring ANN recall shows how to compare approximate results with exact ones, which is the standard way to see this kind of loss.

Type 3: quantization recall loss

Compression (product, scalar or binary quantization) trades accuracy for memory. That trade is usually accepted on launch day and never revisited. But the settings were tuned for the data you had then. Elastic’s write-up on measuring the recall of quantized vector search is a good guide to checking whether the gap has widened.

Type 4: permission state drift

A document is shared with Engineering when it is indexed. Next month, legal restricts it to executives. The source system updates at once. The vector’s metadata does not. Until something reconciles the two, Engineering can still retrieve the chunk.

We cover how permissions should flow into retrieval in our guide to enterprise RAG security architecture and ACL propagation, so we won’t repeat it here. What matters for detection is that this drift is the slowest and the hardest to spot. We’ll build a reconciler for it later.

Don’t confuse index drift with context rot

They sound alike but happen at different stages. Index drift happens before retrieval: the wrong chunks come back. Context rot happens after retrieval: the model handles a long context badly even when the chunks are right. Fix the first and you might never need to touch the second.

Data engineers know this pattern from other systems. Duplicates in a ClickHouse ReplacingMergeTree table look fine until you reconcile them against the source. Vector indexes need the same habit.

In short: Four causes, four detectors. Name which kind of drift you are hunting before you pick a metric.

Ten ways tenants leak into each other

Before we build detection, we need to know what we are detecting. Here is the short version of the failure modes people hit in production:

1. The tenant filter is left out of a query
2. Hybrid or batch search skips the isolation check
3. A cache returns one tenant’s answer to another
4. Permission changes arrive late in the vector store
5. Deleted documents live on as ghost vectors
6. A privileged database role skips row-level security
7. The app trusts a tenant ID sent by the client
8. A reindex swaps aliases before the new index is correct
9. A cross-reference links objects across tenants
10. Embeddings are inverted back into text

Three recurring causes

Practitioner threads keep describing the same pattern. Treat these as community observations rather than hard statistics, but they line up well with the incidents later in this article.

1. The forgotten filter. In a r/LangChain discussion of moving multi-tenant RAG into production, the author argues that metadata filtering alone is a fragile way to separate clients. A r/Supabase thread on cross-tenant access failures adds the sibling mistake: trusting a tenant identifier supplied by the client, rather than deriving it from the authenticated session.

2. The unaudited side door. Hybrid search, batch search, “find similar” helpers and admin scripts are written later than the main search path. They often skip the review the main path got. Milvus issue #50398 is one documented example.

3. The shared cache. A semantic cache answers “close enough” questions without calling the model. That’s the point. But if the cache key ignores the tenant, “close enough” can cross a customer boundary. Researchers studying GPTCache showed in a paper on timing side channels in LLM serving that attackers can infer private attributes by probing whether a similar request is already cached. A 2026 paper on secure multi-tenant enterprise LLM design notes that cache layers remain vulnerable to cross-tenant influence at the semantic level.

What about embedding inversion and the Weaviate cross-reference case?

Embedding inversion is a different kind of problem. It does not need a bug in your filter. A paper presented at ACL 2024 reports that an attacker can infer sensitive information from text embeddings without access to the original embedding model, by training a surrogate model. The authors tested several embedding models and a clinical dataset, so treat the result as a demonstrated risk, not a measure of your own exposure. OWASP lists inversion alongside cross-context leaks in its LLM08 entry. Your defenses here are access control on raw vectors, rate limits and anomaly review rather than filters.

On Weaviate, the docs say cross-references between different tenants are not supported. A community forum thread on multi-tenant cross-references shows users hitting edge-case errors when mixing multi-tenant and non-multi-tenant collections. Nothing in it reports a cross-tenant leak. You don’t need to settle the details. You need one test that tries a cross-tenant reference on purpose.

Sizing the exposure: how bad could a filter bug get?

A leak you can’t measure is easy to ignore. This estimate turns a vague worry into a number. It is illustrative, not predictive, and every input is your own guess to refine with real logs.

With the default numbers, a filter bug that touches just 0.1% of 50,000 daily queries produces about 600 leaking responses a month. That is roughly 20 a day. Without detection, you would learn about it from a customer. The severity labels use arbitrary cut-offs of 1 and 50 leaks per month. Set your own with your risk owner.

In short: Ten modes, three recurring causes. In the cases above, the failures sat in the code around the database.

Three real incidents, and what each one teaches

Theory is cheap. These cases are documented, and each one maps to a layer of the detection architecture that follows.

Case study 1: Milvus hybrid_search skips partition key isolation (June 2026)

The challenge. A user testing a collection with partitionkey.isolation enabled wanted to confirm that tenant boundaries held. Per the issue, the setup was a managed Zilliz Cloud Dedicated cluster, PyMilvus 3.0.0 and the REST v2 API, a partition key field named tenant, and two dense vector fields. The reporter listed the Milvus server version as unknown. They checked normal search first.

The action. They then ran the same checks through hybrid_search, which combines several vector fields. Milvus issue #50398, filed in June 2026, records what they saw.

The outcome. The reporter found that the filter itself worked: the multi-tenant hybrid request excluded a control row from tenant_c. The gap was that hybrid_search accepted multi-tenant and no-tenant requests at all, and the filterless request returned results from several tenants, including tenant_c.

The reporter added that if this behavior is intentional, it should be documented, because it differs from search. A maintainer then added regression-test coverage in PR #50401 and described the expected behavior after a server-side fix. When we read the thread on October 16, 2026, it did not name a release containing the fix, so treat the status as unknown and check it yourself. Because the environment was a managed cluster with an unknown server version, the case tells you what to test, not which versions are affected.

The lesson. Isolation guarantees are path-specific. Every query API, including the ones added later, needs its own negative test. That is Layer 5 below.

Case study 2: OpenAI's March 2023 cache bug exposed other users' data

The challenge. ChatGPT used Redis to cache user information, with the redis-py library connecting a Python server to a Redis cluster, as OpenAI's own postmortem explains.

The action. On March 20, 2023, a server change caused a spike in cancelled Redis requests. The bug was in the asyncio redis-py client for Redis Cluster. A request cancelled at the wrong moment left a connection in a corrupted state, and the next response on it could land on an unrelated request.

The outcome. Some users saw chat titles from another active user's history. OpenAI later found that payment-related details of about 1.2% of ChatGPT Plus subscribers who were active in a nine-hour window may have been visible, including name, email, payment address, card type, the last four digits of the card number and the expiry date.

OpenAI said full card numbers were not exposed, and took ChatGPT offline while it fixed the bug. Its list of follow-up actions included redundant checks that the data returned by the cache matches the requesting user, a programmatic review of its logs, correlating several data sources to identify affected users, and improved logging to confirm the problem had stopped.

The lesson. This was a plain cache, not a vector database. But the failure class is the same: a shared layer in front of the data returned the wrong user's content. OpenAI's write-up adds a detail every detection designer should read twice.

Usually the corrupted connection caused an error, but sometimes the stray data happened to match the expected type, so what came back from the cache looked valid. That is why result checks have to compare ownership, not just format. The follow-up actions map to Layer 2 (check results on the way out) and Layer 1 (logs good enough to scope the damage). Our inference is that a semantic cache carries the same risks, plus similarity matching.

Case study 3: EchoLeak in Microsoft 365 Copilot (CVE-2025-32711)

The challenge. Microsoft 365 Copilot retrieves from a user's email, files and Teams chats to answer questions. Retrieval bridges trust boundaries by design.

The action. Researchers at Aim Security chained several weaknesses. A single crafted email could steer Copilot into leaking internal data, with no click from the victim, as The Hacker News reported. The EchoLeak research paper describes the full chain.

The outcome. Microsoft assigned CVE-2025-32711, described in its security advisory, with a CVSS score of 9.3 and fixed it server-side in June 2025, requiring no customer action. The reporting at the time said there was no evidence of exploitation in the wild.

The lesson. Aim Security calls this class an "LLM scope violation": untrusted content and privileged content meeting inside one context window. EchoLeak was not a multitenant bug, but a multitenant vector store has the same shape of bridge. If one tenant's content can reach another tenant's context, the boundary has to hold at the retrieval layer. Our recommendation is not to rely on the model to enforce it.

A pattern, not an outlier

Legit Security reported a related problem in GitLab Duo: a hidden instruction planted in project content, such as a merge request description, could make the assistant leak private source code that the victim's account could access, per their write-up. GitLab confirmed the issue and shipped a fix. Neither this flaw nor EchoLeak was a tenant leak in the vector-database sense.

Both were indirect prompt injection that pulled privileged data into a context an attacker could influence. We include them because they show the same retrieval boundary failing. The common thread across all four incidents is content crossing a boundary nobody was watching.

In short: Each incident was invisible until someone outside the team noticed. The architecture below exists to change who notices first.

A six-layer detection architecture

No single control catches every failure. So we stack six, each aimed at a different class of problem. If one layer misses, the next one has a chance.

How do you detect cross-tenant leaks in a vector database? Work through these steps in order:

1. Derive the tenant from identity. Take it from the authenticated session, not from a request field.
2. Log every query. Record tenant, namespace, filter, result IDs and a timestamp in an append-only store.
3. Validate every result. Compare each returned ID's owner with the caller and block the response on any mismatch.
4. Seed canaries. Plant unique marker documents in a canary tenant and alert if one ever appears elsewhere.
5. Test in CI. Attempt cross-tenant access on every query path and fail the build if it succeeds.
6. Reconcile on a schedule. Compare vector metadata with source ACLs and remove stale or deleted entries.

The rest of this section shows how to build the key layers.

Layers 1 and 2: log it, then verify it

Start here, because these two are cheap and they catch the most common leaks.

Layer 1: a tamper-evident audit log

A log is only evidence if nobody can quietly edit it. The usual trick is a hash chain: each entry includes the hash of the one before it. Change any old entry and every later hash breaks.

Log IDs rather than content. You want to answer "which chunks did this query return," not duplicate sensitive text into a second system. In a multi-instance service, send entries to a central append-only sink, such as a write-once bucket or a log service with immutability, so the chain does not fork per process.

Python Code: hash-chained audit log for vector queries:

import hashlib import json import time   class AuditLog: """Append-only, hash-chained log of every vector query."""  def __init__(self, sink): self.sink = sink # file, queue, or WORM bucket writer self.prev_hash = "0" * 64  def record(self, tenant_id, namespace, filter_expr, result_ids): entry = { "ts": time.time(), "tenant_id": tenant_id, # from the authenticated identity "namespace": namespace, # what was actually queried "filter": filter_expr, # the exact filter sent "result_count": len(result_ids), "result_ids": result_ids, # IDs only, never content "prev_hash": self.prev_hash, } payload = json.dumps(entry, sort_keys=True) entry["entry_hash"] = hashlib.sha256(payload.encode()).hexdigest() self.sink.append(entry) self.prev_hash = entry["entry_hash"] return entry["entry_hash"]   def verify_chain(entries): """Return the index of the first broken link, or None if intact.""" prev = "0" * 64 for i, e in enumerate(entries): body = {k: v for k, v in e.items() if k != "entry_hash"} expected = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest() if e["prev_hash"] != prev or e["entry_hash"] != expected: return i prev = e["entry_hash"] return None

Run verify_chain on a schedule. A broken link is itself an alert.

A r/Rag builder describing a RAG service where the LLM never touches raw documents lists a tamper-evident log of every query as one of its core guarantees. That's a community example rather than a standard, but it matches the practice.

Make the logs usable by more than your security team. If you already trace agent runs, the same spans can carry tenant and filter fields. Our guide to multi-agent observability with MLflow shows how to attach that kind of context to traces. And because agents often call the vector store with a service credential, see how to handle non-human identities for agentic AI so the tenant in your log is the tenant who actually asked.

Layer 2: result-tenant validation

This is an assertion on the way out. After the database answers, check that every returned ID belongs to the caller. If not, alert and suppress the response.

It is a belt-and-braces check that catches the Milvus-style bug even when the database layer is wrong. OpenAI's own list of follow-up actions after the incident described above included redundant checks that the data returned by its cache matches the requesting user.

Python Code: block responses that contain another tenant's results:

class TenantLeakError(Exception): pass   def validate_results(requesting_tenant, hits, owners_of, alert): """ hits: list of hits, each with an "id" owners_of: function list_of_ids -> {id: owning_tenant}, read from your source of truth in ONE batched lookup alert: function that pages the on-call engineer """ ids = [h["id"] for h in hits] owners = owners_of(ids) # An ID with no known owner is treated as foreign: fail closed. foreign = [i for i in ids if owners.get(i) != requesting_tenant] if foreign: alert( severity="critical", tenant=requesting_tenant, foreign_count=len(foreign), sample_ids=foreign[:20], ) # Suppress the response. A blocked answer beats a leaked one. raise TenantLeakError(f"{len(foreign)} foreign results blocked") return hits

Where does tenant_of get its answer? For low-risk apps, owner metadata stored with each vector is a reasonable source. For high-stakes data, look it up in your relational system of record, so a corrupted index cannot vouch for itself.

In short: The log tells you what happened. The validator stops it from reaching anyone.

Layer 3: recall tracking for drift

Leaks are loud once you know where to look. Drift is quiet. To hear it, you need a baseline.

Build a golden set of 100 to 500 queries per important tenant, with known-good answers. Run them hourly or daily against both your approximate index and an exact (brute-force) search over the same tenant's vectors. The overlap between the two is your recall.

Recall@k asks a simple question: of the k true nearest neighbors, how many did the approximate index find? A baseline of 0.92 means it finds 92% of them. Drift is how far you've fallen from that starting point, as a percentage.

With the defaults, recall falling from 0.92 to 0.78 is a 15.2% drift, which crosses a 10% alert line. Nobody changed any code. That's the point. The 10% line is a starting point, not a standard, so tune it to your own baseline noise.

Python Code: golden-query recall monitor:

def recall_at_k(ann_ids, exact_ids, k): return len(set(ann_ids[:k]) & set(exact_ids[:k])) / k   def run_golden_set(golden_queries, ann_search, exact_search, k=10): """Average recall@k over a fixed set of golden queries for one tenant.""" scores = [ recall_at_k(ann_search(q, k), exact_search(q, k), k) for q in golden_queries ] return sum(scores) / len(scores)   def check_drift(tenant, current, baseline, emit_metric, alert, threshold_pct=10): drift_pct = (baseline - current) / baseline * 100 emit_metric("vector.recall_at_k", current, tags={"tenant": tenant}) if drift_pct > threshold_pct: alert(severity="warning", tenant=tenant, baseline=baseline, current=current, drift_pct=round(drift_pct, 1)) return drift_pct

A few practical rules make this reliable:

• Track per tenant. An average hides the one tenant whose data shifted.
• Re-run after every change. Reindex, model swap, parameter edit and alias switch all count.
• Keep the exact baseline cheap. Brute-force search over a sample is enough. You do not need it over everything.
• Include a foreign-ID check. While the golden queries run, assert that no returned ID belongs to another tenant. That turns one job into a drift monitor and a leak monitor.

AWS publishes prescriptive guidance on detecting drift in production ML applications that is worth reading for alerting patterns beyond recall.

You can't watch drift directly. You watch a number that moves when it happens.

Layer 4: reconciliation for permission drift and ghost documents

A vector store is a copy. Copies go stale.

Reconciliation is a scheduled job that walks the index and checks each vector against the source of truth. It asks two questions: does this document still exist, and does this vector still carry the right permissions?

Follow these steps:

1. Export IDs and metadata. Pull vector IDs with their owner, ACL hash and source version in pages.
2. Fetch the source state. For the same IDs, read current existence and ACL from the system of record.
3. Compare. Flag missing documents (ghosts), changed ACLs (stale permissions) and changed versions (stale content).
4. Quarantine first. Mark flagged vectors as hidden from retrieval before you delete anything.
5. Delete or re-ingest. After review, remove ghosts and re-ingest changed documents.
6. Report the lag. Track how long each class of mismatch lived. That number is your real propagation delay.

The Redis engineering team describes these "ghost documents" as vector search returning a reference to something that no longer exists in the primary database. Add a content hash next to each embedding at ingest time, and a mismatch tells you immediately that the embedding is stale.

Dormant tenants add a twist. Weaviate's docs say that deleting expired objects only runs for active tenants and is skipped for inactive or offloaded ones until they become active again. Stale data can therefore outlive its deadline in a dormant tenant, so include dormant tenants in your reconciliation scope.

Better still, shrink the window. If your source of truth is Postgres, change-data-capture can push permission changes toward the index within seconds rather than waiting for the next sweep. Our guide to Postgres CDC in production covers the patterns. Reconciliation then becomes the safety net rather than the main mechanism.

Where does the final tenant decision belong? In a dedicated authorization engine, ideally, rather than scattered across filters. Our explainer on Zanzibar-style relationship-based access control lays out the model, and the comparison of OpenFGA, SpiceDB and Zanzibar for RAG authorization helps you pick an engine.

Treat the vector index as a cache of your permissions, and verify it like one.

Layer 5: CI negative tests

Most tests check that things work. These check that forbidden things fail.

A negative test plays the attacker. Tenant A asks for tenant B's data, through every door you have built, and the test passes only if nothing comes back. Add the tests to your build, and a regression like the hybrid-search gap fails the pipeline before it ships.

Seed a canary document that exists only in tenant B, and search for it as tenant A.

Python Code: pytest negative tests for tenant isolation:

import pytest  QUERY_PATHS = ["search", "hybrid_search", "batch_search"] CANARY = "CANARY-TENANT-B-7f3a" # seeded only into tenant_b's data   @pytest.mark.parametrize("path", QUERY_PATHS) def test_tenant_a_never_sees_tenant_b(path, vector_client, seeded_index): hits = vector_client.query(path=path, as_tenant="tenant_a", text=CANARY) assert all(h.tenant_id == "tenant_a" for h in hits) assert not any(CANARY in h.text for h in hits)   @pytest.mark.parametrize("path", QUERY_PATHS) def test_unscoped_query_is_rejected(path, vector_client): with pytest.raises(Exception): vector_client.query(path=path, as_tenant=None, text="anything")   def test_app_role_cannot_bypass_rls(pg_admin): role = pg_admin.fetchone( "SELECT rolsuper, rolbypassrls FROM pg_roles WHERE rolname = 'app_user'" ) assert role == (False, False)  forced = pg_admin.fetchone( "SELECT relforcerowsecurity FROM pg_class WHERE relname = 'doc_chunks'" ) assert forced == (True,)

Your matrix should cover every combination of:

• Query path: single search, hybrid, batch, "similar to this," export and admin tools
• Role: end user, service account, support engineer, admin
• Tenant state: active, inactive, offloaded, newly created, being deleted
• Filter shape: none, wrong tenant, several tenants, malformed, OR conditions

That grid is big, so generate it rather than hand-write it. A r/PostgreSQL post about running 24 checks in every CI pipeline describes the same approach for Postgres: a set of automated checks that fail the build whenever multi-tenant data could leak between tenants.

Two catalog queries make the pgvector checks concrete. Run them against a migrated test database in CI. Any row from the first query is a finding. The second should list only the roles you expect, usually your administrator role.

SQL Code: find tenant tables and roles that skip row-level security:

-- 1) Tables that carry tenant_id but lack forced row-level security SELECT c.relname AS table_name, c.relrowsecurity AS rls_enabled, c.relforcerowsecurity AS rls_forced FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace JOIN pg_attribute a ON a.attrelid = c.oid WHERE c.relkind = 'r' AND n.nspname = 'public' AND a.attname = 'tenant_id' AND NOT a.attisdropped AND NOT (c.relrowsecurity AND c.relforcerowsecurity) ORDER BY c.relname;  -- 2) Roles that skip row-level security entirely SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolsuper OR rolbypassrls ORDER BY rolname;

We ran both queries on a PostgreSQL 16.15 sandbox with one table forced and one merely enabled. The first query listed only the merely-enabled table. The table owner could read every row of that table without a tenant setting, a non-owner role read none, and on the forced table the owner was subject to the policy. That matches what the PostgreSQL documentation says. It is one small observation on one version, and it proves nothing about your schema.

For pgvector, the role check in the test above matters most. The Supabase guide to RAG with permissions shows how RLS fits a retrieval setup. The AWS Database Blog on multi-tenant isolation with PostgreSQL RLS is a good second reference.

If no test can fail because of a cross-tenant result, nothing is guarding the boundary.

Layer 6: forensic replay when a customer says "I saw something"

Eventually, someone reports it. Your response in the first hour decides how the next month goes.

Forensic replay means you can re-run a past query against the state of the world at that moment. That needs three things stored together: query logs, an index snapshot or version, and the ACL state at that time.

Here is the playbook:

1. Freeze the evidence. Preserve logs, index versions and ACL history before any cleanup or reindex.
2. Pin the incident. Get the tenant, approximate time, and what the user saw.
3. Find the queries. Pull Layer 1 entries for that tenant and window.
4. Check the IDs. Compare returned IDs with their owners at that time.
5. Replay. Re-run the suspect queries against the matching snapshot to confirm and reproduce.
6. Scope the blast radius. Search the whole log for the same filter pattern, code path or cache key, and list every affected tenant.
7. Fix, then verify. Patch, add a CI test that would have caught it, and replay again to prove the fix.

OpenAI's own response followed this shape: it fixed the root cause, programmatically reviewed its logs, correlated several data sources to identify affected users, and improved logging to confirm the problem had stopped.

If agents drive your retrieval, forensics gets harder, because the "who asked" chain is longer. Reliable harness design helps here, and our piece on agent harness engineering covers how to keep that chain traceable.

You can't investigate what you didn't record, and you can't replay what you didn't version.

What this architecture does not catch

Detection is not proof of isolation. A clean week of logs shows that nothing tripped your checks. It does not show that nothing leaked. This table is our own analysis of where each layer goes blind.

Threat model assumptions. This design assumes an honest but fallible application. It does not address a malicious insider with database administrator rights, stolen credentials, prompt injection, or embedding inversion. Those need their own controls.

A caution about the logs themselves. Audit entries hold tenant IDs, filter strings and result IDs. Those can be personal data or can reveal business activity. Limit who can read them, tie retention to your investigation window and your legal duties, and avoid storing document content. Ask your privacy counsel what applies to you.

Prove the detectors work: a fault-injection protocol

A detector you have never seen fire is a guess. Run this protocol in a staging environment with synthetic tenants only. Never inject faults into production or into real tenant data. This is a protocol, not a set of results: we have not run it against your system, and we report no timings.

For each fault, record four things: whether a detector fired, the time from injection to alert, whether the alert named the tenant, path and IDs well enough to act, and how many false alarms a clean control week produced. Fix any gap, then re-run the protocol after every major index or infrastructure change. Set time-to-detect targets with your risk owner. We offer none.

Before and after: what changes once detection is in place

Here's what a team feels in practice. The architecture does not change how fast you ship. It changes what you learn, and when.

The last row is easy to overlook. Enterprise buyers don't just want isolation. They want evidence of it. Logs, tests and alerts are that evidence.

Detection turns "we believe it's isolated" into "we can show it was."

Common mistakes (and how to avoid them)

These are the errors that keep showing up in postmortems and practitioner threads.

Treating the vendor flag as the control

A setting like partition key isolation or a namespace is a starting point. Pair it with a test that fails when it stops holding.

Testing only the main search path

The leak in Milvus issue #50398 lived in the second API, not the first. List every way your system can query vectors, then test each one.

Trusting a tenant ID from the request

Derive the tenant from the authenticated session or token. Treat any client-supplied value as a hint at best.

Caching without the tenant in the key

Any response, retrieval or semantic cache needs the tenant, and ideally the policy version, in its key.

Logging only control-plane events

Knowing who created an index doesn't tell you who queried what. Log data-plane queries yourself.

Watching only latency and errors

Both stay flat during a leak and during drift. Add recall and foreign-ID checks to your dashboard.

Connecting as a privileged database role

For pgvector, an app that connects as the table owner or a superuser walks straight past RLS. Check roles in CI.

Reindexing without a safety net

Before you swap aliases, run golden queries against the new index and keep the old one until they pass.

Three expert moves most teams skip

1. Run a canary tenant

Create a fake tenant with synthetic documents that contain unique, meaningless tokens, like CANARY-7f3a-ORCHID. No real user should ever retrieve them. Then, on a schedule, query as other tenants using text that would match those documents.

Scan three places for the tokens: the search results, your response cache and your LLM outputs. A hit anywhere is strong evidence of a boundary failure, and it should be treated as an incident until you prove otherwise. You found it instead of a customer. It's the cheapest tripwire you can build, and it can fire whichever layer failed, as long as the leaked content includes a canary token.

Python Code: scan results, caches and model output for canary tokens:

import re  # Tokens exist ONLY in the canary tenant's synthetic documents. CANARY_PATTERN = re.compile(r"CANARY-[0-9a-f]{4}-[A-Z]+")   def scan_for_canaries(source, tenant, texts, alert): """ source: where the texts came from: "search_results", "cache" or "llm_output" tenant: the tenant the request was served for texts: strings to inspect (chunk text, cached answers, model replies) """ hits = set() for text in texts: hits.update(CANARY_PATTERN.findall(text)) if hits and tenant != "canary_tenant": alert( severity="critical", source=source, tenant=tenant, tokens=sorted(hits), ) return sorted(hits)

This is illustrative code. Wire it into the places where retrieval results, cache reads and model replies already pass through your service.

2. Measure isolation coverage like code coverage

Count your query paths, roles, tenant states and filter shapes. That product is your isolation surface. Track what share of it has a negative test.

A team with 5 paths, 4 roles, 5 states and 5 filter shapes has 500 combinations. If 40 are tested, coverage is 8%. A new hybrid API adds a whole new row. Watching that percentage makes gaps visible before attackers or bugs find them.

3. Make cache keys policy-aware

Include the tenant, the embedding model version and a policy version in every cache key. When permissions change, bump the policy version and old entries stop matching. That one habit closes two failure modes at once: cross-tenant cache hits and stale answers after an ACL change.

How this connects to the wider security picture

Detection for vector stores sits inside bigger frameworks, and linking them helps with audits.

• OWASP. LLM08:2025 covers cross-context leaks, embedding inversion and poisoning in vector stores. This architecture gives you the runtime evidence that mitigations are working.
• NIST. The NIST AI Risk Management Framework organizes AI risk work into govern, map, measure and manage. Golden queries, audit logs and canary hits are concrete "measure" artifacts.
• Privacy law. Erasure rights reach embeddings, as noted earlier, so reconciliation doubles as compliance evidence.
• Agents. As more systems let agents call retrieval on a user's behalf, identity and traceability matter more. See securing non-human identities and agent harness engineering.
• Authorization. Moving tenant decisions into a policy engine, as discussed in the OpenFGA vs SpiceDB vs Zanzibar comparison, reduces the number of places a filter can be forgotten.

A 30-day rollout plan

You don't need all six layers at once. Order them by how much risk each removes per hour of work.

Then check where you stand today.

Detection turns isolation claims into evidence

Namespaces, shards, partition keys and RLS policies are good tools. None of them is evidence on its own.

Evidence looks like a result validator with no foreign hits, a canary token that has not surfaced, a CI matrix that goes red when someone adds a path, and a log you can replay when a customer asks. Even together they do not prove isolation, as the limits table above explains. What they change is who learns about a leak first: your own on-call engineer instead of your customer.

Start with the 30-day plan above, and score yourself with the readiness check. If you haven't yet closed the prevention side, our vector database security hardening guide is the natural companion to this one.

Then ask your team one question on Monday: if a query path dropped its tenant filter today, how would we find out? If the answer is "a customer would tell us," you know where to begin.

Frequently asked questions

About The Author

A Gadallh

Ahmed Gadallah is the Founder and Editor of Vertex Frontier, where he publishes research-driven articles on AI, data science, cloud computing, cybersecurity, software engineering, and emerging technologies, with a focus on technical accuracy, clarity, and practical insights.

View all articles by A Gadallh →