RAG Retrieval Tests for Permission-Filtered Vector Search

A support engineer asks an internal assistant how to restore a replica. The assistant cites one general runbook. A platform administrator asks the same question and gets three detailed procedures. Both answers look plausible. Before anyone changes the prompt or grants the engineer broader access, the search team needs to answer a smaller question. Which chunks, if any, should the engineer’s search have returned?

To test the vector retriever in a retrieval-augmented generation (RAG) system, build a reference result by scoring every eligible chunk for the same query and identity, then taking the nearest k. Compare the production result at the point where evidence leaves the trusted retrieval service. Count forbidden chunks, missing slots and missed neighbors separately. A retriever can fill every slot while missing the closest eligible procedure. It can also return one hit and be exactly right.

Where a good chunk falls out

Suppose three chunks fit in the prompt and the search service fetches four candidates before applying a document access list. If three candidates belong to another team, the service returns one. The fifth and sixth candidates might have been excellent matches that the engineer could read. Ranking the whole corpus perfectly would still produce this failure.

A search engine may evaluate a filter during its search, search a small eligible subset exhaustively, or apply a filter to a bounded candidate list. Inspect the execution plan. OpenSearch documents the different behaviors, including the possibility of fewer than k hits after post-filtering. The pgvector project describes how iterative scans can continue looking after candidates are filtered, subject to scan limits. OpenSearch filtering options and pgvector documentation.

Some filtering can legitimately occur inside a service trusted to inspect the whole index. The same design discloses data if it sends forbidden text to an unauthorized external reranker before trimming the final list. Define the boundary at every consumer of candidate data, including debug traces. Run the eligibility check against the inputs each consumer actually receives. The application’s source of truth for those decisions belongs in its RAG governance design.

Build a reference independent of the index filter

Freeze a corpus revision, a query vector, a distance function and a policy snapshot. Ask the source authorization system which document revisions this principal may read. Expand that answer into chunk IDs using the ingestion manifest. Include tenant, group membership, explicit denies, validity dates and any chunk-level restrictions that affect this request. In a test environment, an independently reviewed fixture can supply that answer. Copying the index’s permission labels into the reference lets a broken label pass both sides of the test.

First reconcile the manifest with the vectors actually present. If the newest runbook never made it into the index, record an ingestion failure. Dropping the missing chunk from both sides can make a broken index score perfectly. Once vectors and revisions agree, score the eligible set exhaustively with the same metric and normalization as the candidate system. Break ties consistently for a small fixture. With quantized production vectors, specify whether the reference uses original or quantized values, and decide in advance how equal distances at the cutoff will be credited.

Let allowed be the independently eligible chunk IDs and T the exact nearest min(k, len(allowed)) IDs. For a first-stage vector retriever, inspect up to k results after mandatory access checks and before optional reranking or relevance thresholds. Compare chunks on both sides. Comparing distinct documents to a chunk-level reference introduces an apparent recall loss whenever two chunks share a parent.

Flag duplicate IDs separately. Duplicating a chunk should never earn a second slot of credit. If allowed is empty, the correct response contains no evidence. Keep that case as an explicit test and leave both ratios undefined. An average padded with perfect scores for empty tenants can hide a regression in the tenants that have documents.

An eight-chunk reproduction

This Python 3 fixture uses one-dimensional vectors, a query at zero and absolute distance. The letters identify synthetic chunks in increasing distance order. The support engineer can read B, E, F and H. A contractor can read only H. The broad-access identity demonstrates how an administrator’s smoke test can pass while an ordinary user’s search fails.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

vectors = dict(A=1, B=2, C=3, D=4, E=5, F=6, G=7, H=8)
query, k = 0, 3
support = {"B", "E", "F", "H"}

def nearest(ids):
    return sorted(ids, key=lambda i: (abs(vectors[i] - query), i))

def postfilter(allowed, depth=4):
    candidates = nearest(vectors)[:depth]
    return [i for i in candidates if i in allowed][:k]

def inspect(allowed, returned):
    assert len(returned) <= k
    target = set(nearest(allowed)[:k])
    hits = set(returned)
    forbidden = hits - allowed
    duplicates = len(returned) - len(hits)
    fill = len(hits & allowed) / len(target) if target else None
    recall = len(hits & target) / len(target) if target else None
    return forbidden, duplicates, fill, recall

assert postfilter(support) == ["B"]
assert inspect(support, ["B"]) == (set(), 0, 1/3, 1/3)
assert inspect(support, ["B", "F", "H"]) == (set(), 0, 1, 2/3)
assert inspect(support, ["B", "B", "E"]) == (set(), 1, 2/3, 2/3)
assert inspect(support, ["A", "B", "E"])[0] == {"A"}
assert inspect(set(vectors), postfilter(set(vectors)))[3] == 1
assert inspect({"H"}, postfilter({"H"}))[2:] == (0, 0)
assert inspect({"H"}, ["H"])[2:] == (1, 1)
assert inspect(set(), [])[2:] == (None, None)
assert postfilter(support, depth=6) == ["B", "E", "F"]

At depth four, B is the sole eligible candidate for support. At depth six, E and F enter the window. For the contractor, even depth six is insufficient. A fixed multiplier on k can buy headroom, but its success on one identity says little about another. The ordering here is exact by construction. These assertions isolate candidate truncation and scoring logic. To measure approximate nearest-neighbor performance, replay the requests on the deployed engine and time them there.

The deliberately substituted result B, F, H is worth keeping in a regression suite. It fills all three slots, discloses nothing and still loses E. A monitor that only counts empty or short responses would never notice. Conversely, returning H alone to the contractor earns full credit because there is exactly one eligible chunk.

Turn the discrepancy into a repair decision

Use the same query text and vector under several principals. Include a broadly entitled user, a narrow group, an empty tenant and a freshly revoked membership. Pair old and candidate indexes on those requests. For a revocation test, use the new entitlement state as the reference and check every consumer, including warm caches. A frozen replay proves little about how fast a running service enforces an access change. That timing has its own revocation and invalidation tests.

Keep the comparisons inside the authorized evaluation service. Showing users how many other tenants’ chunks were filtered would itself reveal information. Record the principal class, policy and corpus revisions, embedding version, requested k, effective search parameters, eligible population, returned IDs and search latency in the restricted evaluation environment. Publish aggregates by eligible fraction and absolute eligible count. A one-percent filter over a hundred chunks is a different execution problem from a one-percent filter over ten million. Keep empty cases, duplicates, disclosure failures and timeouts visible alongside the recall distribution. Excluding failed requests makes a slow exact-search fallback look better than it is.

For cutover, agree on a latency budget and the minimum acceptable retrieval quality for each material access cohort. Any observed forbidden result blocks that route. For quality, compare paired misses on questions where the team knows the evidence required for the task. An approximate result can miss a mathematical neighbor and still contain the correct procedure. An exact result can faithfully reproduce a poor embedding. The workflow release gate should retain both the vector comparison and the downstream evidence judgment.

Back at the support desk, ask for one diagnostic record before widening the search. If E was forbidden for this engineer at that revision, respect the decision. If E was allowed, indexed and closer than what came back, the search team now has a failing test it can reproduce without changing anyone’s permissions.