Access control belongs in the index. Not the model, not a post-filter.

You have a shared memory store for your agents, and some of what it holds is not for everyone. Customer A's pricing, an HR record, a deal under NDA. When an agent runs a recall, something has to decide which memories the caller is allowed to see. There are three places you can put that decision, and two of them break in ways you won't notice until it's too late.

The three placements

Put a permission rule anywhere in a retrieval pipeline and it lands in one of three spots. You can write it into the model as an instruction. You can bolt it on after retrieval as a filter. Or you can push it down into the index so the search itself never considers a memory the caller can't see.

The essay this post follows already argued the model is the wrong place. A system prompt that says never reveal docs the user can't see is a request, not a guarantee. The model can be jailbroken, it can be confused by an injected string in a retrieved chunk, and it has no reliable way to know the caller's real ACL at inference time. Assume that's settled. The interesting fight is between the other two.

Post-filter looks obviously fine. You run the vector search, take your top-k by similarity, then walk the results and drop anything the caller isn't cleared for. The grant is correct: nothing forbidden reaches the caller. Engineers reach for this first because it's a five-line function that's easy to reason about. That's the trap.

Why post-filter quietly wrecks recall

Here is the failure. You ask for the 10 most relevant memories. Similarity returns 10. The permission filter drops 6 of them because the caller can't see them. You hand back 4.

You did not ask for the 4 most relevant visible memories. You asked for 10, and the layer that enforced permissions silently converted your k=10 into k=4. Worse, you can't tell from the outside whether the 6 that got dropped were the ones you actually needed. The ranker ranked over a set that included forbidden documents, so the visible results you got are the leftovers from a competition they lost.

This gets sharper the more restricted the caller is. A junior support agent scoped to one customer might have 90% of any top-k dropped out from under them. Their recall collapses and nothing in the response says so. The similarity scores look healthy. The count is just low, and a low count is easy to explain away as a thin corpus.

You can paper over it by over-fetching: ask for top-200 so that after filtering you still have 10. Now you're paying to rank 200 candidates to keep 10, the over-fetch multiplier depends on the caller's selectivity so you can't set it once, and a caller restrictive enough will still starve. Over-fetching turns a correctness bug into a cost-and-tuning problem you never fully close.

You asked for 10, and the layer that enforced permissions silently converted your k=10 into k=4.

The index fixes both problems at once

Move the permission decision in front of the ranker. Compile the caller's scope, their identity, group memberships, and the ACL tokens they carry, into the retrieval query as a mandatory pre-filter. The search only ever ranks memories the caller can see. The k you get back are the top-k among visible memories, because visible was the only pool that ever entered the ranking.

That one move fixes both things. The guarantee is now structural: a forbidden memory can't reach the caller because it was never a candidate, not because a downstream filter happened to catch it. And recall is honest again, because k=10 means the 10 best of what you're allowed to see, not the survivors of a race that included documents you'll never receive.

This is the shape Verity ships. Scope is materialized into the index and applied as a mandatory pre-filter before ranking, with no live ReBAC-engine call on the read path. Enforcement lives in one shared layer above the storage adapter, so every connector, Drive, Gmail, Salesforce, and the rest, inherits the same fail-closed check instead of re-implementing its own. No visibility token means invisible. An unresolvable subject means an empty result, never a permissive fallback.

A forbidden memory can't reach the caller because it was never a candidate, not because a downstream filter happened to catch it.

The cost you actually pay

A pre-filtered approximate-nearest-neighbor search is not free, and pretending otherwise would make this post the exact thing it argues against. Constraining the search to a subset before ranking degrades ANN performance relative to an unfiltered search. The index graph was built over everything, so forcing traversal through only the visible nodes can mean touching more of the structure to fill k.

How much it costs depends on the corpus size and how selective the caller's scope is. Verity's measured curve makes this concrete. Dense and hybrid recall come in under 50ms p95 warm at around 100k chunks with a selective filter. Push to a much larger corpus and it climbs: at 1M chunks, filtered-ANN measured roughly 75ms p95 at a 0.1% filter and rose toward 1.2s p95 at a 50% filter under memory contention, where the working set stops fitting in page cache and the search goes disk-bound. Broad scope plus huge corpus is the worst case, and the honest move is to name it.

There's also a case where a live permission check would have been fine. A tiny, hyper-restricted subject class, a caller who can see only a handful of records, is cheap to resolve on the fly. The index pre-filter is the right default across the range of real callers, not a law every workload must obey. Point reads and BM25 stay fast even at 1M chunks. It's specifically dense and hybrid recall over a broad scope on a very large corpus where you feel the bill.

The honest part

The index pre-filter is not a free lunch, and the failure modes are specific enough to state plainly before I claim it wins.

Pre-filtered ANN recall degrades relative to an unfiltered search, because constraining the candidate set before ranking makes the index work harder to fill k. A broad scope over a very large corpus can push the working set past page cache and go disk-bound, at which point latency climbs well past the fast-path numbers.

The sub-50ms figure holds at a stated corpus size and selectivity, around 100k chunks with a selective filter, and not at every scale. Verity publishes the whole curve rather than a single p95 for that reason. And a tiny, hyper-restricted subject class can be cheap enough that a live permission check would have served fine.

So this is the right default, not a universal win. Anyone who tells you their pre-filtered vector search has no cost is selling you the model-in-a-system-prompt version of a benchmark.

Run the filtered-ANN benchmark on your own hardware and see where it breaks: github.com/RunAlphaLoop/verity