TL;DR — Scaling RAG is rarely about more documents; it is about more audiences — tenants, departments, verticals, each with its own corpus and rules. Two decisions decide whether that stays cheap. Index layout: a shared index with a metadata filter is the obvious choice and has a silent failure — post-filter starvation. The ANN search fetches
Kcandidates globally, the filter keeps only that tenant's share, and a small tenant gets back fewer than you asked for. A 50% tenant is safe atK=50; a 2% tenant still starves 99.7% of the time at K=200 and needs K≈990 for a 1% failure rate — 99× the wasted work. So shared-index cost is set by your smallest tenant, not your average one. Index-per-tenant fixes that and pays a minimum provisioned size per index — 2.4× the raw data at any tenant count in the model below — but cuts blast radius from 100% to 1%. Onboarding: make a vertical a directory of config, not a branch in code. That is what keeps vertical twenty as cheap as vertical two, because the code path never changes.
1. Simple explanation
"Scaling" a RAG system usually gets read as "more documents." That is the easy axis — vector databases handle it, and the arithmetic is in Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For.
The axis that actually hurts is more audiences. The second department wants the same assistant over their own documents, with their own filters and their own vocabulary. Then a third. Then a customer wants their data isolated from every other customer's.
Now you face two questions that have no default answer. Does everyone share one index with a tenant_id filter, or does each get their own? And when a new vertical arrives, is that a code change or a config change?
Both get answered badly by default. The shared index is the obvious first choice and it degrades in a way that is genuinely hard to see. And the first two verticals are always added with an if statement, which is fine — until vertical eight, when the routing logic is a thicket nobody wants to touch.
Analogy — a shared warehouse versus separate stockrooms. One warehouse for all your clients is efficient: one building, one heating bill, one team. To fill a client's order you walk the aisles and pick out anything with their label. That works well when each client owns a big share of the stock. When a client owns 2% of it, you walk almost the whole warehouse to find ten of their items — and if you decide to only walk the first few aisles, you come back with three items and no indication that seven are missing. Separate stockrooms fix the walking problem and mean you now heat fifty small buildings, most of them mostly empty.
2. Diagram
SHARED INDEX + FILTER INDEX PER TENANT
───────────────────── ────────────────
┌───────────────────┐ ┌────┐ ┌────┐ ┌────┐
│ all tenants │ │ T1 │ │ T2 │ │ T3 │ ...
│ ~~~~~~~~~~~~~~~ │ └────┘ └────┘ └────┘
└─────────┬─────────┘ │
│ fetch K globally │ fetch k directly
▼ ▼
filter to tenant no filter needed
│ │
▼ ▼
maybe < k results! exactly k results
(silently)
POST-FILTER STARVATION — P(fewer than 10 survive), §5
tenant share K=10 K=50 K=200 K=1000
50.0% 99.8% 0.0% 0.0% 0.0%
10.0% 100.0% 98.3% 0.7% 0.0%
2.0% 100.0% 100.0% 99.7% 0.9%
0.5% 100.0% 100.0% 100.0% 97.8%
▲▲▲▲▲
a 2% tenant is still starving at K=200
FETCH DEPTH FOR <1% STARVATION COST OF ISOLATION
50.0% -> 40 (4x waste) tenants shared per-tenant
10.0% -> 200 (20x) 1 2.0G 2.0G 1.0x
2.0% -> 990 (99x) 10 8.2G 20.0G 2.4x
0.5% -> 3,970 (397x) 100 81.9G 200.0G 2.4x
1,000 819.2G 2000.0G 2.4x
BLAST RADIUS OF ONE BAD INGEST
one shared index 100% of traffic
index per tier (4) 25%
index per tenant (100) 1%
3. How it works
3.1 Post-filter starvation
Approximate nearest-neighbour search does not know about your filter. The common implementation fetches the global top K by vector similarity and then discards rows failing the metadata predicate. If a tenant owns fraction f of the corpus, roughly K·f survive.
Want 10 results for a tenant holding 2% of the corpus? K=200 leaves you about 4 on average — and, because it is a random draw, fewer than 10 99.7% of the time.
What makes this dangerous is not the shortfall; it is that nothing reports it. The query succeeds. The API returns a list. It is just shorter than you asked for, and the generator answers from whatever arrived. Downstream, this looks exactly like a corpus that lacks the answer.
Three mitigations, in increasing order of how much they actually fix:
OVER-FETCH set K from your SMALLEST tenant's share. Correct, and
the wasted work is brutal: 99x for a 2% tenant.
PRE-FILTERED ANN the index restricts the search to matching rows BEFORE
traversing. Some engines support it; performance varies
a lot with filter selectivity. Check what yours does --
"supports filtering" is not the same as "pre-filters".
PARTITION separate index (or namespace) per tenant. The filter
becomes routing, and the problem disappears.
Whatever you choose, alert when a filtered query returns fewer than k results. It is a one-line check and it converts a silent failure into a visible one — the same argument as every other signal in RAG Monitoring: Your Error Rate Will Not Tell You Anything.
3.2 The shared-vs-partitioned trade
SHARED + FILTER INDEX PER TENANT
storage one copy, no minimums minimum size x N tenants
small tenants starve unless K is huge fine
cross-tenant search natural needs fan-out and merge
noisy neighbour one tenant's load hurts isolated
everyone
blast radius 100% 1/N
ops one thing to manage N things to manage
per-tenant config (key, region, retention) awkward natural
Measured: at 200,000 vectors per tenant with a 2 GB minimum per index, per-tenant layout costs 2.4× the shared layout from 10 tenants upward. That multiplier is driven entirely by the minimum size — the overhead is worst with many small tenants, which is exactly the case where a shared index is also most likely to starve them. The two designs are uncomfortable in the same place.
A middle option resolves most real cases: partition by tier, not by tenant. Group tenants into a handful of indexes — by region, by plan, by data-residency requirement — so you get isolation where it is contractually or operationally required and share everywhere else. Blast radius drops to 25% with four partitions rather than 1% with a hundred, at a fraction of the overhead.
The decision is usually forced by something other than performance: data residency, contractual isolation, per-tenant encryption keys, or a customer's right to have their data deleted on demand. If any of those apply, partition and stop optimising.
3.3 Make a vertical a config pack
The second question — how a new vertical gets added — determines whether this stays maintainable.
The path of least resistance is a branch in the router, then a subclass, then a registry of subclasses. Each is reasonable in isolation and together they mean every new vertical is a code change, a review, and a deploy — and, worse, that verticals can quietly diverge in behaviour because each has its own code path.
The alternative is to make a vertical a directory of configuration that the same code loads at startup:
domains/<vertical>/
domain.yaml label, which index, which credential, enabled flag
routing.yaml specialists/sub-retrievers eligible, routing hints
corpus.yaml sources, scoring weights, ingestion filters
tools.yaml which domain tools are available
prompts/ any prompt overrides
Adding a vertical becomes: create the directory, fill the files, restart or hot-reload. No Python changes. The code path is identical for every vertical, so a fix to retrieval fixes it everywhere at once, and behaviour cannot silently diverge.
Three properties make this work in practice:
- Discovery at startup, not a hardcoded list — otherwise you have reintroduced the registry.
- Validate on load and fail loudly. A typo in a config pack should stop the service at boot with a clear message, not surface as a mysterious empty result at 3am. This is the main risk of config-over-code: you have traded compile-time errors for runtime ones, and schema validation is how you get them back.
- Version the packs with the code. They are behaviour, so they belong in review and in git — the same argument as Prompt Management Architecture: Prompts as Files, Not Strings.
What stays in code is anything that is genuinely logic: a new retrieval algorithm, a new tool implementation. Configuration selects and parameterises; it should not grow into a programming language. When a config pack starts needing conditionals, that is the signal you have pushed too far.
3.4 The things that break at N tenants
RATE LIMITS shared provider quota, so one tenant's burst throttles
everyone. Per-tenant concurrency caps, not just global.
CACHES key on (tenant, query) or you will serve one tenant's
answer to another. This is a data-leak bug, not a
performance bug.
EVALUATION one aggregate eval set hides per-tenant regressions.
Segment it, as with monitoring.
MIGRATIONS re-embedding N indexes is N migrations. Stagger them,
and expect mixed model versions during the window.
COLD TENANTS a tenant with 50 queries/day cannot support alerting;
fall back to scheduled evaluation.
The cache-key one deserves emphasis: it is the highest-severity bug in this list and the easiest to write. A cache keyed on the query string alone will serve tenant A's answer to tenant B, and no test that runs as a single tenant will catch it.
3.5 Where this is overkill
For one audience, none of this applies — one index, no filter, no packs. Introduce the config-pack structure when you add the second vertical, because that is when the cost of retrofitting is still near zero and the temptation to write the first if is highest. Partitioning is worth deferring until you have a concrete reason: a starving small tenant, a residency requirement, or an incident whose blast radius was unacceptable.
4. The math
4.1 Starvation
Surviving results after post-filtering are binomial: S ~ Binomial(K, f) for tenant share f and global fetch depth K. You want P(S < k) small:
E[S] = K·f so a naive K = k/f gives you k on AVERAGE
-- which means starving roughly half the time
That is the trap: sizing K = k/f feels right and fails 50% of the time. You need enough margin above the mean to cover the variance:
K such that P(S < 10) < 1%:
f = 50.0% -> K = 40 4x the results you want
f = 10.0% -> K = 200 20x
f = 2.0% -> K = 990 99x
f = 0.5% -> K = 3,970 397x
Roughly K ≈ k/f + z·sqrt(k/f), so the required depth grows a little faster than 1/f. Your smallest tenant sets K for everybody, because the fetch depth is a property of the query path, not of the tenant. Serving one 0.5% tenant means 397× the work on every query unless you special-case it.
4.2 The cost of isolation
With n tenants, v vectors each, b bytes per vector, and a minimum provisioned size M per index:
shared = max(M, n·v·b)
per-tenant = max(M·n, n·v·b)
overhead = per-tenant / shared = max(1, M / (v·b))
The overhead is independent of n — it is set entirely by how a tenant's data size compares with the minimum index size. In the model: v·b = 0.82 GB against M = 2 GB, giving 2.4× at every tenant count above one.
The practical reading: if your typical tenant's data comfortably exceeds the minimum index size, per-tenant indexes are nearly free. If tenants are small relative to that minimum, you are paying for empty space, and tier-based partitioning is the compromise.
4.3 Blast radius
fraction of traffic affected by one bad ingest ≈ 1 / (number of partitions)
1 partition -> 100%
4 tiers -> 25%
100 tenants -> 1%
Combined with §4.2's constant overhead, this is the actual decision: a fixed multiple of storage buys a linear reduction in blast radius. Four partitions remove three-quarters of the exposure at a small fraction of the cost of a hundred.
5. Real code
Standard library only, exact arithmetic, runs instantly.
import math
from statistics import NormalDist
_ND = NormalDist()
def starvation(tenant_share, fetch_k, need_k=10):
"""Post-filtering: fetch K globally, keep the tenant's rows, want need_k.
Survivors ~ Binomial(fetch_k, tenant_share). Normal approximation with a
continuity correction; exact enough for the shape and far cheaper than
enumerating the tail.
"""
mean = fetch_k * tenant_share
var = fetch_k * tenant_share * (1 - tenant_share)
if var <= 0:
return (0.0 if mean >= need_k else 1.0), mean
z = (need_k - 0.5 - mean) / math.sqrt(var)
return _ND.cdf(z), mean
def part_a():
print("A. POST-FILTER STARVATION (want 10 results for one tenant)")
shares = (0.50, 0.10, 0.02, 0.005)
fetches = (10, 50, 200, 1000)
print(f"{'tenant share':>14} " + "".join(f"{'K=' + str(k):>12}"
for k in fetches))
print("-" * 64)
tab = {}
for s in shares:
cells = []
for K in fetches:
p, mean = starvation(s, K)
tab[(s, K)] = p
cells.append(f"{p:11.1%}")
print(f"{s:14.1%} " + "".join(cells))
print("\n cell = P(fewer than 10 results survive the filter)")
print(f" a 50% tenant is safe by K=50 ({tab[(0.50, 50)]:.1%}); a 2% tenant"
f" still starves {tab[(0.02, 200)]:.1%} of the time at K=200.")
print(" the smaller the tenant, the deeper the global fetch must go --")
print(" so shared-index cost scales with your SMALLEST tenant.")
print("\n fetch depth needed for <1% starvation:")
print(f"{'tenant share':>14} {'required K':>12} {'wasted work':>13}")
print("-" * 42)
need = {}
for s in shares:
K = next((k for k in range(10, 20001, 10)
if starvation(s, k)[0] < 0.01), None)
need[s] = K
print(f"{s:14.1%} {K:12,} {K / 10:12.0f}x")
return tab, need
def part_b():
print("\n\nB. WHAT INDEX-PER-TENANT COSTS")
VEC_PER_TENANT = 200_000
BYTES_PER_VEC = 1024 * 4
MIN_INDEX_GB = 2.0 # smallest billable/provisioned unit per index
print(f" {VEC_PER_TENANT:,} vectors/tenant, 1024-d float32, "
f"{MIN_INDEX_GB}GB minimum per index")
print(f"{'tenants':>9} {'data GB':>9} {'shared GB':>11} "
f"{'per-tenant GB':>15} {'overhead':>10}")
print("-" * 60)
rows = {}
for n in (1, 10, 100, 1000):
data = n * VEC_PER_TENANT * BYTES_PER_VEC / 1e9
shared = max(MIN_INDEX_GB, data)
per = max(MIN_INDEX_GB * n, data)
rows[n] = (shared, per)
print(f"{n:9,} {data:8.1f}G {shared:10.1f}G {per:14.1f}G "
f"{per / shared:9.1f}x")
print("\n per-tenant indexes pay a MINIMUM SIZE for every tenant, so the")
print(" overhead is worst when you have many small tenants -- exactly the")
print(" case where a shared index also works best.")
return rows
def part_c():
print("\n\nC. BLAST RADIUS OF ONE BAD INGEST")
print(f"{'architecture':>28} {'tenants affected':>18} {'traffic affected':>18}")
print("-" * 68)
for name, frac in (("one shared index", 1.0),
("index per tenant (100)", 0.01),
("index per tier (4 tiers)", 0.25)):
print(f"{name:>28} {frac:17.0%} {frac:17.0%}")
print("\n isolation is bought with fixed overhead (part B) and paid back")
print(" in incidents you do not have.")
print("\n\nD. ONBOARDING A NEW VERTICAL")
print(f"{'approach':>26} {'to add one':>28} {'deploy?':>9} {'reviewable':>11}")
print("-" * 78)
for name, work, dep, rev in (
("branch in code", "edit router + add module", "yes", "diff"),
("subclass per domain", "new class + registration", "yes", "diff"),
("config pack (YAML)", "add a directory of files", "no", "diff")):
print(f"{name:>26} {work:>28} {dep:>9} {rev:>11}")
print("\n a config pack keeps the code path identical for every vertical,")
print(" which is what makes vertical number 20 as cheap as number 2.")
tab, need = part_a()
rows = part_b()
part_c()
# claims made in the prose
assert tab[(0.02, 50)] > 0.9, "a small tenant must starve at shallow K"
assert tab[(0.50, 50)] < 0.01, "a large tenant must be fine at the same K"
assert need[0.005] > need[0.50] * 10, \
"required depth must blow up for small tenants"
assert rows[1000][1] / rows[1000][0] > 2, \
"per-tenant overhead must dominate at many tenants"
assert rows[1][1] == rows[1][0], "with one tenant the designs coincide"
print("\nasserts passed")
# Output:
# A. POST-FILTER STARVATION (want 10 results for one tenant)
# tenant share K=10 K=50 K=200 K=1000
# ----------------------------------------------------------------
# 50.0% 99.8% 0.0% 0.0% 0.0%
# 10.0% 100.0% 98.3% 0.7% 0.0%
# 2.0% 100.0% 100.0% 99.7% 0.9%
# 0.5% 100.0% 100.0% 100.0% 97.8%
#
# cell = P(fewer than 10 results survive the filter)
# a 50% tenant is safe by K=50 (0.0%); a 2% tenant still starves 99.7% of the time at K=200.
# the smaller the tenant, the deeper the global fetch must go --
# so shared-index cost scales with your SMALLEST tenant.
#
# fetch depth needed for <1% starvation:
# tenant share required K wasted work
# ------------------------------------------
# 50.0% 40 4x
# 10.0% 200 20x
# 2.0% 990 99x
# 0.5% 3,970 397x
#
#
# B. WHAT INDEX-PER-TENANT COSTS
# 200,000 vectors/tenant, 1024-d float32, 2.0GB minimum per index
# tenants data GB shared GB per-tenant GB overhead
# ------------------------------------------------------------
# 1 0.8G 2.0G 2.0G 1.0x
# 10 8.2G 8.2G 20.0G 2.4x
# 100 81.9G 81.9G 200.0G 2.4x
# 1,000 819.2G 819.2G 2000.0G 2.4x
#
# per-tenant indexes pay a MINIMUM SIZE for every tenant, so the
# overhead is worst when you have many small tenants -- exactly the
# case where a shared index also works best.
#
#
# C. BLAST RADIUS OF ONE BAD INGEST
# architecture tenants affected traffic affected
# --------------------------------------------------------------------
# one shared index 100% 100%
# index per tenant (100) 1% 1%
# index per tier (4 tiers) 25% 25%
#
# isolation is bought with fixed overhead (part B) and paid back
# in incidents you do not have.
#
#
# D. ONBOARDING A NEW VERTICAL
# approach to add one deploy? reviewable
# ------------------------------------------------------------------------------
# branch in code edit router + add module yes diff
# subclass per domain new class + registration yes diff
# config pack (YAML) add a directory of files no diff
#
# a config pack keeps the code path identical for every vertical,
# which is what makes vertical number 20 as cheap as number 2.
#
# asserts passed
The K=200 row for a 2% tenant is the one worth pausing on. Two hundred candidates feels generous — it is 20× the ten results you asked for — and it fails 99.7% of the time. Intuition about fetch depth is calibrated on the majority tenant and is badly wrong for the tail.
6. Real-world example
A team ran a shared index across about forty client corpora with a client_id filter, K = 100, returning the top 10. It worked well for two years. Their three largest clients were most of the corpus and most of the traffic.
Small clients complained sporadically that answers were "thin" or "missed obvious documents." Each report was investigated as a relevance problem — embeddings, chunking, prompt — and each time the team found the documents were in the index and did match the query. So the reports were logged as subjective and closed.
The actual mechanism: a client holding 1% of the corpus got about one surviving row from a 100-candidate fetch. The API returned a list of length one. Nothing anywhere logged that ten had been requested.
They found it by accident, from a dashboard someone added for a different reason: results-returned-per-query, broken down by client. The large clients sat flat at 10. The small ones scattered between 0 and 4, and had done so for two years.
The fix was staged. Immediately: alert whenever a filtered query returns fewer than k. Then: raise K for small tenants specifically, rather than globally, so the majority did not pay for it. Then: move the twelve smallest clients to their own indexes, which removed the class of bug for the tenants most exposed to it.
The lesson is the one that recurs across RAG operations — the failure was a returned-length, not an error, and every layer of the stack reported success. What found it was a metric nobody had thought to plot, segmented by the dimension that mattered.
7. Interview questions companies actually ask
Q1 [easy] "Shared index with a tenant filter, or one index per tenant?"
A Shared is cheaper and simpler until small tenants starve: ANN fetches K
globally, the filter keeps ~K*f, so a 2% tenant still returns fewer than 10
results 99.7% of the time at K=200. Per-tenant removes that and pays a minimum
provisioned size per index -- 2.4x the raw data in the model here. If data
residency or contractual isolation applies, partition and stop optimising.
Q2 [easy] "What is post-filter starvation and why is it hard to spot?"
A The filter runs AFTER the ANN fetch, so you get roughly K*f survivors instead
of the k you asked for. It's hard to spot because nothing errors: the query
succeeds and returns a short list. Downstream it's indistinguishable from a
corpus that lacks the answer. Alert on returned-count < k.
Q3 [medium] "How do you size K for a shared index?"
A From your SMALLEST tenant, not the average -- fetch depth is a property of the
query path. K = k/f gives you k on average, which starves ~50% of the time, so
you need margin for variance: roughly k/f + z*sqrt(k/f). For 10 results: a 50%
tenant needs K=40, a 2% tenant K=990, a 0.5% tenant K=3,970. That's 397x the
work on every query.
Q4 [medium] "Does per-tenant indexing get more expensive as tenants grow?"
A No -- the overhead ratio is constant, max(1, M/(v*b)), set by how a tenant's
data compares with the minimum index size. It was 2.4x at 10, 100, and 1,000
tenants. So if a typical tenant comfortably exceeds the minimum index size,
per-tenant is nearly free; if tenants are small, you're paying for empty space.
Q5 [medium] "What's the middle ground?"
A Partition by tier rather than by tenant -- group by region, plan, or residency
requirement. Four partitions cut blast radius from 100% to 25% at a fraction of
the overhead of a hundred indexes, and you keep isolation where it's actually
required.
Q6 [medium] "How should a new vertical be added?"
A As a directory of config the existing code loads at startup -- index, routing,
corpus rules, tools, prompt overrides -- not as a branch or a subclass. The
code path stays identical for every vertical, so fixes apply everywhere and
behaviour can't silently diverge. Discover packs at startup, validate on load
and fail loudly at boot, and version them alongside the code.
Q7 [hard] "What's the risk of config-over-code?"
A You trade compile-time errors for runtime ones. A typo that would have been a
build failure becomes a mysterious empty result in production. Schema-validate
every pack on load and refuse to start on a bad one. Also watch for config
growing conditionals -- when a pack needs if-statements you've built a bad
programming language and the logic belongs in code.
Q8 [hard] "What breaks at N tenants that worked fine at one?"
A Caches keyed on the query string alone -- that serves tenant A's answer to
tenant B, a data leak no single-tenant test catches. Shared provider quota, so
one tenant's burst throttles everyone; you need per-tenant concurrency caps.
Aggregate eval sets that hide per-tenant regressions. And migrations become N
migrations, with mixed model versions during the window.
8. When to use / tradeoffs
CHOOSE THE LAYOUT BY THE BINDING CONSTRAINT:
data residency / contractual isolation / per-tenant keys -> PARTITION
a tenant under ~5% of the corpus -> PARTITION or
per-tenant K
many small tenants, no isolation requirement -> SHARED
unsure -> tier partitions
MAKE A VERTICAL A CONFIG PACK FROM THE SECOND ONE ONWARD:
discovered at startup - validated on load - versioned with the code
| Situation | Why it breaks | Use instead |
|---|---|---|
Shared index, K sized for the average tenant | Small tenants starve silently — 99.7% at 2% share | Size K from the smallest, or partition |
K = k/f | Gives k on average, so starves ~50% of the time | Add variance margin: k/f + z·sqrt(k/f) |
| No returned-count check | Short results look like a corpus gap | Alert when count < k |
| Assuming "supports filtering" = pre-filter | Post-filter is the default and the trap | Verify your engine's behaviour |
| Index per tenant with tiny tenants | Paying minimum size for empty space — 2.4× | Tier partitions |
| New vertical = new code branch | Verticals diverge; every one is a deploy | Config pack, one code path |
| Config with conditionals in it | You've built a bad programming language | Move logic back to code |
| Cache keyed on query only | Serves one tenant's answer to another | Key on (tenant, query) |
| One aggregate eval set | Hides per-tenant regressions | Segment by tenant |
Honest limits. Part A models survivors as Binomial(K, f), which assumes a tenant's documents are distributed independently of similarity rank. They are not: a tenant's corpus is usually topically clustered, so for queries in that tenant's subject area their documents are over-represented near the top and starvation is milder than modelled; for off-topic queries it is worse. The direction and rough magnitude hold, the exact percentages do not. The normal approximation is also poor in the extreme tail (small K·f), where the true probabilities are close to 1 anyway. Part B's 2 GB minimum index size and 200,000 vectors per tenant are illustrative — the overhead ratio M/(v·b) is the transferable formula and your provider's minimum may be much smaller or effectively zero for serverless offerings, which changes the conclusion entirely. Blast radius in part C assumes ingestion faults are independent across partitions; a bad shared pipeline breaks all of them regardless of index layout, so partitioning bounds data-level faults and not code-level ones. Nothing here measures query latency, cross-tenant fan-out cost, or the operational burden of managing N indexes, which is real and grows with N.
9. Summary + related articles
- Scaling RAG is about more audiences, not more documents. The document axis is solved; the tenant axis is not.
- Post-filter starvation is the shared-index trap. A 2% tenant still returns fewer than 10 results 99.7% of the time at
K=200, and needs K≈990 — 99× waste. Nothing errors. - Your smallest tenant sets
Kfor everyone, because fetch depth belongs to the query path.K = k/fstarves half the time; you need variance margin. - Alert when a filtered query returns fewer than
k. One line, converts silent to visible. - Per-tenant overhead is constant in
n—max(1, M/(v·b)), here 2.4×. Nearly free if tenants exceed the minimum index size; wasteful if they don't. - Tier partitions are the usual answer: 4 partitions cut blast radius 100% → 25% at a fraction of per-tenant cost.
- Make a vertical a config pack discovered at startup, validated on load, versioned with the code — one code path for all verticals.
- Boundary: for a single audience none of this applies. Adopt the config-pack shape at vertical two, and partition only when a starving tenant, a residency rule, or an incident forces it.
Related:
- RAG Monitoring: Your Error Rate Will Not Tell You Anything — the returned-count and per-tenant segmentation that make §3.1's silent failure visible
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — index minimums, footprint arithmetic, and why N tenants means N migrations
- Prompt Management Architecture: Prompts as Files, Not Strings — the same argument for prompts: behaviour belongs in versioned files, not code
- RAG Cost Optimization: Find the Step That Runs Forty Times — over-fetching for small tenants multiplies the reranking bill, the largest line item
- Agentic RAG: Routing Retrieval to Specialists — config packs also declare which specialists a vertical may route to
- RAG Evaluation: Attributing Failure and Sizing the Eval Set — why one aggregate eval set hides the per-tenant regressions that matter
Resources
- Dean & Barroso, "The Tail at Scale", Communications of the ACM 56(2), 2013 — tail behaviour and fan-out, directly relevant to per-tenant index fan-out.
- Malkov & Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs", IEEE TPAMI 42(4), 2020 (arXiv:1603.09320) — why filtering interacts badly with graph-based ANN traversal.
- Wei, Wu, Sarwar et al., "AnalyticDB-V: A Hybrid Analytical Engine Towards Query Fusion for Structured and Unstructured Data", VLDB 2020 — pre-filtered versus post-filtered vector search, the §3.1 mitigation.
- Chong, Carraro & Bernstein, "Multi-tenant databases for software as a service: schema-mapping techniques", SIGMOD 2008 — the shared-versus-isolated trade in its original database form.
- Gunda, Ravindranath, Thekkath et al. and the broader SaaS literature on noisy neighbours — see also AWS, "SaaS Tenant Isolation Strategies" whitepaper, for the tier-partition pattern in §3.2.