TL;DR — A vector database stores one embedding per chunk and answers "which vectors are nearest this one." Choosing it looks like a database decision; the choice that actually constrains you is the embedding model, because an index is only meaningful for vectors produced by the model that built it. Vectors from two models occupy unrelated coordinate systems, so mixing them destroys retrieval outright — measured below, retrieval falls from 98.5% recall@5 to 2.5%, which is chance (2.5%). Padding or truncating to make the dimensions line up does not help; it merely stops the error being raised. That makes changing embedding models a full re-index, not a config change: 10M chunks is 3.8B tokens, ~$494, and ~32 hours of wall clock in the accounting here, and every zero-downtime strategy needs 2× storage for the duration. Dimension is the other lever, and it is linear in both directions — 3072-d float32 → 1024-d int8 is a 12× reduction, 122.9GB to 10.2GB per 10M vectors. Pick the embedding model deliberately; the database is the easy part to change later.
1. Simple explanation
A vector database does one thing: store a list of numeric vectors, and given a new vector, quickly find the nearest ones. That is it. The "AI" happened earlier, in the embedding model that turned text into those numbers.
This division matters more than it first appears. The database never sees your text and has no idea what the numbers mean. It just does geometry. All the semantics — what counts as similar to what — was baked in by the embedding model, permanently, at the moment each vector was written.
Which leads to the constraint that surprises people: the index and the embedding model are one unit. You cannot swap the model and keep the index. Every vector in there was written in the old model's coordinate system, and a new model's vectors are in a different one. They are not slightly different. They are unrelated.
Analogy — two people mapping the same city with different origins. Both use grid coordinates. One measures from the north-west corner in metres; the other from the cathedral in feet, rotated 40 degrees. Both maps are internally consistent, and either alone will get you around. Now take a coordinate from one map and look it up on the other. You do not arrive somewhere approximately right — you arrive somewhere arbitrary. Nothing errors, because "(412, 908)" is a perfectly valid coordinate on both. It just points to the wrong place. An embedding index is that map, and swapping models mid-flight is reading the second person's coordinates off the first person's grid.
2. Diagram
INGESTION (once, offline) QUERY (per request)
───────────────────────── ───────────────────
documents "why does it drop off wifi?"
│ │
▼ ▼
┌────────┐ ┌────────┐
│ CHUNK │ │ EMBED │ ← MUST be the
└───┬────┘ └───┬────┘ SAME model
▼ │
┌────────┐ the model that runs │
│ EMBED │ here defines the whole │
└───┬────┘ coordinate system │
▼ ▼
┌──────────────────────────┐ nearest-neighbour
│ VECTOR INDEX │◄─────────── search
│ id | vector | metadata │
└──────────────────────────┘
WHAT HAPPENS IF THEY DIFFER (measured, §5, 200 docs)
─────────────────────────────────────────────────────
corpus query dims recall@5
model A model A 128/128 98.5%
model B model B 384/384 100.0%
model A model B 128/384 -> pad 2.5% ← chance is 2.5%
▲
no exception raised. no warning.
just silently random results.
FOOTPRINT — linear in dimension AND in bytes per component
───────────────────────────────────────────────────────────
10M vectors: 3072 f32 ████████████████████████ 122.9 GB 1.00x
1536 f32 ████████████ 61.4 GB 0.50x
1024 f32 ████████ 41.0 GB 0.33x
1024 int8 ██ 10.2 GB 0.08x
768 int8 █ 7.7 GB 0.06x
3. How it works
3.1 What the database actually stores
Three things per record: an id, a vector, and metadata you can filter on. Retrieval is nearest-neighbour search over the vectors, optionally restricted by a metadata predicate.
Two consequences follow from that structure. Metadata filtering interacts badly with approximate search — filtering after the nearest-neighbour step can leave you with far fewer than k results, so check whether your database filters pre- or post-search, because the difference shows up as mysteriously short result lists. And metadata has size limits; oversized payloads get rejected or truncated at write time, which is why storing full chunk text as metadata eventually fails at scale rather than immediately.
3.2 The similarity metric must match the model
Cosine, dot product, and Euclidean are not interchangeable. Cosine ignores magnitude; dot product does not. If your model produces normalised vectors they coincide, and if it does not, they rank differently. Use the metric the model was trained for — it is stated in the model card — and set it at index creation, because most databases will not let you change it afterwards without a rebuild.
Sparse-dense hybrid indexes usually require dot product specifically, so if hybrid search is on your roadmap, choose that metric at the start rather than rebuilding later.
3.3 Approximate search, and what you trade for speed
Exact nearest-neighbour over ten million vectors is too slow, so production indexes are approximate (HNSW, IVF, and relatives). They trade a small amount of recall for orders of magnitude of speed, exposing a knob — ef_search, nprobe, or similar — that trades latency for recall at query time.
Worth knowing: this is a second recall ceiling stacked on top of the retrieval-quality one. If your ANN index returns 95% of the true nearest neighbours, that 5% is gone before reranking begins, and no downstream stage recovers it — the same unrecoverable-loss structure quantified in Reranking: The Second Pass That Decides What the Model Sees. When measuring retrieval quality, check the ANN parameters before concluding the embedding model is at fault.
3.4 Dimension: what it costs and what it buys
Footprint is linear in dimension and linear in bytes per component:
bytes = n_vectors * dimensions * bytes_per_component
Both factors are yours to choose. Measured for 10M vectors: 3072-d float32 is 122.9GB; 1024-d float32 is 41.0GB; 1024-d int8 is 10.2GB. That last is a 12× reduction, and it compounds — smaller vectors mean less memory, cheaper instances, and faster distance computations, so latency improves alongside cost.
What you give up is less obvious than it looks. Higher dimensions can represent finer distinctions, but the relationship with retrieval quality is weak and heavily model-dependent — a well-trained 1024-d model routinely beats a mediocre 3072-d one. Dimension is a property of the model you chose, not an independent quality dial. Some modern models are trained so their vectors can be truncated to a shorter prefix with graceful degradation, which turns dimension into a real runtime knob; if that matters to you, it is a model-selection criterion.
Quantisation (float32 → int8) is usually the better first move: it cuts 4× with a small, measurable recall cost, and unlike a dimension change it does not require re-embedding — you re-encode vectors you already have.
3.5 Why changing the embedding model is a migration
Here is the thing that catches teams out. Swapping embedding models is not a configuration change. Every vector in the index was produced by the old model, and the new model's output lives in an unrelated geometry. There is no adapter, no conversion, and no partial migration: an index containing vectors from two models returns nonsense for whichever model's queries it is not built for.
The failure mode is what makes this dangerous. Retrieval does not error. It returns results — plausible-looking, correctly-formatted, confidently-ranked results — that are effectively random. Measured in §5, retrieval collapsed from 98.5% to 2.5% recall@5 against a chance rate of 2.5%. If dimensions happen to match, nothing anywhere in the stack will tell you.
And when dimensions do not match, the temptation is to make them match. Truncating or zero-padding to align dimensions does not work — it converts a loud error into a silent one. The 2.5% row above is the padded case.
So a model change means re-embedding the entire corpus:
10,000,000 chunks x 380 tokens = 3.8B tokens
re-embedding, one pass = ~$494
at 2M tokens/min = ~32 hours wall clock
Money is rarely the obstacle; 32 hours is. That is a long window to plan around, and it is one pass with no failures.
3.6 Migration strategies
IN-PLACE OVERWRITE cheapest, 1x storage, and the index is WRONG for the
whole 32-hour window. No rollback. Only acceptable
if you can take the feature offline.
BUILD NEW + CUTOVER build a second index alongside, switch reads when
complete, delete the old one after a soak period.
2x storage during the window. Instant rollback.
This is the default choice.
DUAL-WRITE WINDOW write new documents to BOTH indexes while
backfilling the new one. Necessary when the corpus
is changing during the migration -- otherwise the
new index is stale on arrival.
SHADOW + COMPARE serve from the old index, run queries against the
new one in parallel, and diff the results before
cutting over. Catches a bad migration BEFORE users
see it. Costs an extra query per request.
Every zero-downtime option costs 2× storage while it runs — 246GB rather than 123GB for the 3072-d index above. Provision for the peak, not the steady state; discovering the quota ceiling 20 hours into a 32-hour backfill is a bad evening.
Two details that are easy to miss: keep the chunk ids stable across the migration so anything referencing them still resolves, and re-run your evaluation set against the new index before cutover, because a different model genuinely changes which documents rank highest, and "different" is not automatically "better."
3.7 Where this stops mattering
Below a few hundred thousand vectors, most of this is over-thinking. An exact search over 100k vectors with a library like FAISS, or even numpy in memory, is fast and free, and the operational simplicity beats a managed service. The migration concern still applies at any size — it is about correctness, not scale — but a re-index of 100k chunks takes minutes and costs cents, so you can simply do it.
4. The math
4.1 Why cross-model vectors are not merely "a bit off"
Both models map meaning into a vector space, but each learns its own basis. Model A's dimension 7 might encode something like formality; model B's dimension 7 encodes something unrelated. Comparing component 7 to component 7 compares two unrelated quantities.
Formally, if x is a document's latent meaning and each model applies its own transform, a = f_A(x) and b = f_B(x), then cos(a, b) carries no information about x unless f_A and f_B happen to share a basis. They do not — the bases come from independent training runs, and even the same architecture retrained with a different seed produces an incompatible space.
This is why the measured cross-model recall lands exactly at chance:
chance recall@5 over 200 documents = 5 / 200 = 2.5%
measured cross-model recall@5 = 2.5%
Not degraded. Uninformative. The similarity scores still come back in a plausible-looking 0-to-1 range, which is precisely what makes this hard to spot in production.
4.2 Footprint arithmetic
bytes = n * d * b
n = 10,000,000 vectors
d = 3072, b = 4 -> 122.9 GB
d = 1024, b = 4 -> 41.0 GB (d/3 -> memory/3)
d = 1024, b = 1 -> 10.2 GB (b/4 -> memory/4, so 12x total)
The two factors multiply, which is why combining a smaller model with quantisation is far more effective than either alone. Add the index structure itself — HNSW graphs commonly add 10–50% on top of the raw vectors — and the working set must fit in RAM for good latency, so this arithmetic is really about instance size.
4.3 Migration time and the 2× window
tokens = chunks * tokens_per_chunk = 10M * 380 = 3.8e9
cost = 3.8e9 / 1e6 * $0.13 = $494
wall clock = 3.8e9 / 2e6 per min = 1900 min = 31.7 hours
peak storage (zero-downtime) = 2 * 122.9 GB = 245.8 GB
Throughput, not price, is the binding constraint — the same conclusion as RAG Cost Optimization: Find the Step That Runs Forty Times, for the same reason: per-minute quota caps you regardless of budget. Parallelising across more workers only helps until you hit the embedding endpoint's own rate limit.
5. Real code
Standard library only, deterministic, runs in about five seconds. Part A gives each "model" its own random projection of a shared latent space — the mathematical stand-in for two models with independently learned bases. B and C are arithmetic.
import math
import random
SEED = 3
LATENT = 24 # "meaning" lives here
N_DOCS = 200
N_QUERIES = 200
def rand_matrix(rows, cols, rng):
return [[rng.gauss(0, 1) for _ in range(cols)] for _ in range(rows)]
def project(vec, mat):
"""mat is out_dim x in_dim."""
return [sum(w * v for w, v in zip(row, vec)) for row in mat]
def cosine(a, b):
na = math.sqrt(sum(x * x for x in a))
nb = math.sqrt(sum(x * x for x in b))
if na == 0 or nb == 0:
return 0.0
return sum(x * y for x, y in zip(a, b)) / (na * nb)
def fit_dim(vec, target):
"""The 'just make the dimensions match' hack: truncate or zero-pad."""
if len(vec) >= target:
return vec[:target]
return vec + [0.0] * (target - len(vec))
def build():
rng = random.Random(SEED)
# each document has a latent meaning; each query is a noisy version of one
latents = [[rng.gauss(0, 1) for _ in range(LATENT)] for _ in range(N_DOCS)]
targets = list(range(N_QUERIES))
q_latents = [[latents[t][i] + rng.gauss(0, 0.95) for i in range(LATENT)]
for t in targets]
# two embedding models: different dimensions, different learned geometry
model_a = rand_matrix(1024 // 8, LATENT, rng) # stand-in for a 1024-d model
model_b = rand_matrix(3072 // 8, LATENT, rng) # stand-in for a 3072-d model
return latents, q_latents, targets, model_a, model_b
def recall_at_k(doc_vecs, query_vecs, targets, k=5):
hits = 0
for qi, qv in enumerate(query_vecs):
scored = sorted(range(len(doc_vecs)),
key=lambda d: -cosine(qv, doc_vecs[d]))
if targets[qi] in scored[:k]:
hits += 1
return hits / len(query_vecs)
latents, q_latents, targets, MA, MB = build()
docs_a = [project(v, MA) for v in latents]
docs_b = [project(v, MB) for v in latents]
qs_a = [project(v, MA) for v in q_latents]
qs_b = [project(v, MB) for v in q_latents]
print("A. CAN YOU MIX VECTORS FROM TWO MODELS?")
print(f"{'corpus':>10} {'query':>10} {'dims':>12} {'recall@5':>10}")
print("-" * 46)
same_a = recall_at_k(docs_a, qs_a, targets)
same_b = recall_at_k(docs_b, qs_b, targets)
print(f"{'model A':>10} {'model A':>10} {f'{len(MA)}/{len(MA)}':>12} {same_a:10.1%}")
print(f"{'model B':>10} {'model B':>10} {f'{len(MB)}/{len(MB)}':>12} {same_b:10.1%}")
# the "just pad it" hack: corpus in A, query in B truncated to A's width
qs_b_fit = [fit_dim(v, len(MA)) for v in qs_b]
mixed = recall_at_k(docs_a, qs_b_fit, targets)
print(f"{'model A':>10} {'model B':>10} {f'{len(MA)}/{len(MB)}->pad':>12} "
f"{mixed:10.1%}")
chance = 5 / N_DOCS
print(f"{'(chance)':>10} {'':>10} {'':>12} {chance:10.1%}")
print(f"\nmixing models: {mixed:.1%} vs {same_a:.1%} same-model "
f"-- {'at chance' if mixed < 3*chance else 'above chance'}")
print("\n\nB. INDEX FOOTPRINT (10,000,000 vectors)")
N = 10_000_000
print(f"{'dims':>6} {'precision':>10} {'bytes/vec':>10} {'total':>10} "
f"{'vs 3072 f32':>12}")
print("-" * 54)
base = None
for dims, prec, byt in ((3072, "float32", 4), (1536, "float32", 4),
(1024, "float32", 4), (1024, "int8", 1),
(768, "int8", 1)):
per = dims * byt
tot = N * per / 1e9
if base is None:
base = tot
print(f"{dims:>6} {prec:>10} {per:10,} {tot:9.1f}GB {tot/base:11.2f}x")
print("\n memory is linear in BOTH dimension and bytes-per-component,")
print(" so 3072-f32 -> 1024-int8 is a 12x reduction.")
print("\n\nC. WHAT THE MIGRATION COSTS")
CHUNKS = 10_000_000
TOK_PER_CHUNK = 380
EMBED_PER_MTOK = 0.13 # illustrative $ per 1M tokens
tokens = CHUNKS * TOK_PER_CHUNK
embed_cost = tokens / 1e6 * EMBED_PER_MTOK
print(f" {CHUNKS:,} chunks x {TOK_PER_CHUNK} tokens = {tokens/1e9:.1f}B tokens")
print(f" re-embedding cost, one pass: ${embed_cost:,.0f}")
THROUGHPUT = 2_000_000 # tokens/min
print(f" at {THROUGHPUT:,} tok/min that is {tokens/THROUGHPUT/60:.1f} hours "
f"of wall clock")
print(f"\n{'strategy':>22} {'downtime':>10} {'peak storage':>13} {'rollback':>10}")
print("-" * 60)
for name, down, storage, rb in (
("in-place overwrite", "full", "1x", "none"),
("build new + cutover", "none", "2x", "instant"),
("dual-write window", "none", "2x", "instant"),
("shadow + compare", "none", "2x", "instant")):
print(f"{name:>22} {down:>10} {storage:>13} {rb:>10}")
print("\n every zero-downtime option costs 2x storage for the window.")
print(f" 2x on the 3072-f32 index above = "
f"{2*N*3072*4/1e9:.0f}GB provisioned, not {N*3072*4/1e9:.0f}GB.")
# claims made in the prose
assert same_a > 0.9 and same_b > 0.9, "same-model retrieval must work"
assert mixed < 3 * chance, "cross-model retrieval must collapse to chance"
assert abs(same_a - same_b) < 0.20, "both models should work on their own"
print("\nasserts passed")
# Output:
# A. CAN YOU MIX VECTORS FROM TWO MODELS?
# corpus query dims recall@5
# ----------------------------------------------
# model A model A 128/128 98.5%
# model B model B 384/384 100.0%
# model A model B 128/384->pad 2.5%
# (chance) 2.5%
#
# mixing models: 2.5% vs 98.5% same-model -- at chance
#
#
# B. INDEX FOOTPRINT (10,000,000 vectors)
# dims precision bytes/vec total vs 3072 f32
# ------------------------------------------------------
# 3072 float32 12,288 122.9GB 1.00x
# 1536 float32 6,144 61.4GB 0.50x
# 1024 float32 4,096 41.0GB 0.33x
# 1024 int8 1,024 10.2GB 0.08x
# 768 int8 768 7.7GB 0.06x
#
# memory is linear in BOTH dimension and bytes-per-component,
# so 3072-f32 -> 1024-int8 is a 12x reduction.
#
#
# C. WHAT THE MIGRATION COSTS
# 10,000,000 chunks x 380 tokens = 3.8B tokens
# re-embedding cost, one pass: $494
# at 2,000,000 tok/min that is 31.7 hours of wall clock
#
# strategy downtime peak storage rollback
# ------------------------------------------------------------
# in-place overwrite full 1x none
# build new + cutover none 2x instant
# dual-write window none 2x instant
# shadow + compare none 2x instant
#
# every zero-downtime option costs 2x storage for the window.
# 2x on the 3072-f32 index above = 246GB provisioned, not 123GB.
#
# asserts passed
The ->pad row is the one to remember. Both models work fine alone — 98.5% and 100%. Padding model B's query to model A's width produces a syntactically valid vector, a valid cosine similarity, a valid ranked list, and 2.5% recall against a 2.5% chance rate. Every layer of the stack reports success.
6. Real-world example
A team upgraded their embedding model to a newer, better one from the same provider. Same API shape, same call, better benchmark scores, and — as it happened — the same 1536 dimensions.
They deployed it to the query path on a Thursday. The index still held vectors from the old model.
Nothing broke. No exception, no alert, no error rate change. Latency was identical. The system returned five documents per query with similarity scores in a normal-looking range, and the model wrote fluent answers citing them. The answers were simply about the wrong documents.
It ran for six days. Support tickets rose, but the complaints were vague — "the assistant seems worse lately" — and were initially attributed to a prompt change shipped the same week. The prompt got rolled back. Nothing improved. Someone eventually spot-checked a query by hand, saw that the top result was unrelated to the question, and the cause took about ten minutes to find once anyone was looking at retrieval at all.
The matching dimensions were the trap. Had the new model produced 3072-d vectors, the database would have rejected every query immediately and the bug would have lasted minutes. Because the shapes agreed, the only thing that disagreed was the meaning, and nothing in the stack checks meaning.
The fix afterwards was one line of defence, not a better process: store the embedding model's name and version as index metadata, and assert on every query that the querying model matches. It turns a silent six-day failure into a loud startup error.
7. Interview questions companies actually ask
Q1 [easy] "What does a vector database actually store?"
A An id, a vector, and filterable metadata per record -- and it answers
nearest-neighbour queries over the vectors. It never sees your text and has no
idea what the numbers mean. All the semantics were fixed by the embedding model
at write time; the database only does geometry.
Q2 [easy] "Can you change the embedding model without re-indexing?"
A No. Every stored vector is in the old model's coordinate system and the new
model's vectors are in an unrelated one -- different training runs learn
different bases, so component 7 means different things. Measured: retrieval falls
from 98.5% to 2.5% recall@5, where chance is 2.5%. Not degraded; uninformative.
Q3 [medium] "The new model has different dimensions. Can you pad or truncate to
match?"
A No, and it's worse than not trying -- it converts a loud dimension-mismatch
error into a silent wrong-results bug. The 2.5% measurement above IS the padded
case. You get a valid vector, a valid cosine score, and a valid ranked list of
the wrong documents.
Q4 [medium] "How would you catch a model/index mismatch in production?"
A You won't catch it from error rates, latency, or output format -- all three look
normal. Store the embedding model name and version as index metadata and assert
the querying model matches, ideally at startup. That converts a silent failure
into a loud one. Also keep a small golden query set with known-correct top hits
and run it as a smoke test after any deploy touching embeddings.
Q5 [medium] "Should you use 3072 dimensions or 1024?"
A Footprint is linear in dimension: 10M vectors is 122.9GB at 3072-f32 versus
41.0GB at 1024-f32. But dimension isn't an independent quality dial -- it's a
property of the model, and a well-trained 1024-d model routinely beats a
mediocre 3072-d one. Pick the model on retrieval quality, then treat its
dimension as a consequence. Quantisation is usually the better first cut anyway:
f32 to int8 is 4x and needs no re-embedding.
Q6 [medium] "How do you plan the migration?"
A Build a new index alongside the old, backfill it, evaluate on a labelled set,
then cut reads over and soak before deleting. Dual-write new documents during the
backfill if the corpus is changing, or the new index is stale on arrival. Budget
2x storage for the window -- 246GB not 123GB -- and expect ~32 hours for 10M
chunks. Keep chunk ids stable so external references still resolve.
Q7 [hard] "What limits migration speed?"
A Throughput, not money. 10M chunks is 3.8B tokens: roughly $494, but ~32 hours at
2M tokens/min. Parallelising helps only until you hit the embedding endpoint's own
rate limit. Same shape as the TPM ceiling on the query path -- per-minute quota
binds before budget does.
Q8 [hard] "Your recall is worse than the embedding benchmark suggests. What else
could it be?"
A The ANN index. Approximate search trades recall for speed via ef_search/nprobe,
and anything it misses is gone before reranking -- an unrecoverable loss stacked
on top of embedding quality. Check those parameters before blaming the model.
Also check whether metadata filtering happens pre- or post-search: post-filtering
can silently return far fewer than k results.
8. When to use / tradeoffs
YOU NEED A MANAGED VECTOR DATABASE WHEN:
+ more than a few million vectors, or they must not fit in one process
+ you need filtered search, replication, and durability you don't maintain
+ write throughput is continuous rather than a nightly rebuild
YOU DON'T WHEN:
- under ~100k vectors: an in-process exact search is fast, free, simpler
- the corpus is static and rebuilt wholesale on a schedule
- you only ever query with one filter -- separate indexes may be simpler
| Situation | Why it breaks | Use instead |
|---|---|---|
| Swapping embedding model in place | Vectors incomparable; recall → chance, silently | Re-index; new index + cutover |
| Padding/truncating to match dims | Turns a loud error into a silent wrong answer | Re-embed with one model |
| Same dims, different model | Nothing errors anywhere in the stack | Assert model version on every query |
| Choosing dimension for "quality" | Dimension is a model property, not a dial | Pick model on recall; quantise for footprint |
| Metric ≠ what the model was trained for | Cosine and dot product rank differently unless normalised | Use the model card's metric; set at creation |
| Blaming the model for poor recall | ANN parameters cap recall before the model does | Check ef_search/nprobe first |
| Provisioning for steady-state storage | Zero-downtime migration needs 2× | Provision for the migration peak |
| Post-filtering with ANN | Returns fewer than k unpredictably | Pre-filtered search, or over-fetch |
Honest limits. Part A models each embedding model as a random linear projection of a shared latent space. That is the right intuition for why two models are incompatible — independently learned bases — but real embedding models are non-linear and trained on overlapping data, so their spaces are not perfectly unrelated; a learned linear map between two real embedding spaces can recover some alignment, which is an active research area. The practical conclusion is unchanged, because nobody ships such a map: without one, you get the chance-level result measured here. The 98.5%/100% same-model figures are artefacts of a small 200-document corpus with a generous signal-to-noise ratio; real recall@5 is far lower and the point is only the contrast with the 2.5% row. Costs in part C use an illustrative $0.13/M tokens and 2M tokens/min — both vary by provider and tier by more than an order of magnitude, so re-run the arithmetic rather than quoting the $494. Footprint numbers exclude the ANN graph structure (commonly +10–50%), replication factor, and metadata, so treat 122.9GB as a floor. Finally, this article says nothing about the recall cost of quantisation, which is real, model-specific, and must be measured on your own data before you take the 4×.
9. Summary + related articles
- A vector database stores id + vector + metadata and does geometry. All the meaning was fixed by the embedding model at write time.
- The index and the embedding model are one unit. Mixing models collapsed retrieval from 98.5% to 2.5% recall@5 — exactly chance — with no error raised anywhere.
- Padding or truncating to align dimensions makes it worse, converting a loud failure into a silent one. That is the 2.5% row.
- Store the model name/version as index metadata and assert it on query. This is the whole defence, and it is one line.
- Footprint is linear in dimension and in precision: 3072-f32 → 1024-int8 is 12× (122.9GB → 10.2GB per 10M vectors). Quantise first — it needs no re-embedding.
- Dimension is a property of the model, not a quality dial. Choose the model on measured recall.
- A model change is a ~32-hour migration for 10M chunks, needing 2× storage for any zero-downtime strategy. Throughput binds before budget.
- Boundary: below ~100k vectors, use an in-process exact index. The migration hazard applies at every scale, but a small re-index is minutes and cents.
Related:
- Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — what goes into each vector, and how chunk size sets the number of vectors you store
- Reranking: The Second Pass That Decides What the Model Sees — §4.1 on unrecoverable recall loss, which is the same structure as the ANN ceiling in §3.3
- RAG Cost Optimization: Find the Step That Runs Forty Times — the query-side counterpart: throughput caps bind before budget there too
- Multi-Index RAG: Merging Several Retrievers Into One Answer — running several indexes at once, including ones built by different models on purpose
- Query Transformation: Fixing the Question Before You Retrieve — improving what you send to the index, usually cheaper than changing the index
Resources
- Malkov & Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs", IEEE TPAMI 42(4), 2020 (arXiv:1603.09320) — the HNSW algorithm behind most production indexes, and the recall/latency knobs in §3.3.
- Johnson, Douze & Jégou, "Billion-Scale Similarity Search with GPUs", IEEE Transactions on Big Data 7(3), 2021 (arXiv:1702.08734) — FAISS; the reference for exact and IVF search at the small end of §3.7.
- Jégou, Douze & Schmid, "Product Quantization for Nearest Neighbor Search", IEEE TPAMI 33(1), 2011 — the foundation of the compression in §3.4.
- Kusupati et al., "Matryoshka Representation Learning", NeurIPS 2022 (arXiv:2205.13147) — embeddings trained so a truncated prefix stays usable, which makes dimension a runtime knob.
- Muennighoff et al., "MTEB: Massive Text Embedding Benchmark", EACL 2023 (arXiv:2210.07316) — how to compare embedding models on retrieval rather than on dimension count.