TL;DR — Retrieval finds passages by comparing meaning rather than words: each passage becomes a vector, the question becomes a vector, and you rank by cosine similarity — the angle between them, which ignores length. Two things follow that trip up almost every first RAG system. Top-k always returns k results, however irrelevant, so a question about the weather still gets handed your most weather-ish document with a score of 0.219 and answered from it. And cosine, not raw dot product, is what makes ranking length-invariant. So a working retriever needs a minimum score as well as a top-k, or it cannot distinguish "here is the answer" from "here is the closest thing I have." It stops working when relevance isn't semantic similarity — exact identifiers, negations, and freshness all rank badly no matter how good the embedding.
1. Simple explanation
Keyword search matches letters. Ask "can I send this back?" and a keyword search for those words will miss a document headed "Returns policy" — no shared words, same meaning.
Vector search fixes this by turning text into numbers first. An embedding model reads a passage and produces a list of numbers — a vector — positioned so that passages about similar things end up pointing in similar directions. Do the same to the question, and finding the relevant passage becomes: whose vector points most nearly the same way as the question's?
"Most nearly the same way" is measured as an angle. Same direction scores 1.0, unrelated scores near 0. You sort by that number and take the top few.
Analogy — a library with no catalogue, arranged by topic. Books about returns end up on the same shelf, whether their titles say "returns", "refunds", or "sending things back". You walk to the position your question belongs at, and grab what's nearest. That works beautifully — and it has a catch the analogy makes obvious: there is always something nearest. Ask about the weather in a library with no weather section and you'll still be standing next to some book. Nothing about "grab the nearest" tells you whether the nearest is any good. That is the whole argument for a score threshold, and it's the mistake this article is really about.
2. Diagram
TEXT → VECTOR → RANK BY ANGLE
"refund policy" ──embed──▶ [0.94, 0.20, 0.10, 0.02]
"shipping times" ──embed──▶ [0.18, 0.95, 0.12, 0.03]
"opening hours" ──embed──▶ [0.09, 0.14, 0.96, 0.05]
"how do I get a refund" ─embed─▶ [0.90, 0.25, 0.08, 0.01]
│
cosine similarity against each document
▼
0.998 refund policy ◀── answer from here
0.448 shipping times
0.122 opening hours
THE TRAP: top-k ALWAYS returns k
query: "what is the weather" (no weather document exists)
0.219 office location ◀── ranked FIRST, and useless
0.122 opening hours
top-k has no opinion about whether 0.219 is good enough.
It ranks. It never declines.
THE FIX: rank, THEN apply a floor
┌──────────────────────────┐
query ──embed──▶ │ cosine vs every document │
└────────────┬─────────────┘
▼ sorted
┌──────────────────────────┐
│ top-k (k=2) │ ranks
└────────────┬─────────────┘
▼
┌──────────────────────────┐
│ score >= 0.60 ? │ DECIDES
└──────┬────────────┬──────┘
yes│ │no
▼ ▼
answer ABSTAIN
WHY COSINE, NOT RAW DOT PRODUCT
same direction, 5x the length:
dot(q, short) = 0.879 dot(q, long) = 4.395 ← differ 5x
cos(q, short) = 1.000 cos(q, long) = 1.000 ← identical
raw dot lets a LONGER vector outrank a more relevant one.
3. How it works
3.1 Embedding: text becomes a direction
An embedding model maps a passage to a fixed-length vector — commonly a few hundred to a few thousand numbers. The individual numbers mean nothing you can name; what matters is that the geometry carries meaning, so passages on the same topic land near each other.
Two practical consequences. Every passage becomes the same size regardless of its length, so you can compare a sentence to a paragraph. And the query and the documents must be embedded by the same model — vectors from different models live in unrelated spaces, and comparing them produces confident nonsense rather than an error. What an embedding is and how it is trained is Embeddings and Cosine Similarity; this article is about what you do with them.
3.2 Cosine similarity, and why not the alternatives
cosine(a, b) = dot(a, b) / (|a| × |b|)
Dividing by both lengths is the entire point: it strips magnitude out and leaves only direction. That matters because vector length correlates with things you don't want deciding relevance — passage length, word repetition, formatting.
Use raw dot product and a long, repetitive document can outrank a short, precisely relevant one purely by having a bigger magnitude. Euclidean distance has the same problem in reverse. Cosine is the default for text retrieval for this reason, and the numbers in §2 show it: two vectors with identical meaning and a 5× length difference score identically under cosine and differ 5× under dot.
One shortcut worth knowing: if all your vectors are normalised to length 1 — many embedding APIs do this for you — then cosine and dot product are the same computation, and dot is faster. Check whether yours are normalised before optimising this.
3.3 Top-k ranks. It does not decide.
This is the section that matters most, and it is one line: top_k is a sort followed by a slice. Nothing in it can conclude that the best match is bad.
So a query with no good match still produces a ranked list, and the top entry still has a score, and code written as answer_from(top_k(q, 1)[0]) will cheerfully answer from it. In §4 the weather question — with no weather document in the corpus — retrieves "office location" at 0.219 and would be answered from it.
The fix is a second, separate decision: a minimum score. Below it, you don't answer, you abstain.
hits = top_k(query, k=3)
hits = [h for h in hits if h.score >= MIN_SCORE] # ← the line teams omit
if not hits:
return abstain()
Choosing MIN_SCORE is empirical, not theoretical. Score distributions differ per embedding model and per corpus, so a threshold copied from a blog post will be wrong for you. Take a set of questions you know the answers to plus a set you know are out of scope, look at both score distributions, and pick the value that separates them. Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems goes further on this.
3.4 Choosing k, and the two-stage pattern
Small k risks missing the passage that held the answer. Large k costs input tokens and dilutes the prompt with near-misses that a model may latch onto.
The common production shape is two stages: retrieve generously (k = 20–50) with cheap vector search, then rerank that shortlist with a slower, more accurate model and keep the best 3–5. You get the recall of a big k and the precision of a small one, and you only pay the expensive model on a shortlist. That's Reranking: The Second Pass That Decides What the Model Sees.
3.5 Exact search versus approximate
Comparing a query against every vector is exact and linear in corpus size — fine for thousands of passages, too slow for millions. At scale you switch to approximate nearest neighbour (ANN) search, which trades a small amount of recall for a very large speed gain.
The word to notice is approximate: ANN can miss a true best match. Usually that's an excellent trade, but it means "the answer was in the corpus and retrieval didn't find it" becomes a possible failure, and it should be on your list when debugging. Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For covers the index types.
3.6 Where semantic similarity is the wrong tool
Vector search retrieves what is about the same thing. Several kinds of relevance are not that:
- Exact identifiers — order numbers, SKUs, error codes, names.
ERR-4021andERR-4012are semantically near-identical and operationally unrelated. Keyword or exact matching wins; the usual answer is hybrid search running both and merging. - Negation — "items that are not returnable" embeds very close to "items that are returnable". Embeddings are famously weak here, and a retriever will happily hand you the passage that says the opposite.
- Freshness and authority — similarity has no notion of which document is current, as What RAG Is and When to Use It §6 shows. Ranking is a similarity question; currency is a metadata question.
- Numeric or date filters — "invoices over £10,000 from last March" is a database query wearing a sentence.
4. The math
4.1 The three quantities
dot(a, b) = sum over dimensions of a_i * b_i
|a| = sqrt(dot(a, a)) (the vector's length)
cosine = dot(a, b) / (|a| * |b|) in [-1, 1]; for text, ~[0, 1]
ranking = sort documents by cosine(query, doc), descending
top-k = take the first k <- ranks
threshold = keep only score >= MIN_SCORE <- DECIDES
4.2 Worked example
Four documents in a 4-dimensional space — real embeddings have hundreds or thousands of dimensions, but the arithmetic is identical. The axes here happen to be interpretable so you can check the result by eye: [refunds, shipping, hours, weather].
refund policy [0.94, 0.20, 0.10, 0.02]
shipping times [0.18, 0.95, 0.12, 0.03]
opening hours [0.09, 0.14, 0.96, 0.05]
office location [0.11, 0.22, 0.44, 0.07]
Three questions. The first two have a genuine answer in the corpus; the third does not.
query: 'how do I get a refund'
0.998 refund policy
0.448 shipping times
-> would answer from 'refund policy' (score 0.998)
query: 'when do you close'
1.000 opening hours
0.936 office location
-> would answer from 'opening hours' (score 1.000)
query: 'what is the weather'
0.219 office location
0.122 opening hours
-> would answer from 'office location' (score 0.219)
Look at the third block. There is no weather document, and retrieval still returned a ranked list, still put something first, and a naive caller would still answer from it. The score is 0.219 — a quarter of the way to relevant — and top-k neither knows nor cares.
Note also the second block's runner-up: "office location" scores 0.936 for "when do you close", because location and hours genuinely are related topics. High scores are not proof of correctness; they mean "close in this space."
Now add a floor at 0.60:
0.998 'how do I get a refund' -> answer from 'refund policy'
1.000 'when do you close' -> answer from 'opening hours'
0.219 'what is the weather' -> ABSTAIN (nothing relevant enough)
One line of code turns a confidently wrong answer into an honest one.
4.3 Cosine versus dot, numerically
Take a vector and multiply every component by 5 — same direction, same meaning, five times the length:
dot(q, short) = 0.879 dot(q, long) = 4.395 <- differ 5x
cos(q, short) = 1.000 cos(q, long) = 1.000 <- identical
Under raw dot product, the longer vector wins by a factor of 5 despite carrying no more relevance. That is why ranking uses cosine.
5. Real code
"""Cosine similarity, top-k, and why top-k alone will always hand you something."""
import math
# Hand-made 4-dimensional vectors so this runs anywhere. Real embeddings have
# hundreds or thousands of dimensions; the arithmetic below is identical.
# [refunds, shipping, hours, weather]
DOCS = {
"refund policy": [0.94, 0.20, 0.10, 0.02],
"shipping times": [0.18, 0.95, 0.12, 0.03],
"opening hours": [0.09, 0.14, 0.96, 0.05],
"office location": [0.11, 0.22, 0.44, 0.07],
}
QUERIES = {
"how do I get a refund": [0.90, 0.25, 0.08, 0.01], # clearly about refunds
"when do you close": [0.10, 0.12, 0.93, 0.04], # clearly about hours
"what is the weather": [0.03, 0.05, 0.06, 0.97], # in NO document
}
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
def norm(a):
return math.sqrt(dot(a, a))
def cosine(a, b):
"""1.0 = same direction, 0.0 = unrelated. Length-invariant, unlike raw dot."""
return dot(a, b) / (norm(a) * norm(b))
def top_k(query_vec, k=2):
scored = [(cosine(query_vec, v), name) for name, v in DOCS.items()]
return sorted(scored, reverse=True)[:k]
print("TOP-K ALONE — it always returns k results, relevant or not")
for q, qv in QUERIES.items():
hits = top_k(qv, k=2)
best_score, best_name = hits[0]
print(f"\n query: {q!r}")
for score, name in hits:
print(f" {score:.3f} {name}")
print(f" -> would answer from {best_name!r} (score {best_score:.3f})")
THRESHOLD = 0.60
print(f"\n\nWITH A SCORE THRESHOLD of {THRESHOLD}")
for q, qv in QUERIES.items():
score, name = top_k(qv, k=1)[0]
if score >= THRESHOLD:
print(f" {score:.3f} {q!r} -> answer from {name!r}")
else:
print(f" {score:.3f} {q!r} -> ABSTAIN (nothing relevant enough)")
# Why cosine and not raw dot product: length must not decide relevance.
short = [0.90, 0.25, 0.08, 0.01]
long_ = [x * 5 for x in short] # same meaning, 5x the magnitude
q = QUERIES["how do I get a refund"]
print("\n\nCOSINE vs RAW DOT — same direction, different length")
print(f" dot(q, short) = {dot(q, short):.3f} dot(q, long) = {dot(q, long_):.3f} <- differ 5x")
print(f" cos(q, short) = {cosine(q, short):.3f} cos(q, long) = {cosine(q, long_):.3f} <- identical")
weather_score = top_k(QUERIES["what is the weather"], k=1)[0][0]
assert weather_score < THRESHOLD, weather_score
assert top_k(QUERIES["how do I get a refund"], k=1)[0][1] == "refund policy"
assert top_k(QUERIES["when do you close"], k=1)[0][1] == "opening hours"
# Top-k ranks; it never decides that nothing is good enough. That is the threshold's job.
assert len(top_k(QUERIES["what is the weather"], k=2)) == 2
assert abs(cosine(q, short) - cosine(q, long_)) < 1e-12
print(f"\nthe weather query's best match scores {weather_score:.3f} -- "
f"retrieved, ranked first, and useless")
print("all assertions passed")
# Output:
# TOP-K ALONE — it always returns k results, relevant or not
#
# query: 'how do I get a refund'
# 0.998 refund policy
# 0.448 shipping times
# -> would answer from 'refund policy' (score 0.998)
#
# query: 'when do you close'
# 1.000 opening hours
# 0.936 office location
# -> would answer from 'opening hours' (score 1.000)
#
# query: 'what is the weather'
# 0.219 office location
# 0.122 opening hours
# -> would answer from 'office location' (score 0.219)
#
#
# WITH A SCORE THRESHOLD of 0.6
# 0.998 'how do I get a refund' -> answer from 'refund policy'
# 1.000 'when do you close' -> answer from 'opening hours'
# 0.219 'what is the weather' -> ABSTAIN (nothing relevant enough)
#
#
# COSINE vs RAW DOT — same direction, different length
# dot(q, short) = 0.879 dot(q, long) = 4.395 <- differ 5x
# cos(q, short) = 1.000 cos(q, long) = 1.000 <- identical
#
# the weather query's best match scores 0.219 -- retrieved, ranked first, and useless
# all assertions passed
The vectors are hand-written so this runs with no dependencies and so you can verify the ranking by eye. Swap in a real embedding model and only the numbers change — cosine, top_k, and the threshold logic are exactly what production code does.
6. Real-world example
A team put vector search over a product support corpus. Retrieval quality looked good in testing: they had a list of realistic questions, checked that the right document came back top, and it did.
In production a specific complaint pattern appeared. Customers quoting an error code got answers about a different error code — plausible-sounding, wrong remedy, occasionally destructive advice.
The cause was §3.6. Error codes like ERR-4021 and ERR-4012 are, to an embedding model, almost the same string in almost the same context; their vectors are nearly identical. Semantic similarity is exactly the wrong measure for an identifier, where a one-character difference changes the meaning completely. The retriever was not malfunctioning — it was answering the question it had been asked, which was "what is this about", when the question that mattered was "which code is this exactly".
Two things had hidden it. Their test questions were all natural language, so no test exercised an identifier. And the scores were high — around 0.95 — because the passages genuinely were about the same topic, so a score threshold offered no protection at all. A threshold catches irrelevant retrieval; it cannot catch confusable retrieval.
The fix was hybrid search: run an exact keyword match alongside the vector search and let an exact identifier hit win. The broader lesson was about the test set — it contained only the kind of question the technique was already good at.
7. Interview questions companies actually ask
Q1. Why use vector search instead of keyword search? Because relevance is usually about meaning, not spelling. "Can I send this back?" shares no words with a document headed "Returns policy", so keyword search misses it while vector search ranks it first. The trade is that you lose exactness, which is why identifiers and codes still need keyword matching — most production systems run both and merge.
Q2. Why cosine similarity rather than dot product or Euclidean distance? Because relevance should not depend on length. Cosine divides out both magnitudes and compares direction only, so a long repetitive passage cannot outrank a short precise one just by being bigger. Raw dot product has exactly that bug — the same vector scaled 5× scores 5× higher while meaning the same thing. If your vectors are already normalised to unit length, cosine and dot are identical and dot is cheaper.
Q3. What's wrong with just taking the top result? Top-k is a sort and a slice — it ranks, it never declines. A question with no good answer in the corpus still produces a first result with a score, and code that answers from it will confidently answer from the closest available thing. In the worked example, a weather question retrieves an office-location document at 0.219 and would be answered from it. You need a separate minimum-score decision.
Q4. How do you choose the score threshold? Empirically, never by copying a number. Score distributions vary by embedding model and corpus, so gather questions you know are answerable and questions you know are out of scope, plot both distributions, and pick the value that best separates them. Then keep watching it, because changing the embedding model invalidates it entirely.
Q5. How do you pick k? Small k risks missing the passage that held the answer; large k costs input tokens and dilutes the prompt with near-misses. The standard resolution is two stages: retrieve generously with cheap vector search, then rerank that shortlist with a slower more accurate model and keep the best few. You get recall from the wide first pass and precision from the second, paying the expensive model only on a shortlist.
Q6. When does semantic similarity fail even with a perfect embedding model? Exact identifiers, where near-identical strings mean unrelated things. Negation, because "not returnable" sits very close to "returnable". Freshness and authority, since similarity has no notion of which document is current. And anything that's really a filter — date ranges, numeric comparisons — which is a database query rather than a similarity question.
Q7. Your retriever returns high scores for the wrong passage. What's happening? Probably confusable content rather than irrelevant content — passages genuinely about the same topic that differ in a detail the embedding compresses away, like a code or a number. That's the dangerous case, because a score threshold gives you no protection when the score is 0.95. Look for hybrid search, and check whether your test set contains the kind of query that exposes it; usually it doesn't.
8. When to use / tradeoffs
Reach for vector search when:
- Users phrase things differently from your documents
- Relevance is topical — "about the same thing" is the right question
- The corpus is large enough that reading it all per question is impossible
- Paraphrases, synonyms, and multiple languages need to match
Reach for something else when:
- The query is an exact identifier → keyword or exact match
- The query is really a filter → database query
- The corpus is small and static → put it in the prompt
- Precision on a shortlist matters more than recall → add a reranker
| Situation | Why it breaks | Use instead |
|---|---|---|
| Error codes, SKUs, order numbers | Near-identical vectors, unrelated meanings | Hybrid: exact match + vector |
| Negations ("not returnable") | Embeds close to the affirmative | Explicit filters, or a reranker |
| "Docs from last March over £10k" | Similarity cannot express a filter | Metadata filtering first |
| Top-1 with no score check | Always returns something; answers from it | Minimum score, then abstain |
| Threshold copied from a blog post | Distributions differ per model and corpus | Measure on your own data |
| Query and docs embedded differently | Unrelated spaces; confident nonsense | Same model for both, always |
| Millions of vectors, exact search | Linear scan is too slow | ANN index, accepting some recall loss |
| Superseded documents in the index | Ranks by similarity, not currency | Delete or date old documents |
Honest limits. The vectors in §4 are hand-written with interpretable axes so the ranking can be checked by eye — real embedding dimensions mean nothing individually, and real scores cluster much more tightly, which makes thresholds harder to place than this example suggests. The 0.60 threshold is illustrative; on a real corpus the answerable and unanswerable distributions overlap, so any threshold trades false abstentions against false answers, and there is no value that avoids both. The article also treats cosine as the settled choice, which is true for text but not universal — some models are trained for dot product on unnormalised vectors, and using cosine there is a small mistake rather than a fix. Finally, a threshold defends against irrelevance and offers nothing against confusable content, as §6 shows: high score, wrong passage, no protection.
9. Summary + related articles
- Retrieval compares meaning: embed the passages, embed the question, rank by similarity. Paraphrases match; spelling doesn't have to.
- Cosine similarity compares direction and ignores length, so a long repetitive passage can't outrank a short precise one. Raw dot product has that bug — same vector at 5× length scored 5× higher.
- Top-k ranks; it never declines. A weather question with no weather document still retrieved something, at 0.219, and would have been answered from it.
- So a working retriever needs two decisions: top-k to rank, and a minimum score to decide. The second is the line teams omit.
- Choose the threshold empirically on your own answerable and out-of-scope questions. A copied number will be wrong.
- Query and documents must use the same embedding model — different models give confident nonsense, not an error.
- Retrieve generously, then rerank to a few. Recall from the wide pass, precision from the second.
- Semantic similarity is the wrong tool for exact identifiers, negation, freshness, and filters — hence hybrid search.
- A threshold catches irrelevant retrieval. It cannot catch confusable retrieval, where the score is high and the passage is still wrong.
Related:
- What RAG Is and When to Use It — why you are retrieving at all
- Anatomy of a RAG Pipeline — where retrieval sits, and the failure a threshold can't fix
- Embeddings and Cosine Similarity — what the vectors are and how they're trained
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — ANN indexes, and storing vectors at scale
- Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — choosing the score floor on real data
- Reranking: The Second Pass That Decides What the Model Sees — the second stage that turns high recall into high precision
- Query Transformation: Fixing the Question Before You Retrieve — rewriting the question before you embed it
- Multi-Index RAG: Merging Several Retrievers Into One Answer — hybrid and multiple indexes, the §6 fix generalised
- RAG Evaluation: Attributing Failure and Sizing the Eval Set — measuring retrieval separately from the answer
- Search Systems — ranking and retrieval as a systems problem
Resources
- Manning, Raghavan & Schütze — Introduction to Information Retrieval, Ch. 6 (term weighting and the vector space model) — free online, and the origin of cosine ranking for text: https://nlp.stanford.edu/IR-book/
- Karpukhin et al. (2020) — Dense Passage Retrieval for Open-Domain Question Answering, arXiv:2004.04906 — dense retrieval versus keyword baselines, with ablations: https://arxiv.org/abs/2004.04906
- Johnson, Douze & Jégou (2017) — Billion-scale similarity search with GPUs (FAISS), arXiv:1702.08734 — the approximate-search machinery behind §3.5: https://arxiv.org/abs/1702.08734
- Malkov & Yashunin (2016) — Efficient and robust approximate nearest neighbor search using HNSW graphs, arXiv:1603.09320 — the index type most vector databases default to: https://arxiv.org/abs/1603.09320
- Muennighoff et al. (2022) — MTEB: Massive Text Embedding Benchmark, arXiv:2210.07316 — how embedding models are compared; useful before picking one: https://arxiv.org/abs/2210.07316
- Companion notebook — swap in your own vectors, sweep the threshold, and see the false-abstain/false-answer trade directly.
