TL;DR — In agentic RAG a router decides, per query, which of several specialist retrievers to run — each with its own index, filters, and tools — and a supervisor merges what comes back. Two things make this different from generic multi-agent orchestration. First, routing errors are wildly asymmetric: measured below, dropping one genuinely needed specialist costs 64 points of answer quality, while running one unnecessary specialist costs 2 — a 29.5× difference. So route for recall, not precision. Second, over-routing is not free either, because every specialist's documents compete for the same context slots: fanning out to all 8 specialists scored 87% against 91% at the optimum of 5, as filler crowded essential documents out. Third, when specialists carry tools, the LLM will happily invent arguments the user never supplied — a literal grounding check on argument values blocked 5 of 5 fabricated tool calls here with 0 false blocks. The pattern taxonomy (supervisor/worker, debate, fan-out) is general and covered elsewhere; what is specific to RAG is that each branch runs a full retrieval pipeline, so width costs real money and real latency.
1. Simple explanation
Plain RAG has one pipeline: embed the query, search one index, stuff the results in a prompt. That works until your corpus stops being one homogeneous thing.
Once you have a legal collection and an engineering wiki and a customer-support archive, one index is wrong for all three. Each needs different filters, different chunk sizes, sometimes a different embedding model, and different tools attached. So you build specialists — a retriever per domain — and put a router in front to pick which ones to run.
The router reads the question and decides: this is a warranty question, run the support and legal specialists, skip the other six. Each chosen specialist runs its own retrieval, returns its best documents, and a supervisor merges them into one answer.
Analogy — a hospital triage nurse. A patient describes chest pain and numbness in one arm. The nurse does not send them to all fourteen departments — that would take days and cost a fortune. Nor do they send them to exactly one, because chest pain plus arm numbness might be cardiac or neurological, and being wrong is not recoverable in the time available. They send the patient to two or three plausible departments, deliberately over-including, because a specialist who turns out to be unnecessary costs an hour, and a specialist who was needed but skipped costs the diagnosis. That asymmetry — not the org chart — is what this article is about.
2. Diagram
"does the extended warranty
cover water damage?"
│
▼
┌──────────────────────┐
│ ROUTER │ reads the question,
│ ranks specialists │ ranks all 8, picks top-w
└──────────┬───────────┘
│
┌──────────────┬───────────┴───────┬──────────────┐
▼ ▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│ LEGAL │ │ SUPPORT │ │ BILLING │ │ (5 not │
│ index A │ │ index B │ │ index C │ │ chosen)│
│ +tools │ │ │ │ +calc │ └─────────┘
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │ each branch = a FULL
│ │ │ retrieval pipeline:
└─────────────┴───────┬───────────┘ embed + search + rerank
▼ (this is why width costs)
┌───────────────────┐
│ SUPERVISOR │ merge, dedupe, fit into
│ 6 context slots │ a fixed context budget
└─────────┬─────────┘
▼
answer
MEASURED (400 queries, 8 specialists, 6 context slots — §5):
width specialist recall answer quality retrievals
1 50% 50% 1
2 77% 77% 2
3 88% 86% 3
5 98% 91% <- best 5
8 100% 87% <- diluted 8
ONE ROUTING MISTAKE, from perfect routing:
drop a NEEDED specialist 100% -> 36% (-64)
add an EXTRA specialist 100% -> 97% ( -2)
^^^^
29.5x asymmetry -> route for RECALL
3. How it works
3.1 What "agentic" adds, and what it does not
The orchestration patterns here — supervisor/worker, parallel fan-out and fan-in, debate — are general multi-agent patterns, not RAG inventions. They are catalogued with a decision table in Multi-Agent Patterns, and partial-failure handling is covered in Agent Orchestration. This article does not repeat them.
What is specific to RAG is the cost structure. In a generic agent system a worker is an LLM call. Here a worker is an entire retrieval pipeline: embed, vector search, possibly a lexical search, merge, rerank, maybe an external API. Fanning out to five specialists is not five LLM calls — it is five pipelines, and the reranking step alone may be twenty LLM calls each. Width is expensive in a way that generic orchestration advice does not prepare you for.
The second RAG-specific constraint is the shared context budget. Every specialist returns documents into the same fixed prompt. Workers in a generic system produce independent outputs; specialists here compete for the same scarce slots, which is why more specialists can make the answer worse.
3.2 The router
The router maps a question to a subset of specialists. Three implementations, in increasing cost:
KEYWORD / REGEX fast, free, transparent, brittle. Good as a first pass
and as a floor: "if the query mentions a part number,
ALWAYS include the catalogue specialist."
EMBEDDING embed the query, compare against a centroid per
specialist. Cheap, handles paraphrase, no training.
LLM CLASSIFIER ask the model which domains apply and why. Handles
implication — "radiation before reconstruction" implies
wound-healing even though neither word appears.
Production routers usually combine them: a union of keyword hits and LLM suggestions, which is a recall-oriented design and, given §3.3, the right one.
Route on the structured form of the query where you have one. Slots extracted before retrieval carry exactly the signal a router needs, and they are already computed — see Query Transformation: Fixing the Question Before You Retrieve.
3.3 The asymmetry that determines everything
Consider the two errors a router can make.
Under-routing — failing to invoke a specialist the question needed. The evidence that specialist holds is now permanently absent. Nothing downstream can recover it: the reranker cannot rank a document it never received, and the supervisor will write a confident answer from an incomplete evidence base. This failure is silent.
Over-routing — invoking a specialist the question did not need. It returns some low-relevance documents, which compete for context slots and mostly lose to better ones. You paid for a pipeline you did not need.
Measured in §5, from perfect routing: dropping one needed specialist takes answer quality from 100% to 36%. Adding one unnecessary specialist takes it from 100% to 97%. Under-routing costs 29.5× what over-routing costs.
The design consequence is direct. Tune the router's threshold toward inclusion. When in doubt, include the specialist. Add deterministic floor rules that force certain specialists in whenever a trigger appears, regardless of what the model thinks. It is worth spending real money on over-routing to avoid a small increase in under-routing.
3.4 But width is not free either
Over-routing is cheap, not free, and the cost is not only money. Every specialist pours documents into a fixed context budget. Past some width, filler from unneeded specialists starts displacing essential documents from better ones.
The measured curve peaks at width 5 (91%) and declines to 87% at width 8, even though specialist recall reaches 100%. The router found everything it needed and the answer still got worse, because the context could not hold it.
This is the single most counter-intuitive result here: perfect routing recall is not the objective. The objective is the best answer within a fixed context budget, and those diverge once dilution sets in. Two mitigations, and you generally want both — a reranker across the merged pool so the best documents win the slots regardless of source (see Reranking: The Second Pass That Decides What the Model Sees), and a per-specialist cap so no single branch can flood the context.
3.5 Specialist tools, and the arguments they invent
Specialists often carry tools: a calculator, a lookup, a scoring function. Two rules keep them safe.
Intent gating — attach a tool only when the question's intent matches it. A dosage calculator should not be in scope for "what does the literature say about dosing?", because a tool the model can see is a tool the model may call. Not attaching it is strictly safer than attaching it and hoping.
Argument grounding — this is the important one, and it is under-appreciated. An LLM asked to call a calculator will supply arguments whether or not the user provided them. Ask "what's the recommended dose for an adult?" and the model will cheerfully call the dose calculator with weight_kg=70 — a population default it invented, presented as this patient's weight. The tool then returns a precise, authoritative-looking number computed from a fabricated input, and the precision of the output disguises the fabrication of the input.
The guard is simple: before executing, verify every argument value is traceable to the user's input. If it is not present, refuse the call and ask. In §5 a literal substring check blocks 5 of 5 fabricated calls with 0 false blocks. Real implementations need normalisation — "seventy kilos" should satisfy 70, and units vary — but the principle holds: a tool must never run on a value the model supplied from its own priors.
3.6 Bounding the loop
An agentic system can loop: retrieve, decide it is unsatisfied, re-route, retrieve again. Useful, and unbounded by default. Three cheap bounds: cap the supervisor's turns and finalise gracefully at the cap; make structuring tools once-only per query so the router cannot re-classify forever; and instantiate agents fresh per request so no history accumulates across queries. None of these improve the good case — they bound the bad one, which is what makes the system's cost predictable.
3.7 Where agentic RAG stops being worth it
If your corpus is homogeneous, specialists have nothing to specialise in and the router is pure overhead and latency. If your router is barely better than random, you get the sign-flip that appears throughout retrieval: a component weaker than what it replaces makes things worse — routing to a random 2 of 8 is worse than searching one combined index. And if queries almost always need the same specialists, hardcode that set and delete the router.
4. The math
4.1 The two error costs
Let q(S) be answer quality given the invoked specialist set S, and N the set the query actually needs. Measured from S = N:
under-route: q(N) - q(N minus one needed) = 1.00 - 0.36 = 0.64
over-route: q(N) - q(N plus one unneeded) = 1.00 - 0.97 = 0.02
asymmetry ratio = 0.64 / 0.02 = 29.5
Now put that in an expected-cost frame. If the router's probability of missing a needed specialist is p_miss and of adding a spurious one is p_extra:
E[loss] = p_miss * 0.64 + p_extra * 0.02
Trading one point of p_miss for even thirty points of p_extra is roughly break-even. That is what "route for recall" means quantitatively: a router operating point with 30% spurious inclusions and 1% misses beats one with 0% spurious and 2% misses.
4.2 Why quality peaks and then falls
Answer quality has two competing terms as width w grows:
coverage(w) probability the needed specialists were invoked — rises,
concave, saturating (98% by w=5, 100% by w=7)
dilution(w) essential documents displaced from a fixed C slots by
filler — rises roughly linearly with w
With w specialists each returning d documents into C slots, the pool is w·d and the fraction that survives is C/(w·d). Essential documents outrank filler on average but not always, so a growing filler pool steadily costs you a few of them:
w = 5: pool 15, slots 6 -> 40% survive
w = 8: pool 24, slots 6 -> 25% survive
Coverage has already saturated by w=5, so everything past it is dilution with no compensating gain. Hence the interior optimum at 5 and the decline to 8. The optimum sits where coverage stops rising, not where it reaches 1.0.
Raising C moves the peak right; a stricter per-specialist cap flattens the decline.
4.3 What the guard checks
For a proposed call with arguments a_1..a_n against user input Q:
execute iff for every i: value(a_i) is traceable to Q
"Traceable" is the whole design decision. Strictest is literal substring containment — what §5 implements. Real systems relax it to normalised numeric matching ("seventy" -> 70, unit conversion), span extraction, or an explicit provenance field the model must fill with the quoted source text.
Note the asymmetry here runs the opposite way to routing: a false block is recoverable (ask the user for the missing value), while a false pass produces a confident wrong number that nobody will question. Bias the guard toward blocking — the mirror image of biasing the router toward including.
5. Real code
Standard library only, deterministic. A simulates the router and context competition; B isolates the cost of a single routing mistake in each direction; C tests the argument-grounding guard on hand-written cases.
import random
SEED = 11
N_SPECIALISTS = 8
N_QUERIES = 400
CONTEXT_SLOTS = 6 # docs that fit in the final prompt
DOCS_PER_SPECIALIST = 3
ROUTER_SKILL = 2.0 # signal-to-noise of the router's ranking
def make_queries():
"""Each query genuinely needs 1-3 specialists; each holds 1 essential doc."""
rng = random.Random(SEED)
qs = []
for _ in range(N_QUERIES):
n_needed = rng.choice([1, 1, 2, 2, 3])
needed = set(rng.sample(range(N_SPECIALISTS), n_needed))
# router score per specialist: high for needed, plus noise
router = [ROUTER_SKILL * (1 if s in needed else 0) + rng.gauss(0, 1)
for s in range(N_SPECIALISTS)]
# relevance of each doc, drawn once so all strategies see the same docs
docs = {}
for s in range(N_SPECIALISTS):
items = []
for j in range(DOCS_PER_SPECIALIST):
essential = (s in needed and j == 0)
# essential docs score higher on average, but not always
items.append((essential, (2.0 if essential else 0.0) + rng.gauss(0, 1)))
docs[s] = items
qs.append({"needed": needed, "router": router, "docs": docs})
return qs
QUERIES = make_queries()
def run(width):
"""Route to the router's top `width` specialists. Returns
(answer quality, specialist recall, retrievals per query)."""
quality = recall = 0.0
for q in QUERIES:
chosen = sorted(range(N_SPECIALISTS), key=lambda s: -q["router"][s])[:width]
pool = [(score, ess) for s in chosen for ess, score in q["docs"][s]]
pool.sort(key=lambda x: -x[0])
kept = pool[:CONTEXT_SLOTS]
got = sum(1 for _, ess in kept if ess)
quality += got / len(q["needed"])
recall += len(set(chosen) & q["needed"]) / len(q["needed"])
n = len(QUERIES)
return quality / n, recall / n, width
def counterfactual():
"""Cost of ONE routing mistake in each direction, from the same baseline."""
base = under = over = 0.0
for q in QUERIES:
ranked = sorted(range(N_SPECIALISTS), key=lambda s: -q["router"][s])
def score(chosen):
pool = [(sc, e) for s in chosen for e, sc in q["docs"][s]]
pool.sort(key=lambda x: -x[0])
return sum(1 for _, e in pool[:CONTEXT_SLOTS] if e) / len(q["needed"])
perfect = list(q["needed"])
base += score(perfect)
# under-route: drop one specialist that was actually needed
under += score(perfect[:-1]) if len(perfect) > 1 else score([])
# over-route: add one specialist that was not needed
extra = next(s for s in ranked if s not in q["needed"])
over += score(perfect + [extra])
n = len(QUERIES)
return base / n, under / n, over / n
# ------------------------------------------------------- C. argument grounding
TOOL_CASES = [
# (question, proposed_args, args_are_real) -- real = present in the question
("Patient weighs 70 kg and is 180 cm tall, what is the dose?",
{"weight_kg": "70", "height_cm": "180"}, True),
("What is the recommended dose for an adult?",
{"weight_kg": "70", "height_cm": "175"}, False), # invented defaults
("Order 12 units of part 4471 for the Denver site.",
{"quantity": "12", "part": "4471"}, True),
("Order the usual amount of that part for Denver.",
{"quantity": "10", "part": "4471"}, False), # invented from context
("Burn covers the whole left arm and front torso in an adult.",
{"region": "left arm", "region2": "front torso"}, True),
("How do I estimate burn area?",
{"region": "left arm", "region2": "front torso"}, False),
("Interest rate is 4.5% over 30 years on 250000.",
{"rate": "4.5", "years": "30", "principal": "250000"}, True),
("What would my mortgage payment be?",
{"rate": "6.0", "years": "30", "principal": "400000"}, False),
("Compare a 15 year and a 30 year term.",
{"years": "15", "years2": "30"}, True),
("Is a shorter term better?",
{"years": "15", "years2": "30"}, False),
]
def grounded(question, args):
"""Every argument value must appear literally in the question."""
q = question.lower()
return all(str(v).lower() in q for v in args.values())
def guard_report():
blocked_bad = passed_bad = blocked_good = passed_good = 0
for question, args, real in TOOL_CASES:
ok = grounded(question, args)
if real:
passed_good += ok
blocked_good += not ok
else:
passed_bad += ok
blocked_bad += not ok
return blocked_bad, passed_bad, blocked_good, passed_good
print("A. FAN-OUT WIDTH -- how many specialists should the router invoke?")
print(f"{'width':>6} {'specialist recall':>18} {'answer quality':>15} "
f"{'retrievals':>11}")
print("-" * 54)
rows = {}
for w in range(1, N_SPECIALISTS + 1):
qual, rec, cost = run(w)
rows[w] = (qual, rec)
print(f"{w:>6} {rec:17.0%} {qual:14.0%} {cost:11}")
best_w = max(rows, key=lambda w: rows[w][0])
print(f"\nbest answer quality at width {best_w} ({rows[best_w][0]:.0%}); "
f"width 1 gives {rows[1][0]:.0%}, width 8 gives {rows[8][0]:.0%}")
print("\n\nB. THE ASYMMETRY -- one routing mistake, each direction")
base, under, over = counterfactual()
print(f" perfect routing {base:.0%}")
print(f" one NEEDED specialist off {under:.0%} ({under-base:+.0%} vs perfect)")
print(f" one EXTRA specialist on {over:.0%} ({over-base:+.0%} vs perfect)")
print(f"\n under-routing costs {(base-under)/max(base-over,1e-9):.1f}x "
f"what over-routing costs")
print("\n\nC. ARGUMENT GROUNDING GUARD")
bb, pb, bg, pg = guard_report()
print(f" fabricated calls blocked {bb}/{bb+pb}")
print(f" legitimate calls blocked {bg}/{bg+pg} (false blocks)")
for question, args, real in TOOL_CASES:
mark = "PASS" if grounded(question, args) else "BLOCK"
tag = "legit " if real else "INVENT"
print(f" [{tag}] {mark:5} {question[:52]}")
# claims made in the prose
assert rows[1][0] < rows[best_w][0], "routing to one specialist under-covers"
assert rows[8][0] < rows[best_w][0], "routing to everything dilutes context"
assert (base - under) > 3 * (base - over), "under-routing must dominate"
assert bb == bb + pb, "guard must block every fabricated call"
assert bg == 0, "guard must not block legitimate calls"
print("\nasserts passed")
# Output:
# A. FAN-OUT WIDTH -- how many specialists should the router invoke?
# width specialist recall answer quality retrievals
# ------------------------------------------------------
# 1 50% 50% 1
# 2 77% 77% 2
# 3 88% 86% 3
# 4 95% 90% 4
# 5 98% 91% 5
# 6 99% 91% 6
# 7 100% 89% 7
# 8 100% 87% 8
#
# best answer quality at width 5 (91%); width 1 gives 50%, width 8 gives 87%
#
#
# B. THE ASYMMETRY -- one routing mistake, each direction
# perfect routing 100%
# one NEEDED specialist off 36% (-64% vs perfect)
# one EXTRA specialist on 97% (-2% vs perfect)
#
# under-routing costs 29.5x what over-routing costs
#
#
# C. ARGUMENT GROUNDING GUARD
# fabricated calls blocked 5/5
# legitimate calls blocked 0/5 (false blocks)
# [legit ] PASS Patient weighs 70 kg and is 180 cm tall, what is the
# [INVENT] BLOCK What is the recommended dose for an adult?
# [legit ] PASS Order 12 units of part 4471 for the Denver site.
# [INVENT] BLOCK Order the usual amount of that part for Denver.
# [legit ] PASS Burn covers the whole left arm and front torso in an
# [INVENT] BLOCK How do I estimate burn area?
# [legit ] PASS Interest rate is 4.5% over 30 years on 250000.
# [INVENT] BLOCK What would my mortgage payment be?
# [legit ] PASS Compare a 15 year and a 30 year term.
# [INVENT] BLOCK Is a shorter term better?
#
# asserts passed
Look at the pairs in C. In each pair the tool and the arguments are identical — only the question changes. The guard needs no knowledge of medicine, procurement, or mortgages; it only asks whether the numbers came from the user or from the model. That is why such a crude check works at all, and why it belongs in the execution path rather than in the prompt.
6. Real-world example
A support-automation team split their single index into six specialists and added an LLM router. Offline, routing accuracy looked excellent: the router picked the correct single best specialist 88% of the time.
Answer quality dropped anyway.
The router had been built and evaluated as a classifier — one query, one label, optimise accuracy. But roughly a third of real questions genuinely spanned two domains ("I was charged for a repair that I think the warranty covers"), and a single-label router structurally cannot serve them. It confidently returned billing, the warranty documents were never retrieved, and the answer was a fluent, well-cited, wrong explanation of the charge. Those queries did not look like failures in any dashboard: they had citations, they had no errors, and users mostly did not come back to complain — they just escalated to a human, which showed up as a support-cost metric nobody had connected to search.
The fix was to stop treating routing as classification. The router returned the top-3 specialists above a low threshold instead of an argmax, with keyword floor rules forcing the legal specialist in whenever a warranty term appeared. Retrieval cost roughly doubled. Escalations fell by more than that, and the two changes together were still cheaper than the human handling they replaced.
The lesson generalises past routing: evaluating a component on its own metric (routing accuracy) instead of the system's outcome (answer quality) hides exactly the errors that matter most.
7. Interview questions companies actually ask
Q1 [easy] "What does agentic RAG add over plain RAG?"
A A router that picks WHICH retrievers to run per query, and specialists with
their own indexes, filters, and tools. It pays off when the corpus is
heterogeneous enough that no single index/chunking/filter setup serves all
queries. On a homogeneous corpus it's latency and cost with no benefit.
Q2 [easy] "Should the router pick one specialist or several?"
A Several, and deliberately over-include. Measured: dropping one needed
specialist costs 64 points of answer quality; adding one unnecessary one costs
2. That's a 29.5x asymmetry, so you should accept a lot of spurious inclusions
to avoid a few misses. Argmax routing is the wrong shape for the problem.
Q3 [medium] "Why is under-routing so much worse?"
A Because it's unrecoverable and silent. A specialist that never ran contributes
no documents, so the reranker can't rank them and the supervisor writes a
confident answer from an incomplete evidence base. Over-routing just adds
low-relevance documents that mostly lose the ranking -- you paid money, but the
answer survives.
Q4 [medium] "If over-routing is cheap, why not always run every specialist?"
A Because they share a fixed context budget. Measured: fanning out to all 8
scored 87% versus 91% at the optimum of 5, even though specialist recall hit
100%. Filler crowds essential documents out of the slots. The optimum is where
coverage STOPS RISING, not where it reaches 1.0.
Q5 [medium] "How do you stop a specialist's calculator running on made-up inputs?"
A Two layers. Intent-gate the tool so it isn't even attached unless the question
is the kind that needs it -- a visible tool is a callable tool. Then, before
execution, verify every argument value traces back to the user's input, and
refuse otherwise. Asked "what's the dose for an adult?" a model will supply
weight_kg=70 from its priors and the tool returns a precise number computed
from a fabricated input.
Q6 [medium] "Which way should the grounding guard err?"
A Toward blocking -- the opposite of the router. A false block is recoverable:
you ask the user for the missing value. A false pass produces an authoritative
number derived from an invented input, and the precision of the output hides
the fabrication of the input. Nobody audits a number that looks computed.
Q7 [hard] "Your router is 88% accurate but answers got worse. What happened?"
A Almost certainly it was built as a single-label classifier while a large
minority of real queries span two domains. Argmax serves those structurally
badly, and the failure is invisible -- cited, fluent, no errors thrown, users
silently escalate. Evaluate the router on end-to-end answer quality, not on
routing accuracy, and return a thresholded top-k rather than an argmax.
Q8 [hard] "How do you stop an agentic RAG loop running forever?"
A Cap supervisor turns with a graceful finalize at the cap; make structuring
tools once-only per query so classification can't repeat; instantiate agents
fresh per request so no history accumulates. None of these improve the good
case -- they bound the bad one, which is what makes cost predictable enough to
put in front of users.
8. When to use / tradeoffs
REACH FOR AGENTIC RAG WHEN:
+ the corpus is heterogeneous -- different domains need different
indexes, filters, chunk sizes, or embedding models
+ some queries need domain-specific TOOLS, not just retrieval
+ you can afford several retrieval pipelines per query
DON'T WHEN:
- one index serves everything (router is pure overhead)
- the router is barely better than random (sign flip: worse than one index)
- queries nearly always need the same specialists (hardcode them, delete
the router)
- the latency budget can't absorb the slowest branch
| Situation | Why it breaks | Use instead |
|---|---|---|
| Router as single-label classifier | Multi-domain queries structurally unserved; silent failure | Thresholded top-k + keyword floor rules |
| Tuning the router for precision | Under-routing costs 29.5× over-routing | Tune for recall; over-include |
| Fan out to everything | Context dilution — 87% at width 8 vs 91% at 5 | Stop where coverage saturates; cap per specialist |
| Tool visible on every query | A visible tool is a callable tool | Intent-gate attachment |
| Trusting model-supplied arguments | Invents plausible defaults; output precision hides it | Grounding check before execution |
| Evaluating router accuracy alone | Hides the errors that matter most | Measure end-to-end answer quality |
| Unbounded re-routing | Cost and latency have no ceiling | Turn cap, once-only tools, fresh agents |
Honest limits. The router here is a Gaussian-noise ranker with a tunable skill parameter, not a real classifier — real routers fail systematically (they consistently confuse two adjacent domains) rather than randomly, and systematic confusion is harder to fix by widening w because the missed specialist is missed consistently, on the same query type, every time. The 29.5× asymmetry depends on my assumption that each needed specialist holds exactly one irreplaceable document; where evidence is redundant across specialists, missing one is far less catastrophic and the ratio shrinks substantially. The width-5 optimum is an artefact of 8 specialists × 3 documents → 6 slots and will move with your context budget — run the sweep on your own numbers rather than adopting 5. Part C is 10 hand-written cases, chosen by me to be unambiguous, so 5/5 and 0/5 demonstrate the mechanism and say nothing about a real false-block rate; literal substring matching will reject "seventy kilograms" against 70 and any unit conversion, and a production guard needs normalisation that this one lacks. Finally, nothing here models latency, which for parallel fan-out is set by the slowest branch and is often what actually constrains width.
9. Summary + related articles
- Agentic RAG = a router choosing which specialist retrievers to run. The orchestration patterns are generic; the RAG-specific parts are that each branch is a full pipeline and all branches share one context budget.
- Routing errors are 29.5× asymmetric. Dropping a needed specialist cost 64 points; adding an unneeded one cost 2. Route for recall, over-include, add keyword floor rules.
- But width still has an optimum. Quality peaked at width 5 (91%) and fell to 87% at width 8 despite 100% specialist recall — context dilution. Optimise where coverage saturates, not where it hits 1.0.
- Intent-gate tools, because a tool the model can see is a tool it may call.
- Check argument grounding before execution. Models invent plausible defaults; a literal check blocked 5/5 fabrications with 0 false blocks. Bias this guard toward blocking — the opposite of the router.
- Bound the loop: turn caps, once-only structuring tools, fresh agents per request.
- Boundary: agentic RAG only pays on a heterogeneous corpus with a router meaningfully better than random. Otherwise one good index beats a badly-routed eight.
Related:
- Multi-Agent Patterns — the pattern taxonomy (supervisor/worker, debate, fan-out) that this article routes retrieval through rather than re-deriving
- Agent Orchestration — partial-failure handling and circuit breakers for the fan-out step
- Tool Use — tool definition and calling mechanics that intent gating sits on top of
- Reranking: The Second Pass That Decides What the Model Sees — the merge-side fix for context dilution: rank the pooled documents so the best win the slots regardless of which specialist found them
- Multi-Index RAG: Merging Several Retrievers Into One Answer — what to do with what comes back: dedup across specialists and fuse their incomparable scores
- Query Transformation: Fixing the Question Before You Retrieve — §3.4 on typed slots, which are the natural input to a router
Resources
- Asai et al., "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection", ICLR 2024 (arXiv:2310.11511) — the retrieve-decide-critique loop that §3.6 bounds.
- Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools", NeurIPS 2023 (arXiv:2302.04761) — how models learn to emit tool calls, and why they emit arguments regardless of evidence.
- Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models", ICLR 2023 (arXiv:2210.03629) — the interleaved reason/act loop underlying most agentic retrieval.
- Jeong et al., "Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity", NAACL 2024 (arXiv:2403.14403) — routing by query complexity; the closest published treatment of §3.2.
- Anthropic, "Building Effective Agents" — https://www.anthropic.com/research/building-effective-agents (argues for the simplest architecture that works, which is the §3.7 test).