TL;DR — Fine-tuning changes behaviour; retrieval supplies knowledge. Almost every "should we fine-tune?" argument confuses the two, and the cost case usually evaporates once you compare against the right baseline. Fine-tuning carries knowledge for $0 per call, which is its one genuine advantage — but keeping that knowledge current costs a training run per change, so at weekly updates it is $20,800/year against $252 for retrieval. It does beat pasting everything into an uncached prompt above roughly 123,000 calls/month — and that comparison is against the worst prompting strategy available. Against retrieval or a cached prefix at the same volume, it loses. So the honest rule is: fine-tune for format, style, and task shape; retrieve for facts. It stops being a cost question entirely when you need citations or the ability to abstain, because a fine-tuned model can do neither.
1. Simple explanation
Two things get called "teaching the model about our business", and they are not the same job.
One is knowledge: your prices, your policies, last quarter's numbers. Facts that exist somewhere, change on their own schedule, and that someone may need to see a source for.
The other is behaviour: always replying as a terse JSON object, adopting a house tone, handling a specialised task shape the model handles awkwardly by default.
Fine-tuning adjusts the model's weights, which is a good way to change behaviour and a poor way to install facts. Facts baked into weights can't be cited, can't be updated without another training run, and can't be distinguished by the model from things it merely believes.
Analogy — training a new hire versus giving them the handbook. You train them on how you do things here: the tone with customers, the format of a report, when to escalate. You do not train them to memorise this quarter's price list — you put the price list on the intranet, because it changes, because they should be able to point at it, and because when it changes you edit one page rather than re-running their induction. Fine-tuning is induction. Retrieval is the intranet. Confusing them is expensive in exactly the way the analogy suggests.
2. Diagram
THE ONE DISTINCTION THAT DECIDES IT
is the thing you want to change ...
KNOWLEDGE BEHAVIOUR
facts, prices, policies format, tone, task shape
changes on its own schedule changes when you decide
someone may ask for a source nobody asks for a source
│ │
▼ ▼
RETRIEVAL FINE-TUNING
COST OF CARRYING THE KNOWLEDGE (200,000 calls/month)
prompt, everything, uncached $54.00 /mo
retrieval (relevant slice) $21.00 /mo
prompt, everything, cached $13.50 /mo
fine-tuned (in the weights) $0.00 /mo ← its one real advantage
COST OF KEEPING IT CURRENT (per year)
facts change retrieval fine-tuning lag before live
once a year $0 $400 3 days
monthly $0 $4,800 3 days
weekly $0 $20,800 3 days
daily $0 $146,000 3 days
▲ ▲
edit a document, every change waits
live immediately for a training run
AND THE COMPARISON THAT USUALLY GETS RIGGED
"fine-tuning saves money above ~123,000 calls/month"
▲
...against an UNCACHED FULL PROMPT — the worst option.
against retrieval ($252/yr) fine-tuning LOSES
against a cached prefix ($162/yr) fine-tuning LOSES
compare against what you would actually ship.
3. How it works
3.1 What fine-tuning is good at
Genuine wins, all of them behavioural:
- Output format, when few-shot examples aren't reliable enough and constrained decoding isn't available.
- House style — a consistent voice across thousands of generations.
- A specialised task shape the base model handles clumsily: an unusual classification scheme, a domain-specific transformation, a structured extraction with idiosyncratic rules.
- Shrinking the prompt permanently. If behaviour lives in the weights, you stop paying for the instructions on every call. At very high volume this is real money.
- Making a small model behave like a large one on one narrow task, which is often the strongest economic case of all — better than the cost arithmetic in §4, because it changes which model you're paying for.
3.2 What it's bad at, and why
Facts. Three separate reasons, each sufficient on its own:
Updating costs a training run. Every change waits for data prep, training, evaluation, deployment. In practice nobody retrains for one price change, so the model stays wrong until the next batch — the real cost isn't the compute, it's the staleness you tolerate between runs.
No citations. A fine-tuned model cannot tell you where an answer came from, because there is no "where" — the fact is diffused across weights. If anyone needs to verify an answer, this alone rules it out.
No abstention. It cannot distinguish "I was trained on this" from "this is a plausible continuation", so it answers either way. Retrieval at least gives you a signal — nothing came back above threshold — that you can act on.
There's also catastrophic forgetting: training hard on a narrow task can degrade general capability. You get a model that's better at your thing and worse at everything adjacent, which surfaces later as odd failures on inputs nobody tested.
3.3 The cost case, and the baseline it usually hides
Fine-tuning's cost argument is that knowledge in weights costs nothing per call. True — and the comparison is almost always made against the worst alternative.
§4 measures it. Against an uncached full prompt, fine-tuning wins above roughly 123,000 calls/month. Against retrieval it loses; against a cached prefix it loses. Both of those are things you'd actually ship.
So when someone shows you a break-even, the first question is: break-even against what? If the baseline is "paste 900 tokens into every request, uncached", the analysis has picked an opponent that loses to everything.
3.4 The ladder
Work down it, and stop as soon as something works:
| step | cost | when to stop here |
|---|---|---|
| 1. Better prompt + consistent examples | ~free | format is now reliable |
| 2. Constrained decoding | ~free | shape must be guaranteed |
| 3. Retrieval | low | the gap was missing facts |
| 4. A bigger model | linear | the task was beyond the small one |
| 5. Fine-tune a small model | training run | behaviour is stable, volume is high |
Most teams that reach for step 5 have skipped steps 1–3. The diagnostic is whether you can articulate what the model does wrong in behavioural terms — if the answer is "it doesn't know X", you're on the wrong ladder entirely.
3.5 Prerequisites people underestimate
If you do fine-tune, you need three things first, and the second is where projects stall:
A scored evaluation set, or you cannot tell whether the run helped. Fine-tuning without a held-out score is not engineering.
Training data. Typically hundreds to thousands of examples of the behaviour you want, consistent with each other. Assembling and cleaning this is usually the bulk of the work and the reason fine-tuning projects run long.
A repeatable pipeline. You will do this more than once — a base-model update, a behaviour change, a data fix. A one-off manual run becomes technical debt the moment the base model version moves.
3.6 Where this framing stops applying
It assumes fine-tuning means parameter-efficient adaptation (LoRA and similar) of a general model — the common case. Full pretraining is a different economic universe. It also assumes the base model is capable of the task and merely inconsistent; fine-tuning does not add capability the model lacks. And it says nothing about privacy or residency constraints, which sometimes force self-hosting and fine-tuning for reasons that have nothing to do with cost or quality.
4. The math
4.1 The two costs
carrying cost = calls * extra_input_tokens * rate_in per period
currency cost = changes_per_period * cost_per_training_run
fine-tuning: carrying = 0, currency = expensive
retrieval: carrying = small, currency = 0
That inversion is the whole decision. Fine-tuning trades a per-call cost for a per-change cost, so it wins when calls are many and changes are few.
4.2 Break-even volume
fine-tuning beats an uncached prompt when
calls_per_month > train_cost / (prompt_tokens * rate_in * months)
4.3 Worked example
200,000 calls/month, 900 tokens of knowledge if pasted in, 350 if retrieved, $400 per training run, three days of lag.
Carrying the knowledge:
prompt, everything, uncached $ 54.00
prompt, everything, cached $ 13.50
retrieval (relevant slice) $ 21.00
fine-tuned (knowledge in weights) $ 0.00
Fine-tuning is free per call. That advantage is real — and note that a cached prefix is already cheaper than retrieval here, which surprises people.
Keeping it current for a year:
facts change retrieval fine-tuning lag before live
once a year $0 $400 3 days
monthly $0 $4,800 3 days
weekly $0 $20,800 3 days
daily $0 $146,000 3 days
Both together, at this volume:
facts change retrieval fine-tuning cheaper
once a year $252 $400 retrieval
monthly $252 $4,800 retrieval
weekly $252 $20,800 retrieval
daily $252 $146,000 retrieval
Retrieval wins every row, including static knowledge — a single training run already costs more than a whole year of retrieval overhead at this volume.
4.4 So where does fine-tuning's cost case exist?
Only against the worst baseline:
calls/month prompt-carry/yr 1 retrain/yr cheaper
200,000 $648 $400 fine-tuning
2,000,000 $6,480 $400 fine-tuning
20,000,000 $64,800 $400 fine-tuning
200,000,000 $648,000 $400 fine-tuning
break-even ~123,457 calls/month against an UNCACHED full prompt.
And immediately:
Against retrieval ($252/yr) fine-tuning still loses at $400/retrain, and against a cached prefix ($162/yr) it loses too.
Most "fine-tuning saves money" arguments compare against stuffing everything into an uncached prompt. Compare against the best alternative you would actually ship, and the cost case mostly vanishes — which is why the decision should be made on capability, not price.
4.5 What cost cannot decide
requirement prompt RAG fine-tune
supplies facts the model lacks yes yes poorly
cite the source of an answer no yes no
abstain when nothing is found no yes no
change ONE fact in minutes yes yes no
teach a house output FORMAT ok no yes
teach a specialised TASK shape weak no yes
shrink the prompt permanently no no yes
The rows fine-tuning wins are all behaviour. The rows retrieval wins are the ones auditors and users ask about.
5. Real code
"""Prompting vs retrieval vs fine-tuning: where each one actually wins."""
# Illustrative. Substitute your own measured figures before deciding anything.
TRAIN_USD = 400.0 # one fine-tuning run: data prep + compute + eval
TRAIN_DAYS = 3.0 # elapsed time before the change is live
RATE_IN = 0.30 / 1e6 # $ per input token
CALLS_PER_MONTH = 200_000
FACTS_TOKENS = 900 # the knowledge, if you paste it into every prompt
RETRIEVED_TOKENS = 350 # the relevant slice, if you retrieve instead
CACHE_DISCOUNT = 0.25 # static prefix billed at this fraction
def monthly(extra_in_tokens: float, cached: bool = False) -> float:
rate = RATE_IN * (CACHE_DISCOUNT if cached else 1.0)
return CALLS_PER_MONTH * extra_in_tokens * rate
print("COST OF CARRYING THE KNOWLEDGE, per month at "
f"{CALLS_PER_MONTH:,} calls")
rows = [
("prompt, everything, uncached", monthly(FACTS_TOKENS)),
("prompt, everything, cached", monthly(FACTS_TOKENS, cached=True)),
("retrieval (relevant slice)", monthly(RETRIEVED_TOKENS)),
("fine-tuned (knowledge in weights)", 0.0),
]
for label, cost in rows:
print(f" {label:<36} ${cost:>8.2f}")
print(" ...fine-tuning carries the knowledge for $0 per call. That is its"
"\n one genuine advantage, and it is real.")
print("\nBUT KNOWLEDGE CHANGES. Cost of KEEPING IT CURRENT for a year:")
print(f" {'facts change':<16} {'retrieval':>12} {'fine-tuning':>14} {'lag before live':>17}")
for label, changes_per_year in (("once a year", 1), ("monthly", 12),
("weekly", 52), ("daily", 365)):
ft = changes_per_year * TRAIN_USD
print(f" {label:<16} {'$0':>12} {f'${ft:,.0f}':>14} {f'{TRAIN_DAYS:.0f} days':>17}")
print(" (retrieval updates by editing a document: no run, no lag)")
print("\nTOTAL YEAR ONE at this volume — carrying cost + keeping it current")
print(f" {'facts change':<16} {'retrieval':>12} {'fine-tuning':>14} cheaper")
crossovers = {}
for label, n in (("once a year", 1), ("monthly", 12), ("weekly", 52), ("daily", 365)):
rag = monthly(RETRIEVED_TOKENS) * 12
ft = n * TRAIN_USD
crossovers[label] = (rag, ft)
print(f" {label:<16} {f'${rag:,.0f}':>12} {f'${ft:,.0f}':>14} "
f"{'fine-tuning' if ft < rag else 'retrieval'}")
print(" -> retrieval wins EVERY row. One training run already costs more than")
print(" a whole year of retrieval overhead at this volume.")
print("\nSO WHERE DOES FINE-TUNING'S COST CASE EXIST? Only at high volume")
print("with a LARGE UNCACHED prompt, where carrying the knowledge dominates:")
print(f" {'calls/month':>12} {'prompt-carry/yr':>17} {'1 retrain/yr':>14} cheaper")
for calls in (200_000, 2_000_000, 20_000_000, 200_000_000):
carry = calls * FACTS_TOKENS * RATE_IN * 12
print(f" {calls:>12,} {f'${carry:,.0f}':>17} {f'${TRAIN_USD:,.0f}':>14} "
f"{'fine-tuning' if TRAIN_USD < carry else 'prompt'}")
breakeven_calls = TRAIN_USD / (FACTS_TOKENS * RATE_IN * 12)
rag_year = monthly(RETRIEVED_TOKENS) * 12
print(f"\n break-even ~{breakeven_calls:,.0f} calls/month against an UNCACHED full prompt.")
print("\n But notice what that comparison is against: the WORST prompting strategy.")
print(f" Against retrieval (${rag_year:,.0f}/yr) fine-tuning still loses at "
f"${TRAIN_USD:,.0f}/retrain,")
print(f" and against a CACHED prefix (${monthly(FACTS_TOKENS, cached=True)*12:,.0f}/yr) "
"it loses too.")
print("\n -> Most 'fine-tuning saves money' arguments compare against stuffing")
print(" everything into an uncached prompt. Compare against the BEST")
print(" alternative you would actually ship, and the cost case mostly vanishes.")
print("\nWHAT COST CANNOT DECIDE")
caps = [
("supplies facts the model lacks", "yes", "yes", "poorly"),
("cite the source of an answer", "no", "yes", "no"),
("abstain when nothing is found", "no", "yes", "no"),
("change ONE fact in minutes", "yes", "yes", "no"),
("teach a house output FORMAT", "ok", "no", "yes"),
("teach a specialised TASK shape", "weak", "no", "yes"),
("shrink the prompt permanently", "no", "no", "yes"),
]
print(f" {'requirement':<34} {'prompt':>8} {'RAG':>6} {'fine-tune':>10}")
for r in caps:
print(f" {r[0]:<34} {r[1]:>8} {r[2]:>6} {r[3]:>10}")
print("\n The rows fine-tuning wins are about BEHAVIOUR, not knowledge.")
print(" The rows RAG wins are the ones auditors and users ask about.")
# Fine-tuning is cheap to carry and expensive to keep current.
assert rows[3][1] == 0.0
assert monthly(RETRIEVED_TOKENS) < monthly(FACTS_TOKENS)
# Caching narrows the prompt-vs-retrieval gap a lot.
assert monthly(FACTS_TOKENS, cached=True) < monthly(RETRIEVED_TOKENS)
# At this volume retrieval wins on cost at EVERY change rate, including static.
assert all(ft > rag for rag, ft in crossovers.values())
# Fine-tuning DOES beat an uncached full prompt above the break-even volume ...
assert breakeven_calls < CALLS_PER_MONTH
assert TRAIN_USD < CALLS_PER_MONTH * FACTS_TOKENS * RATE_IN * 12
# ... and still loses to both retrieval and a cached prefix at the same volume.
assert TRAIN_USD > monthly(RETRIEVED_TOKENS) * 12
assert TRAIN_USD > monthly(FACTS_TOKENS, cached=True) * 12
print("\nall assertions passed")
# Output:
# COST OF CARRYING THE KNOWLEDGE, per month at 200,000 calls
# prompt, everything, uncached $ 54.00
# prompt, everything, cached $ 13.50
# retrieval (relevant slice) $ 21.00
# fine-tuned (knowledge in weights) $ 0.00
# ...fine-tuning carries the knowledge for $0 per call. That is its
# one genuine advantage, and it is real.
#
# BUT KNOWLEDGE CHANGES. Cost of KEEPING IT CURRENT for a year:
# facts change retrieval fine-tuning lag before live
# once a year $0 $400 3 days
# monthly $0 $4,800 3 days
# weekly $0 $20,800 3 days
# daily $0 $146,000 3 days
# (retrieval updates by editing a document: no run, no lag)
#
# TOTAL YEAR ONE at this volume — carrying cost + keeping it current
# facts change retrieval fine-tuning cheaper
# once a year $252 $400 retrieval
# monthly $252 $4,800 retrieval
# weekly $252 $20,800 retrieval
# daily $252 $146,000 retrieval
# -> retrieval wins EVERY row. One training run already costs more than
# a whole year of retrieval overhead at this volume.
#
# SO WHERE DOES FINE-TUNING'S COST CASE EXIST? Only at high volume
# with a LARGE UNCACHED prompt, where carrying the knowledge dominates:
# calls/month prompt-carry/yr 1 retrain/yr cheaper
# 200,000 $648 $400 fine-tuning
# 2,000,000 $6,480 $400 fine-tuning
# 20,000,000 $64,800 $400 fine-tuning
# 200,000,000 $648,000 $400 fine-tuning
#
# break-even ~123,457 calls/month against an UNCACHED full prompt.
#
# But notice what that comparison is against: the WORST prompting strategy.
# Against retrieval ($252/yr) fine-tuning still loses at $400/retrain,
# and against a CACHED prefix ($162/yr) it loses too.
#
# -> Most 'fine-tuning saves money' arguments compare against stuffing
# everything into an uncached prompt. Compare against the BEST
# alternative you would actually ship, and the cost case mostly vanishes.
#
# WHAT COST CANNOT DECIDE
# requirement prompt RAG fine-tune
# supplies facts the model lacks yes yes poorly
# cite the source of an answer no yes no
# abstain when nothing is found no yes no
# change ONE fact in minutes yes yes no
# teach a house output FORMAT ok no yes
# teach a specialised TASK shape weak no yes
# shrink the prompt permanently no no yes
#
# The rows fine-tuning wins are about BEHAVIOUR, not knowledge.
# The rows RAG wins are the ones auditors and users ask about.
#
# all assertions passed
Every constant is illustrative and every one of them varies by an order of magnitude across real setups. Substitute your own measured figures — the structure of the comparison is the transferable part, not the numbers.
6. Real-world example
A team wanted their assistant to know their product catalogue — a few thousand items with prices, availability, and compatibility notes. The proposal was to fine-tune, on the reasoning that the catalogue was "their data" and the model should "learn the business".
They built a training set from the catalogue, ran the fine-tune, and it demoed well: asked about a product, the model produced accurate specifications in the house tone.
Three problems appeared over the following weeks.
Prices changed weekly, and each change needed another run. Nobody would retrain for one price, so updates were batched monthly and the model was confidently wrong for up to four weeks at a time — presented in the same fluent tone as the correct answers.
Then support asked where a spec had come from, and there was no answer to give. The fact was diffused across weights with no source to point at.
And for discontinued products the model produced plausible specifications for items that no longer existed, because it had no way to distinguish "I learned this" from "this is a likely continuation".
They rebuilt with retrieval over the catalogue. Updates went live on save, every answer cited a product record, and discontinued items returned nothing rather than a fabrication.
They did keep a small fine-tune — for the output format, because they needed a specific structured shape and few-shot wasn't reliable enough at their volume. That is the correct division: weights for behaviour, retrieval for facts. The instructive part is that both were needed, and that starting with "let's fine-tune on our data" got the split backwards.
7. Interview questions companies actually ask
Q1. When would you fine-tune rather than use RAG? When the thing you want to change is behaviour, not knowledge: a consistent output format, a house style, a task shape the base model handles clumsily. Fine-tuning is a poor way to install facts — you can't update them without a training run, can't cite them, and can't abstain. The one-line rule is fine-tune for behaviour, retrieve for facts, and they compose fine together.
Q2. Doesn't fine-tuning save money by shrinking the prompt? Per call, yes — knowledge in the weights costs nothing. But keeping it current costs a training run per change, so at weekly updates it's $20,800/year against $252 for retrieval in the worked example. And the break-even people quote is usually measured against an uncached full prompt, which is the worst option available. Against retrieval or a cached prefix, the cost case mostly disappears.
Q3. What do you need before you can fine-tune responsibly? A scored evaluation set, or you cannot tell whether the run helped. Training data — typically hundreds to thousands of consistent examples, which is usually the bulk of the work. And a repeatable pipeline, because you'll do it again when the base model updates. Teams stall on the second one.
Q4. What's catastrophic forgetting? Training hard on a narrow task degrades general capability. You end up with a model better at your thing and worse at things adjacent to it, which surfaces later as odd failures on inputs nobody thought to test. It's an argument for parameter-efficient methods, a conservative learning rate, and keeping a general held-out set in your evaluation.
Q5. Walk me through the ladder before fine-tuning. Better prompt with consistent examples; then constrained decoding if the shape must be guaranteed; then retrieval if the gap is missing facts; then a bigger model if the task exceeded the small one; then fine-tune a small model if behaviour is stable and volume is high. Most teams reaching for the last step skipped the first three. The diagnostic: if you can't state the problem in behavioural terms, you're on the wrong ladder.
Q6. Why can't a fine-tuned model cite its sources? Because there is no source. The fact is distributed across weights rather than stored anywhere retrievable, so there's nothing to point at. For the same reason it can't distinguish "I was trained on this" from "this is a plausible continuation" — which is why it fabricates confidently about things it half-learned, and why regulated use cases usually rule it out on that basis alone.
Q7. What's the strongest economic case for fine-tuning? Not shrinking the prompt — making a small model behave like a large one on one narrow task. That changes which model you're paying for on every call, which dominates the token arithmetic entirely. It needs stable behaviour, high volume, and a scored set to prove the small model actually reached the bar.
8. When to use / tradeoffs
Fine-tune when:
- The gap is behaviour: format, style, task shape
- The behaviour is stable — it isn't changing monthly
- Volume is high enough that per-call savings matter
- You have a scored evaluation set and consistent training data
- You want a small model to do one narrow job as well as a large one
Don't fine-tune when:
- The gap is facts → retrieval
- Anyone needs a citation → retrieval
- The knowledge changes → retrieval
- You haven't tried prompting, constrained decoding, or retrieval yet
- You have no evaluation set
| Situation | Why it breaks | Do this instead |
|---|---|---|
| "Teach it our catalogue" | Facts in weights: stale, uncitable, unabstainable | Retrieval |
| "It'll save prompt tokens" | Break-even measured against the worst baseline | Compare with cached prefix + retrieval |
| Facts change weekly | A training run per change, plus lag | Retrieval |
| Regulated, needs sources | No source exists to cite | Retrieval |
| No evaluation set | Can't tell if the run helped | Build the set first |
| Skipped prompting and RAG | Expensive fix for a cheap problem | Work down the ladder |
| Trained hard on one narrow task | Catastrophic forgetting | PEFT, low LR, general held-out set |
| Base model can't do the task at all | Fine-tuning doesn't add capability | Bigger model, or rethink |
Honest limits. Every number in §4 is illustrative and each varies by an order of magnitude in practice — training cost depends enormously on method and data volume, token rates change, and "3 days of lag" is organisational rather than technical. The model also treats a training run as a fixed price when the real cost is usually the data work, which is lumpy and mostly paid once. It ignores serving costs for a fine-tuned model, which for a self-hosted deployment can dominate everything here. And it deliberately frames the decision as knowledge-versus-behaviour, which is the right first cut and a simplification: fine-tuning does install some facts, just unreliably and unverifiably, and there are narrow domains where that is genuinely the best available option. Treat this as a framework for structuring the argument, not a verdict.
9. Summary + related articles
- Fine-tuning changes behaviour. Retrieval supplies knowledge. Nearly every bad decision here confuses the two.
- Fine-tuning carries knowledge for $0 per call — its one genuine advantage.
- But keeping it current costs a run per change: $20,800/year at weekly updates, against $252 for retrieval.
- At 200k calls/month, retrieval wins at every change rate, including static knowledge — one run already exceeds a year of retrieval overhead.
- Fine-tuning beats an uncached full prompt above ~123,000 calls/month — and that's the worst baseline. Against retrieval or a cached prefix, it loses.
- So ask break-even against what? Compare with what you'd actually ship.
- It cannot cite and cannot abstain. For regulated work that decides it before cost is discussed.
- Beware catastrophic forgetting: better at your task, worse at adjacent ones, discovered later.
- The ladder: prompt → constrained decoding → retrieval → bigger model → fine-tune. Most people skip to the end.
- The strongest economic case isn't a shorter prompt — it's making a small model do one narrow job well.
Related:
- What RAG Is and When to Use It — the alternative this article keeps pointing at
- Prompt Fundamentals — step 1 of the ladder
- Structured Outputs — step 2, and often the real answer to "we need consistent format"
- Fine-tuning Techniques — LoRA, QLoRA, and how it's actually done
- When to Fine-tune: data preparation — the work that dominates the timeline
- Model Evaluation — the scored set §3.5 insists on
- Model Routing Patterns — cheap default, expensive fallback, without training anything
- Reasoning Budgets: When Thinking Tokens Are Waste — prefix caching, which changes the cost comparison
- Cost-Latency Tradeoffs — where this sits among the other levers
Resources
- Hu et al. (2021) — LoRA: Low-Rank Adaptation of Large Language Models, arXiv:2106.09685 — the method that made fine-tuning cheap enough for this decision to be close: https://arxiv.org/abs/2106.09685
- Dettmers et al. (2023) — QLoRA: Efficient Finetuning of Quantized LLMs, arXiv:2305.14314 — cheaper still, and what most practical fine-tuning uses now: https://arxiv.org/abs/2305.14314
- Ovadia et al. (2023) — Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, arXiv:2312.05934 — direct empirical comparison on knowledge tasks; supports §3.2: https://arxiv.org/abs/2312.05934
- Luo et al. (2023) — An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning, arXiv:2308.08747 — how much general capability you lose: https://arxiv.org/abs/2308.08747
- Zhou et al. (2023) — LIMA: Less Is More for Alignment, arXiv:2305.11206 — evidence that a small, very clean dataset can outperform a large messy one, which is the §3.5 point about data quality: https://arxiv.org/abs/2305.11206
- Provider fine-tuning documentation is the authority on cost, supported methods, and minimum dataset sizes; all three change frequently.