TL;DR — Versioning a model is two obligations, not one: pin what your services call, and record what produced every output you keep. Most teams do neither for third-party models, because a hosted endpoint feels like infrastructure rather than a dependency — and then the provider retires it. In the audit below, five service references produced one already broken, one unpinned, and one retiring in 76 days, from a config nobody had read since it was written. The second obligation is the one that bites later: of three stored artifacts, only one could be reproduced — one because its model no longer exists, one because no version was ever recorded. A floating alias like
latestis the specific anti-pattern, since your output can change with no deployment, no commit and no alert. This stops being sufficient when reproducibility must survive the model itself, because no amount of pinning brings back a retired endpoint — at that point you need the outputs archived, not just the version number.
1. Simple explanation
A model your system depends on has a version, whether or not you wrote it down. If you host the weights yourself, that version is a file you control. If you call a provider's API, it is a string in their catalogue that they control — and they will eventually remove it.
The second case is the one that surprises people, because a hosted endpoint does not feel like a dependency. There is no lockfile entry, nothing in the package manifest, and no build step that fails when it changes. It is a string in a config file, so it inherits none of the machinery the industry built for exactly this problem. When the provider retires the model, the failure arrives in production, at their schedule, not yours.
There are two separate jobs here and teams routinely do the first while forgetting the second:
- Pin what you call. Name an exact version, not an alias, so the thing answering your requests today is the thing that answered them last week.
- Record what produced each output. Every stored result should carry the model and version that generated it, or you cannot later explain, defend, or reproduce it.
Analogy — a recipe that says "use the house stock." It works for years because the kitchen always has house stock. Then the supplier changes, or the kitchen stops making it, and the dish tastes different — or cannot be made at all. Nobody changed the recipe; the recipe never actually specified anything. Worse, the dishes you served last year were photographed and reviewed, and you now cannot say which stock they were made with, so you cannot reproduce the good ones or explain the bad ones. Pinning is writing the supplier and the product code into the recipe. Recording provenance is noting, on each plate that went out, which one was used.
2. Diagram
TWO OBLIGATIONS, USUALLY ONLY ONE REMEMBERED
(1) PIN WHAT YOU CALL (2) RECORD WHAT PRODUCED THE OUTPUT
config: stored artifact:
+---------------------------+ +----------------------------+
| model: vendor-a/latest X | | report-2026-05 |
| ^^^^^^ floating | | result: ... |
+---------------------------+ | model: (not recorded) X|
| model: vendor-a/text- | +----------------------------+
| large-v2 OK| | report-2026-04 |
+---------------------------+ | model: text-large-v1 OK |
+----------------------------+
protects TODAY's behaviour protects your ability to EXPLAIN
and REBUILD yesterday's output
THE PROVIDER'S CLOCK RUNS WHETHER YOU LOOK OR NOT (audited, §5)
active ----------> deprecated ----------> retired
announced endpoint gone
(a date you (calls fail)
must notice)
|
scoring-api --------> URGENT retires in 76 days
batch-scorer -------------------> BROKEN already failing
summariser ? FLOATING - not on the clock at all; it moves under you
search-index OK
eval-harness OK
REPRODUCIBILITY DEGRADES IN TWO DIFFERENT WAYS
version recorded + model alive -> YES, rebuildable
version recorded + model retired -> NO - pinning cannot resurrect it
no version recorded -> NO - and you cannot even say what to look for
^^^ this one is self-inflicted
3. How it works
3.1 A hosted model is a dependency without a lockfile
Package managers solved this for libraries: a manifest declares what you want, a lockfile records exactly what you got, and CI fails loudly when they disagree. A model referenced by a string in a config participates in none of that. Nothing resolves it, nothing records the resolution, and nothing notices when the string starts meaning something else.
The practical consequence is that the usual safety net — "the build will catch it" — does not exist. Detection has to be added deliberately: a check that reads your references, compares them against the provider's catalogue, and fails when one is deprecated or gone. That check is the subject of §5, and it is short enough that its absence is a choice rather than an engineering cost.
3.2 Aliases move; versions do not
Providers offer convenience aliases — latest, stable, or an unsuffixed family name — that resolve to whatever the current release is. They are attractive because you get improvements without doing anything, and that is exactly the problem: your system's behaviour changes with no deployment, no commit, and no entry in any log you keep.
The failure this produces is uniquely hard to debug because the usual first question, "what changed on our side?", has the answer "nothing", and it is true. Output quality shifts, a downstream parser starts failing on a new response format, or costs move, and the change is invisible in your history. Pin the exact version and take upgrades deliberately.
3.3 Deprecation is a schedule you are expected to read
Providers usually publish a lifecycle: a model is announced, becomes generally available, is later marked deprecated with a retirement date, and is eventually removed. The deprecation window is the interval in which migration is cheap and voluntary; after retirement it is neither.
Nobody reads changelogs reliably, so the window has to be surfaced by a check rather than by attention. Two thresholds are worth distinguishing — "retires eventually, plan it" and "retires inside the migration window, do it now" — because collapsing them produces either constant noise or a single alarm that arrives too late.
3.4 Record the version on the output, not just in the config
The config tells you what you call now. It says nothing about what produced the result you stored six months ago, and the config has been edited since.
Every persisted artifact — a score, a summary, an embedding, a report — should carry the model identifier and version that produced it. This is what makes it possible to explain a result to someone who challenges it, to compare two runs and know whether the model or the input changed, and to decide what has to be regenerated when you migrate. Without it, an evaluation comparing "before" and "after" is uninterpretable, because the before-set has no attributable producer.
The same argument applies to derived stores. An embedding index built with one model and queried with another returns confident nonsense, which is why the index-provenance discipline in Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For exists, and why the chunking settings deserve the same treatment as covered in Token-Aware Chunking and Reproducible Index Builds.
3.5 What the tooling does and does not solve
For models you train, a model registry — MLflow, DVC, Weights & Biases artifacts, or a cloud provider's equivalent — gives you versioned storage, stage transitions, and lineage back to the data and code that produced each version. This is genuinely valuable and worth adopting, and it addresses the first-party half of the problem well.
It does not address the third-party half. A registry records that your pipeline called vendor-a/text-large-v1; it has no influence over whether that endpoint still exists next quarter. Lineage tracking tells you what you did, not what your supplier will do. The provider-side check is a separate, small piece of work that no registry performs for you.
3.6 Where pinning stops helping
Pinning guarantees stability only while the pinned version is served. Once retired, the pin is a precise description of something unavailable — better than nothing, since you at least know what you lost, but it does not restore the capability or let you regenerate the output.
If reproducibility must outlive the provider's support window, the version number is insufficient and you need the outputs archived, or the weights held somewhere you control. This is the real argument for open-weight models in regulated or long-lived work, and it is a different decision from which model scores best today.
4. The math
4.1 The migration window
For a model deprecated with retirement date R, checked on date T, with an estimated migration effort of M days and a required safety buffer B:
slack = (R - T) - M - B
slack >= 0 -> migration fits comfortably
slack < 0 -> you are already late; the only levers are scope or buffer
A useful alert threshold is therefore not a fixed number of days but a function of your own migration cost:
warn_days = M + B
4.2 Worked example
Using the audit in §5, checked on T = 2026-06-01, with a migration estimated at 45 days and a 30-day buffer:
warn_days = M + B = 45 + 30 = 75 days
scoring-api -> vendor-a/text-large-v1, R = 2026-08-16
days remaining = 76
slack = 76 - 45 - 30 = +1 day -> URGENT, but feasible
batch-scorer -> vendor-a/text-small-v1, R = 2026-03-01
days remaining = -92
slack = -92 - 45 - 30 = -167 days -> already retired; calls fail now
The scoring-api case is the instructive one: one day of slack. It looks fine on a calendar — over two months away — and is in fact on the edge of being unrecoverable, which is why the threshold has to be derived from migration cost rather than picked as a round number. A default of "warn at 30 days" would have reported this as comfortable.
5. Real code
from datetime import date
TODAY = date(2026, 6, 1)
# What the provider currently offers. Providers publish this; almost nobody reads it.
CATALOGUE = {
"vendor-a/text-large-v2": {"status": "active", "retires": None},
"vendor-a/text-large-v1": {"status": "deprecated", "retires": date(2026, 8, 16)},
"vendor-a/text-small-v1": {"status": "retired", "retires": date(2026, 3, 1)},
"vendor-b/embed-v3": {"status": "active", "retires": None},
}
# What our services actually reference. Note the two floating aliases.
DEPLOYMENT = [
# service, reference, pinned?
("scoring-api", "vendor-a/text-large-v1", True),
("batch-scorer", "vendor-a/text-small-v1", True),
("summariser", "vendor-a/latest", False),
("search-index", "vendor-b/embed-v3", True),
("eval-harness", "vendor-a/text-large-v2", True),
]
WARN_DAYS = 120
def audit(dep, catalogue, today, warn_days):
status, entry = [], catalogue.get(dep[1])
service, ref, pinned = dep
if not pinned:
return "FLOATING", "alias resolves to whatever ships next; output can change with no deploy"
if entry is None:
return "UNKNOWN", "reference not in the provider catalogue at all"
if entry["status"] == "retired":
return "BROKEN", f"retired {entry['retires']} — calls fail now"
if entry["status"] == "deprecated":
days = (entry["retires"] - today).days
level = "URGENT" if days <= warn_days else "PLAN"
return level, f"retires {entry['retires']} ({days} days)"
return "OK", "active"
print(f"today = {TODAY}\n")
print(f"{'service':<15}{'reference':<26}{'verdict':<9}note")
print("-" * 92)
counts = {}
for dep in DEPLOYMENT:
verdict, note = audit(dep, CATALOGUE, TODAY, WARN_DAYS)
counts[verdict] = counts.get(verdict, 0) + 1
print(f"{dep[0]:<15}{dep[1]:<26}{verdict:<9}{note}")
print("\nsummary:", ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
# --- reproducibility: can we rebuild an output from six months ago? ---
ARTIFACTS = [
{"id": "report-2026-01", "model": "vendor-a/text-small-v1", "version_recorded": True},
{"id": "report-2026-04", "model": "vendor-a/text-large-v1", "version_recorded": True},
{"id": "report-2026-05", "model": None, "version_recorded": False},
]
print("\ncan each stored artifact be reproduced today?")
for art in ARTIFACTS:
if not art["version_recorded"]:
verdict = "NO - no model version was recorded; provenance is lost"
elif CATALOGUE[art["model"]]["status"] == "retired":
verdict = "NO - model retired; the exact output can never be regenerated"
else:
verdict = "YES - model still served"
print(f" {art['id']:<18}{verdict}")
assert counts.get("BROKEN") == 1, "one service should already be broken"
assert counts.get("FLOATING") == 1, "one reference should be unpinned"
assert counts.get("URGENT") == 1, "one deprecation should be inside the warning window"
print("\nasserts passed")
# Output:
# today = 2026-06-01
#
# service reference verdict note
# --------------------------------------------------------------------------------------------
# scoring-api vendor-a/text-large-v1 URGENT retires 2026-08-16 (76 days)
# batch-scorer vendor-a/text-small-v1 BROKEN retired 2026-03-01 — calls fail now
# summariser vendor-a/latest FLOATING alias resolves to whatever ships next; output can change with no deploy
# search-index vendor-b/embed-v3 OK active
# eval-harness vendor-a/text-large-v2 OK active
#
# summary: BROKEN=1, FLOATING=1, OK=2, URGENT=1
#
# can each stored artifact be reproduced today?
# report-2026-01 NO - model retired; the exact output can never be regenerated
# report-2026-04 YES - model still served
# report-2026-05 NO - no model version was recorded; provenance is lost
#
# asserts passed
Run in CI on a schedule with the catalogue fetched from the provider rather than hard-coded, this turns an outage into a ticket. The check is about thirty lines; the absence of it is what makes deprecations feel sudden.
6. Real-world example
A team ran a document-scoring service against a hosted model, pinned to an exact version, which they considered done. The pin was correct and it protected them from silent behavioural drift for almost a year.
The provider then announced that the model family was being retired, with roughly two months' notice, and removed it on schedule. Nobody on the team saw the announcement — it arrived by email to an address nobody monitored and in a changelog nobody subscribed to. The failure surfaced as authentication-shaped errors in production, which sent the first hour of debugging in the wrong direction entirely, because "the model no longer exists" is not a hypothesis anyone starts with.
The migration itself was quick: change the string, re-run the evaluation, ship. What was not quick was everything downstream of the missing second obligation. The stored scores from the previous months recorded the score and the timestamp but not the model version, so when the new model produced systematically different numbers, there was no way to separate "the new model is stricter" from "the inputs that month were harder." The comparison that would have justified the migration to stakeholders could not be constructed, and the team ended up re-scoring a sample by hand to establish a baseline they had been generating automatically all along.
The lesson they took was ordering. Pinning is the obvious half and the cheap half. Recording the version on every output is the half that determines whether you can say anything useful after the thing you pinned disappears — and by then it is too late to start.
7. Interview questions companies actually ask
Q1. Why is pointing at a latest alias a problem? Because your system's behaviour can change with no deployment, no commit, and nothing in your change history. When output quality shifts or a parser starts failing, the first diagnostic question — "what changed on our side?" — correctly returns "nothing", which sends debugging in the wrong direction. Aliases trade a small convenience for the loss of the property that makes systems debuggable: that behaviour changes only when you change something.
Q2. You pinned an exact model version. What can still go wrong? The provider can retire it. A pin protects against drift, not against removal, and it is precisely the well-pinned systems that fail abruptly, because nothing was gradually changing to warn them. It also does nothing for reproducibility once the model is gone — the pin then documents exactly what you can no longer run. Add a scheduled check against the provider's catalogue so deprecation arrives as a ticket rather than an outage.
Q3. What should be stored alongside a model's output? The model identifier and version, at minimum, and ideally the relevant generation parameters. Without it you cannot explain a result that is challenged, cannot tell whether a difference between two runs came from the model or the input, and cannot scope what needs regenerating during a migration. The config records what you call today; it has been edited since the output was produced.
Q4. Does a model registry like MLflow or DVC solve this? It solves the first-party half well — versioned storage, stage transitions, and lineage back to the data and code for models you train. It has no influence over a third party's lifecycle. A registry will faithfully record that you called a hosted model that no longer exists. The provider-side check is separate work that no registry does for you.
Q5. How would you choose the deprecation warning threshold? From your own migration cost rather than a round number: warn at roughly the estimated migration effort plus a safety buffer. In the worked example a model retiring in 76 days had only one day of slack against a 45-day migration and a 30-day buffer — a conventional 30-day alert would have called that comfortable and been wrong.
Q6. Your provider retires a model and you must reproduce last year's outputs exactly. What are your options? If the outputs themselves were archived, you serve those. If only the version was recorded, you cannot regenerate them — the pin identifies what is missing without restoring it. Going forward the options are archiving outputs rather than assuming regeneration, or running open weights on infrastructure you control, which is the substantive reason to prefer open weights for long-lived or regulated work.
8. When to use / tradeoffs
Invest in versioning discipline when:
- Any part of the system calls a model you do not host.
- Outputs are persisted and may later be audited, contested, or compared across time.
- More than one model is in use, since migrations then happen one service at a time and the fleet drifts.
- A derived store exists — an embedding index, a cache, a fine-tune — where the producing model must match the consuming one.
| Situation | Why it breaks | Use instead |
|---|---|---|
| Pin set once, never checked | Pinning stops drift but not retirement; the failure is abrupt and looks like an outage | A scheduled catalogue check in CI, thresholded on your own migration cost |
| Version recorded only in config | The config has changed since the output was written; provenance is not retrospective | Stamp the model and version onto each persisted artifact |
| Reproducibility required beyond the support window | No pin survives retirement of the endpoint | Archive outputs, or self-host open weights |
| Prototype, single model, nothing persisted | The machinery costs more than the failure would | Pin the version and move on; add the rest when outputs start being kept |
Honest limits. The catalogue in §5 is hard-coded, and the entire value of the check depends on fetching it from the provider — an interface that is not standardised, is sometimes only a documentation page, and occasionally announces retirements by email rather than by API, which is not automatable at all. The audit also validates a declared inventory: a model reference constructed at runtime, read from an environment variable, or buried in a notebook is invisible to it, and those are common. Deprecation dates move, usually later but not always, so slack computed today is an estimate rather than a guarantee. Recording a version makes an output attributable, not reproducible — many hosted endpoints are not deterministic even at a fixed version and fixed parameters, so the same input may not return the same output regardless of how carefully you pinned. And none of this addresses whether the replacement model is actually equivalent for your task, which requires an evaluation set you have to build separately.
9. Summary + related articles
- Versioning is two obligations: pin what you call, and record what produced each stored output. Teams routinely do the first and skip the second.
- A hosted model is a dependency with no lockfile — nothing resolves it, records it, or fails when it changes meaning.
- Floating aliases are the specific anti-pattern: behaviour changes with no deployment and no history, so debugging starts from a false premise.
- In the audit, five references yielded 1 broken, 1 unpinned, 1 retiring in 76 days — and that 76 days carried only 1 day of slack against a 45-day migration plus buffer.
- Derive the alert threshold from migration cost, not a round number.
- Of three stored artifacts, only one was reproducible: one model was retired, one had no version recorded. The second failure is self-inflicted.
- Registries handle the first-party half well and do nothing about a provider's lifecycle.
- Boundary condition: a pin is only good while the version is served. If reproducibility must outlive the provider's support window, archive the outputs or hold the weights yourself.
Related:
- LLM Observability — surfacing the signals a scheduled check produces, and where they should land
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — the same provenance argument for a derived store, and the silent failure it prevents
- Token-Aware Chunking and Reproducible Index Builds — recording build settings so two indexes can be compared
- Adopting Versioned Schema Migrations in a Database That Already Has Data — the same versioning discipline applied to schema
- RAG Cost Optimization: Find the Step That Runs Forty Times — model choice as a cost decision, the other axis of a migration
Resources
- Sculley et al., "Hidden Technical Debt in Machine Learning Systems," NeurIPS 2015 — the "changing anything changes everything" principle and undeclared consumers, which is what an unpinned model reference creates.
- Gebru et al., "Datasheets for Datasets," Communications of the ACM 64(12), 2021 — the documentation-as-artifact argument, applied to data; model cards extend the same idea to models.
- Mitchell et al., "Model Cards for Model Reporting," FAT* 2019 — what a model version should document about itself.
- MLflow Model Registry documentation — stage transitions and lineage for models you train.
- DVC — data and model versioning tied to Git history, for the first-party half of the problem.