TL;DR — A prompt embedded as a string literal inside application code can't be edited, reviewed, or swapped without a code change and a redeploy, and it silently duplicates the moment two call sites need "almost the same" prompt. The fix is treating prompts as files: static system prompts loaded and cached once, user-facing templates rendered fresh per call with real variable injection, and provider-specific variants selected by a lookup rather than an if/else scattered through the codebase. In the harness below, a static prompt is read from disk exactly once no matter how many times it's requested — proving the cache works — while a template rendered with different variables produces genuinely different output every time, proving the two are correctly handled differently. This design stops paying for itself once there's only one prompt, used in one place, that never changes.
1. Simple explanation
The easiest way to send an LLM a system prompt is to write it as a string directly in the function that calls the API. This works until the second call site needs almost the same prompt with one clause changed, or a non-engineer needs to tweak the wording without touching code, or the prompt needs to differ depending on which provider is handling the request — at which point the string-in-code approach means either duplicating the prompt with small variations scattered across the codebase, or threading conditional logic through every call site that needs it.
Analogy — recipe cards versus a chef who memorizes everything. A kitchen where every recipe lives only in one chef's head works fine with one chef and a handful of dishes. The moment a second chef needs to cook the same dish, or a dish needs a slightly different version for a dietary restriction, or the head chef wants to review and improve a recipe without pulling someone off the line to recite it, the "memorized" approach breaks down. A kitchen with recipe cards — written down, filed by dish, with clearly marked variations — lets anyone find, read, revise, or swap a recipe without interrupting whoever's cooking. Prompt files are the recipe cards; the code that calls the LLM is the chef who just follows whichever card is handed over.
2. Diagram
STRING-IN-CODE PROMPTS-AS-FILES
def call_llm(): prompts/
system = ( reviewer_system_default.txt
"You are a reviewer..." reviewer_system_compact.txt
) review_request.jinja
# ... duplicated with tweaks
# in every other call site
PromptManager
.get(key) -> static prompt,
read once, CACHED
.render(key,
variables) -> template,
RE-RENDERED
every call
PROVIDER-AWARE SELECTION (one lookup, not scattered conditionals)
provider system prompt file
---------------------------------------------
default-tier -> reviewer_system_default.txt
small-context -> reviewer_system_compact.txt
MEASURED (harness in §5):
static prompt file reads after 4 total requests for the
same key: 2 (one per distinct key,
repeats served from cache)
template renders with different variables: 2 genuinely different
outputs from 2 render() calls
3. How it works
3.1 Two different things need two different handling strategies
A system prompt that never changes for a given configuration ("you are a meticulous reviewer, cite line numbers, never invent issues") is a static asset: load it once, keep it, reuse it. A user-facing prompt built from per-request data (the actual code diff, the specific question, retrieved context) is a template: it has to be rendered fresh every time because the data plugged into it is different every time. Treating both the same way — either re-reading a static prompt from disk on every call, or trying to cache a rendered template's output — is a category error in one direction or the other: the first wastes I/O on something that never changes, the second silently serves stale, wrong content to a request whose actual variables have changed.
3.2 A registry, not a conditional
Selecting which prompt applies — for a given provider, tier, or mode — is naturally expressed as a lookup: a dictionary mapping a logical key (which provider is active) to a concrete file. This keeps the selection logic in one place, visible and auditable, instead of scattered as if provider == "x": ... elif provider == "y": ... blocks repeated at every call site that needs to make the same decision. Adding a new provider variant becomes adding one entry to the registry, not finding and updating every conditional that made the same choice independently.
3.3 Caching a static prompt is safe; caching a rendered template is not
A static prompt's content is fully determined by its file — reading it twice returns the identical string, so caching the read (not re-hitting disk on every call) is a pure win with no correctness risk. A template's rendered output depends on the variables passed in for that specific call; caching the rendered result would mean the second call, with different variables, either gets served the first call's stale output or requires the cache key to somehow encode every variable combination that could ever be passed — which is just re-implementing "render it again" with extra steps. The correct design caches the read of the template file (which doesn't change) but never the rendered result (which is supposed to change every time).
Where this stops working: treating prompts as files pays off when a prompt is reused, needs review, needs a provider-specific variant, or changes often enough that a code deploy for a wording tweak is genuinely disruptive. For a single, permanent, never-varying prompt used in exactly one place, the whole apparatus — a manager class, a cache, a registry — is more machinery than the problem needs; a string constant is a perfectly reasonable choice for a system that will never grow past that one case, and the pattern shouldn't be applied preemptively for a variation that doesn't yet exist.
4. The math
4.1 The actual saving is in I/O and duplication, not computation
There's no formula here worth deriving — the benefit of caching a static read is simply: file reads without caching = number of requests, versus file reads with caching = number of distinct keys ever requested, and the gap between those two grows without bound as request volume grows while the number of distinct prompts stays fixed. In §5's run, four total requests for provider system prompts (two providers, one of them requested four times total) produced exactly two file reads — one per distinct key — regardless of how many times each key was actually requested afterward.
4.2 Worked comparison from the actual run
requests for provider system prompts (2 distinct providers): 4 total calls
file reads for get() after all 4 calls: 2
-> ratio of reads to requests: 2 / 4 = 0.5, and this ratio keeps
falling as request volume grows, since the numerator (distinct
keys) is fixed and the denominator (requests) is not
5. Real code
import tempfile
from pathlib import Path
from functools import lru_cache
import jinja2
def build_prompt_directory():
"""Sets up prompts as separate files on disk, the way a real project
does -- never as strings embedded inside application code."""
tmp = Path(tempfile.mkdtemp())
(tmp / "reviewer_system_default.txt").write_text(
"You are a meticulous code reviewer. Be specific and cite line "
"numbers. Never invent a bug that isn't present in the diff."
)
(tmp / "reviewer_system_compact.txt").write_text(
"You are a code reviewer. Provider has a small context budget: be "
"terse, list only the top three issues, no preamble."
)
(tmp / "review_request.jinja").write_text(
"Review this {{ language }} change for {{ focus }}.\n\n"
"{% if prior_findings %}"
"Known recurring issues in this codebase: {{ prior_findings }}\n\n"
"{% endif %}"
"Diff:\n{{ diff }}"
)
return tmp
PROVIDER_SYSTEM_PROMPT = {
"default-tier": "reviewer_system_default",
"small-context-tier": "reviewer_system_compact",
}
class PromptManager:
"""get(key) -> static system prompt, loaded once, cached.
render(key, vars) -> Jinja2 template, re-rendered every call."""
def __init__(self, prompt_dir):
self.prompt_dir = Path(prompt_dir)
self.read_count = {"get": 0, "render": 0}
self._env = jinja2.Environment(
loader=jinja2.FileSystemLoader(str(self.prompt_dir)),
autoescape=False,
trim_blocks=True,
)
@lru_cache(maxsize=None)
def _read_static(self, key):
self.read_count["get"] += 1
return (self.prompt_dir / f"{key}.txt").read_text()
def get(self, key):
return self._read_static(key)
def render(self, key, variables):
self.read_count["render"] += 1
template = self._env.get_template(f"{key}.jinja")
return template.render(**variables)
def system_prompt_for_provider(self, provider):
key = PROVIDER_SYSTEM_PROMPT[provider]
return self.get(key)
prompt_dir = build_prompt_directory()
pm = PromptManager(prompt_dir)
print("=== provider-aware static prompt selection ===")
for provider in PROVIDER_SYSTEM_PROMPT:
text = pm.system_prompt_for_provider(provider)
print(f"{provider:20} -> {text[:50]}...")
print(f"\nunderlying file reads for get(): {pm.read_count['get']} "
f"(one per distinct key, even if requested repeatedly)")
for _ in range(3):
pm.system_prompt_for_provider("default-tier")
print(f"after 3 more repeat requests, file reads for get(): "
f"{pm.read_count['get']} (unchanged -- served from cache)")
print("\n=== template rendering, different variables each call ===")
rendered_1 = pm.render("review_request", {
"language": "Python",
"focus": "security",
"prior_findings": "",
"diff": "+ eval(user_input)",
})
rendered_2 = pm.render("review_request", {
"language": "Python",
"focus": "security",
"prior_findings": "unsanitized eval() calls (flagged 3 times before)",
"diff": "+ eval(user_input)",
})
print("--- call 1 (no prior findings) ---")
print(rendered_1)
print("\n--- call 2 (same diff, WITH prior findings) ---")
print(rendered_2)
assert rendered_1 != rendered_2
assert "Known recurring issues" not in rendered_1
assert "Known recurring issues" in rendered_2
print(f"\nasserts passed: identical diff, different variables, "
f"genuinely different rendered output")
print(f"template reads for render(): {pm.read_count['render']} "
f"(not cached -- variables differ per call, so caching the result "
f"would be wrong)")
# Output:
# === provider-aware static prompt selection ===
# default-tier -> You are a meticulous code reviewer. Be specific an...
# small-context-tier -> You are a code reviewer. Provider has a small cont...
#
# underlying file reads for get(): 2 (one per distinct key, even if requested repeatedly)
# after 3 more repeat requests, file reads for get(): 2 (unchanged -- served from cache)
#
# === template rendering, different variables each call ===
# --- call 1 (no prior findings) ---
# Review this Python change for security.
#
# Diff:
# + eval(user_input)
#
# --- call 2 (same diff, WITH prior findings) ---
# Review this Python change for security.
#
# Known recurring issues in this codebase: unsanitized eval() calls (flagged 3 times before)
#
# Diff:
# + eval(user_input)
#
# asserts passed: identical diff, different variables, genuinely different rendered output
# template reads for render(): 2 (not cached -- variables differ per call, so caching the result would be wrong)
Both asserts passed on the run that produced this output: the two rendered calls genuinely differ, and the difference is exactly the conditional block that should have fired for one call and not the other — not an accident of two separately-written strings drifting apart.
6. Real-world example
A team's first version of their LLM integration had the system prompt as a string literal at the top of the one function that called the API. It worked fine until a second feature needed a nearly identical prompt with one additional constraint, and the fastest path under deadline pressure was copying the string and editing the copy — now two prompts existed that were meant to express the same underlying policy, maintained independently.
Months later, a compliance requirement meant one specific instruction had to be added to every system prompt in the product. The team searched the codebase for the phrase they expected to find and updated it everywhere they found a match — but by then there were five call sites, not two, each with a slightly hand-edited copy of the original string, and two of the five had wording different enough that the search missed them entirely. Those two shipped without the required instruction for several weeks, discovered only when an internal audit sampled prompts directly from the running system rather than from the codebase's search results.
Moving to a shared prompt-file architecture didn't just fix the immediate gap — it made the fix to the underlying problem structural rather than procedural. With every system prompt loaded from a single registry of files, adding a required clause became one edit to one file (or a small, enumerable set of files), and a follow-up audit could check the actual files on disk rather than trusting that every code search had found every copy.
7. Interview questions companies actually ask
Q1. What's actually wrong with putting a prompt directly in the function that calls the LLM? Nothing, until it needs to be reused, reviewed by someone who doesn't want to read code, or varied slightly for a second use case — at that point a string literal in code either gets copy-pasted with small edits (creating drift between copies that were supposed to represent the same policy) or forces prompt logic to be threaded through conditionals at every call site that needs a variant.
Q2. Why cache a static system prompt's file read but never cache a rendered template's output? A static prompt's content never changes between reads, so caching the read is a pure win — the second read would return the identical string anyway. A template's rendered output depends on the variables passed in for that specific call, so caching it would mean either serving stale output to a call with different variables, or building a cache keyed on every possible variable combination, which is more complex than simply re-rendering.
Q3. How would you design provider-specific prompt selection so a new provider doesn't require touching every call site? A lookup table mapping provider identifier to the concrete prompt file to use, consulted in one place, rather than conditional logic duplicated at every call site that needs to make the same decision. Adding a new provider becomes one new entry in the table; removing the old scattered-conditional approach means there's only one place that can get the mapping wrong.
Q4. A compliance requirement means one clause must be added to every system prompt in the product. Why would this be hard in a codebase with prompts as string literals scattered through the code, and easy in one with prompts as managed files? In the scattered-string version, finding "every" system prompt means trusting a code search to catch every copy, including ones that were hand-edited enough to no longer match the search terms — exactly the failure mode where some copies get missed. In the managed-file version, "every system prompt" is an enumerable, auditable set of files in one directory, so a script can mechanically confirm the required clause is present in all of them, rather than relying on a human's code search being complete.
Q5. When is a full prompt-manager architecture (registry, caching, templating) overkill? When there's a single prompt, used in exactly one place, that has never needed a variant and shows no signs of needing one — at that scale, a string constant is simpler and the manager, cache, and registry are machinery solving a problem that doesn't exist yet. The pattern earns its cost once reuse, review, or variation actually shows up, not before.
Q6. How do you decide whether something belongs in the "static, cached" bucket or the "template, rendered fresh" bucket? Ask whether the content is fully determined once, independent of any specific request, or whether it depends on data that's different for every call. A system prompt describing a role and constraints is usually the former; anything that includes the actual user question, retrieved context, or per-request data is the latter, and conflating the two either wastes I/O re-reading something static or silently serves the wrong per-request content from an incorrectly cached template.
8. When to use / tradeoffs
Reach for a prompt-file architecture with a registry when:
- more than one call site needs the same or a closely related prompt
- prompts need to vary by provider, tier, or mode in a way that would otherwise require scattered conditionals
- someone who isn't editing code (or a compliance/audit process) needs to review or verify prompt content directly
| Situation | Why it breaks | Use instead |
|---|---|---|
| A single prompt, used in one place, with no expected variants | The registry/cache/template machinery adds complexity with nothing to manage | A plain string constant |
| Caching a rendered template's output rather than just its file read | Serves stale output to a call whose variables have changed, or requires an unwieldy cache key covering every variable combination | Cache the template file read; always re-render with the current call's variables |
Provider selection implemented as scattered if/elif at each call site | A new provider requires finding and updating every conditional independently, and it's easy to miss one | A single lookup table consulted in one place |
| Prompts needed only during initial prototyping, expected to be thrown away | Building durable file/registry infrastructure for throwaway exploration is wasted effort | Inline strings, cleaned up (or formalized) once the prototype becomes real |
Honest limits. Prompt-file architecture makes prompts easier to find, review, and vary — it does not make a prompt's content better, and a well-organized bad prompt is still a bad prompt. Caching solves a performance and consistency problem for static content; it does nothing for the actual quality of what the LLM produces from that content. And a registry keyed by provider or tier only helps if the key genuinely captures the distinction that matters — a registry with the wrong granularity has the same failure mode as any other lookup table with a poorly chosen key: either forcing genuinely different cases to share a prompt, or fragmenting one policy across more variants than the situation actually requires.
9. Summary + related articles
- A prompt embedded as a string literal in application code duplicates and drifts the moment it's needed in a second place or needs a provider-specific variant.
- Static prompts (fully determined, reusable) should be loaded from files and cached; templates (rendered from per-request data) should be re-rendered every call and never cached by their output.
- Provider or tier-specific prompt selection belongs in one lookup table, not scattered conditionals repeated at every call site.
- Measured: four requests for provider system prompts produced exactly two file reads (one per distinct key, cache hits for repeats); two template renders with different variables produced two genuinely different, verified outputs.
- Boundary: this architecture pays for itself once a prompt is reused, reviewed, or varied — for a single permanent prompt used in exactly one place, a plain string constant is the right choice.
Related:
- API Best Practices — the same "make the right thing structurally easy and the wrong thing structurally hard" principle applied to provider payload limits and retries instead of prompt selection
- Consistency Memory: Making Repeated AI Judgments Agree With Each Other — another case of "derive once, reuse deliberately" rather than re-deriving independently every time, applied to judgments instead of prompt content
