Production Agents

TL;DR

Shipping an agent that works in a demo and shipping one that survives production are different jobs. Production is where non-determinism meets real money, real latency budgets, and real users who will do things your gold set never imagined. The production checklist:

• Cost control — model routing (cheap model for easy sub-tasks), prompt caching, and hard step/budget caps so a loop can't run away.
• Latency — parallelize independent tool calls, stream tokens to the user, cache stable prefixes.
• Guardrails — a policy filter on inputs and outputs (prompt-injection, unsafe content) that the agent cannot route around.
• Human-in-the-loop for high-stakes — approval gates on irreversible actions.
• Error handling + retries — typed error classes, exponential backoff, graceful degradation, fallbacks.
• Determinism where it matters — "LLM points, code decides": let the model choose, let deterministic code do.
• Observability / tracing — every prompt, tool call, retrieval, cost, and latency captured and correlated.
• Scaling — stateless workers, shared session store, backpressure.
• Versioning / rollback — version prompts + agent config, gate releases on an eval, roll back on regression.

The one sentence: a production agent is a non-deterministic core wrapped in deterministic scaffolding — caps, guardrails, retries, tracing, and an eval gate — so that its unpredictability is bounded, observable, and reversible.

---

Simple explanation + analogy

A production agent is a race car on a real road, not on a track.

On the track (the demo), you show off raw speed. On the road (production), you need everything around the engine: a speed governor so it can't run away (budget caps), guardrails on the shoulder so a slip doesn't go off a cliff (safety guardrails), a dashboard telling you speed, fuel, and engine temp in real time (observability), airbags and ABS for when something does go wrong (retries, error handling), and a mechanic who logged exactly which parts are on the car so a bad part can be swapped back (versioning/rollback).

The engine — the LLM — is the exciting, unpredictable part. But nobody ships a car that's just an engine. The boring scaffolding is what makes the exciting part safe to drive at scale. Production agent engineering is 20% prompt and 80% the stuff around it.

---

Diagram

                         PRODUCTION AGENT ARCHITECTURE

  user ─► ┌──────────────┐   guardrail (in)   ┌──────────────────────────┐
          │  API / edge  │ ─────────────────► │   INPUT GUARDRAIL         │
          └──────────────┘                    │  prompt-injection filter  │
                                              └────────────┬─────────────┘
                                                           ▼
   ┌──────────────────────────────── AGENT CORE (bounded loop) ───────────────┐
   │  ┌────────────┐   route by task tier   ┌─────────────────────────────┐   │
   │  │  ROUTER    │ ─────────────────────► │  cheap model  |  strong model│   │
   │  └────────────┘                        └─────────────────────────────┘   │
   │        │  step cap · budget cap · retries+backoff                        │
   │        ▼                                                                 │
   │  ┌───────────┐  parallel  ┌──────────┐   "LLM points, code decides"     │
   │  │  PLAN/ACT │ ─────────► │  TOOLS   │   deterministic execution        │
   │  └───────────┘            └──────────┘                                   │
   │        │  low-confidence / irreversible? ──► HUMAN-IN-THE-LOOP gate      │
   └────────┼────────────────────────────────────────────────────────────────┘
            ▼
   ┌──────────────────┐   guardrail (out)   ┌──────────────────────────┐
   │  OUTPUT GUARDRAIL│ ◄─────────────────  │   stream tokens to user   │
   └──────────────────┘                     └──────────────────────────┘
            │
            ▼
   ┌───────────────────────────── CROSS-CUTTING ─────────────────────────────┐
   │  OBSERVABILITY: trace every prompt · tool call · retrieval · cost · p95   │
   │  VERSIONING:  prompt vN + agent config vN ──► eval gate ──► deploy/rollback│
   └──────────────────────────────────────────────────────────────────────────┘

---

How it works (deep)

Cost control

Three levers, in order of impact:

• Model routing (task-tier routing). Not every step needs the flagship model. Route bulk/easy work (classification, screening, routing decisions) to a cheap fast model and reserve the expensive model for hard steps (final synthesis, complex extraction). At today's prices — roughly Opus-tier $5/$25, Sonnet-tier $3/$15, Haiku-tier $1/$5 per million input/output tokens — routing a screening step from Opus to Haiku is a 5× input cost cut for that step. A fixed multi-step review pipeline does exactly this with a resolve_model_for_task helper.
• Prompt caching. Keep the stable prefix (system prompt, tool definitions, frozen context) byte-identical at the front, mark it cacheable. Cache reads cost ~0.1× base input and writes ~1.25×, so re-sending a cached prefix is ~10× cheaper than cold. In production, system prompts dominate input tokens — this is enormous leverage, and it's free if you just stop interpolating timestamps/UUIDs into the prefix.
• Step & budget caps. A cap is a cost control and a safety control. Unbounded loop expected cost = c/q (q = per-step termination prob) → ∞; a step cap S bounds it to ≤ c·S. Cap tokens per task, tool calls per task, and total steps.

Latency

• Parallel tool calls. If the model requests three independent tools in one turn, execute them concurrently and return all results together — don't serialize. Splitting parallel results across turns also silently trains the model to stop parallelizing.
• Streaming. Stream tokens to the user so time-to-first-token is small even when total generation is long. Essential for perceived latency; also avoids request timeouts on long generations.
• Caching (again) — a cache hit skips prefill on the shared prefix, cutting both cost and latency.

Guardrails

A guardrail is a policy filter on inputs and outputs that runs outside the model's control, so the agent can't be prompted into bypassing it. It catches prompt-injection attacks, disallowed content, and unsafe outputs before they reach a tool or the user. The critical property is that it's a separate enforcement layer — real-time, deterministic, and non-bypassable — not an instruction in the system prompt (which a jailbreak can override).

Human-in-the-loop for high-stakes

Route irreversible or low-confidence actions to a human approval gate (see the Autonomous Agents article's reversibility gate). In production this means: classify actions by reversibility, auto-execute the safe ones, and block on delete/deploy/pay/send until a human approves. This is HITL-per-action inside an otherwise fast agent.

Error handling + retries

• Typed errors, not string matching. Catch specific error classes and branch: retryable (429 rate-limit, 5xx, network) → exponential backoff with jitter; non-retryable (400 bad request, 404) → fail fast. A single broad catch-all throws away the retryable/non-retryable distinction.
• Backoff + jitter. On 429/5xx, wait base · 2^attempt + random_jitter, capped. Most SDKs retry automatically; add custom logic only for behavior beyond that.
• Fallbacks & graceful degradation. If the primary model is overloaded, fall back to another model or a cached/partial answer. Return something useful rather than a 500.

Determinism where it matters — "LLM points, code decides"

The single most important production reliability principle. The LLM is non-deterministic and can hallucinate; deterministic code cannot. So split every consequential operation into a decision (model) and an action (code): the model chooses which cell to read, which tool to call, which branch to take — and Python reads the number, executes the tool, follows the branch. The model can't emit a value it isn't allowed to type. This is the core rule that makes such a pipeline's numeric outputs trustworthy.

Observability / tracing

You cannot debug or improve what you can't see. Trace every interaction: the prompt, each tool call and its result, retrievals, guardrail trips, token counts, cost, and latency — correlated by a request/trace id. This is how you find the failing trajectory, attribute cost, and catch drift. Phase-4 observability maturity is tracing the reasoning steps and multi-model routing, not just logging final responses.

Scaling

• Stateless workers + shared session store. Keep per-request workers stateless; put session memory in a shared store (Redis/DB) so any worker can serve any session (the Memory Management article's InMemoryChatMessageHistory → RedisChatMessageHistory swap).
• Backpressure & rate-limit handling. Respect provider rate limits; queue or shed load rather than hammering into 429s.

Versioning / rollback

Prompts and agent configs are deployable artifacts — version them like code. Every release passes an eval gate (the offline gold set from the Agent Evaluation article) before it ships, and a regression triggers rollback to the last known-good version. Default-off feature flags are a form of this: a new capability ships dark, is enabled only after the eval proves it helps, and can be flipped back off instantly.

---

The math

Cost per task, with routing

$$ \text{Cost}{\text{task}} = \sum{i=1}^{N} \Big( T^{in}i , c{in}(m_i) + T^{out}i , c{out}(m_i) + T^{cache}i , c{cache}(m_i) \Big) $$

where step i runs on model m_i and prices depend on the model. Routing changes c(m_i) per step: send k of N steps to a model that's f× cheaper on input and the input cost of those steps drops by f×. Example: 8 of 10 screening steps moved from Opus ($5/M) to Haiku ($1/M) cuts the input cost of those steps 5×.

Caching break-even

Cache write costs 1.25× base input (5-min TTL); cache read costs 0.1×. For a prefix reused R times:

$$ \text{cached total} = 1.25 + 0.1(R-1) \quad\text{vs.}\quad \text{uncached total} = R $$

Break-even is at R ≈ 2: from the second reuse onward, caching wins. For an agent that resends a large system prompt every turn, R is large and caching approaches a 10× saving on the prefix.

Bounded worst-case cost

With per-step cost c and a hard step cap S, worst-case task cost is ≤ c·S — unconditionally, regardless of whether the loop converges. Without a cap, expected cost is c/q (q = per-step termination probability), which diverges as q → 0. The cap is what turns an unbounded tail into a line item you can budget.

---

Real code

A Strands multi-agent trip-planning assistant on Bedrock uses, as its production safety layer, a Bedrock GUARDRAIL attached to the model. The guardrail is created once (content-policy filters) and its id is bound to the model, so every agent call passes through it — the agent can't route around it.

# guardrail.py — create a Bedrock guardrail as a policy filter
def create_guardrail(guardrail_name="guardrail-travel-safety"):
    response = bedrock_client.create_guardrail(
        name=guardrail_name,
        description="Ensures travel recommendations are safe and appropriate",
        contentPolicyConfig={
            "filtersConfig": [
                {"type": "HATE",          "inputStrength": "NONE", "outputStrength": "NONE"},
                {"type": "MISCONDUCT",    "inputStrength": "NONE", "outputStrength": "NONE"},
                {"type": "PROMPT_ATTACK", "inputStrength": "NONE", "outputStrength": "NONE"},
            ]
        },
    )
    return response.get("guardrailId"), response.get("guardrailArn")

def get_guardrail_id(guardrail_name="guardrail-travel-safety"):
    for g in bedrock_client.list_guardrails().get("guardrails", []):
        if g.get("name") == guardrail_name:
            return g.get("id")
    return None

# orchestrator.py — bind the guardrail to the model so ALL calls are filtered
bedrock_model = BedrockModel(
    model_id="anthropic.claude-3-5-sonnet-20241022-v2:0",
    region_name=os.getenv("AWS_REGION", "us-east-1"),
    guardrail_id=get_guardrail_id(),      # ← every agent call passes the guardrail
)

What makes this a production pattern, not a demo:

1. The guardrail is bound to the model, not the prompt. A jailbreak can rewrite the system prompt; it cannot remove a Bedrock guardrail attached at the model layer. Enforcement lives outside the model's controllable surface — that's the whole point.
2. PROMPT_ATTACK is an explicit filter category. The orchestrator coordinates several sub-agents (flights, hotels, activities, budget); the guardrail is the single choke point that catches prompt-injection before it reaches any of them.
3. get_guardrail_id is idempotent lookup + create_guardrail is create-if-absent. The safety layer is provisioned as infrastructure, discovered by name, and reused — versionable and rollback-able like any other config, not hand-wired per request.

Wrap this in the deterministic scaffolding from the Autonomous Agents loop (step cap, budget cap, retries with backoff, human gate on irreversible actions) and you have the full production shape: guardrail → bounded loop → "LLM points, code decides" → traced and versioned.

---

Real-world example

A fixed multi-step review pipeline is a reference implementation of production-grade agent engineering, and nearly every item on the checklist is visible in it:

• Determinism where it matters — "LLM points, Python reads": the model selects which evidence to use; deterministic Python reads the actual numbers. Hallucinated values are structurally impossible.
• Cost control via routing — resolve_model_for_task maps each task tier (bulk scoring, screening, extraction, methodology, orchestration) to a configured model, so cheap work runs on a cheap model.
• Versioning / rollback via flags + eval gate — new extraction tiers ship behind default-off flags and are enabled only after the offline gold-set eval (score.py) shows a measurable win; a regression flips the flag back off. That's a release gate and a rollback lever in one.
• Reproducibility / observability — every run persists a checksummed artifact bundle under runs/<run_id>/, so any autonomous step is inspectable, resumable, and re-scorable. That's tracing at the artifact level.
• Human-in-the-loop — low-confidence reconciliations are routed to human review rather than auto-accepted.

The recommender contributes the guardrail layer above: even in an orchestrated multi-agent flow, a Bedrock guardrail (hate / misconduct / prompt-attack filters) sits on the model so no sub-agent can be prompted past the safety boundary. Together they show the division of labor: one system demonstrates reliability engineering (determinism, eval gates, reproducibility, routing); the other demonstrates safety enforcement (a non-bypassable guardrail). Production needs both.

---

Interview questions companies actually ask

Q1. What changes when you move an agent from demo to production? [easy] The engine (the LLM) stays; you add the scaffolding around it — cost caps, guardrails, retries, observability, versioning, and an eval gate. Production is bounding, observing, and making reversible a non-deterministic core so its unpredictability is safe at scale. (LLM observability guide 2026)

Q2. Name three ways to control an agent's cost. [medium] Model routing (cheap model for easy sub-tasks, expensive only for hard ones), prompt caching (stable prefix → cache reads ~0.1× vs cold), and hard step/budget caps (bound the runaway tail to ≤ c·S). Add semantic caching for repeated queries. (Best LLM routers 2026, LLMOps guide 2026)

Q3. Explain "LLM points, code decides" and why it matters in production. [hard] Split every consequential operation into a decision (model chooses which cell/tool/branch) and an action (deterministic code reads the number / executes the tool). The model can't emit a value it isn't allowed to type, so hallucinations can't reach the output. It's the core reliability principle behind trustworthy numeric/structured agent output.

Q4. What is a guardrail and why can't it just be an instruction in the system prompt? [medium] A guardrail is a real-time policy filter on inputs/outputs that runs outside the model's control (e.g. a Bedrock guardrail bound to the model). A system-prompt instruction can be overridden by a jailbreak/prompt-injection; a separate enforcement layer cannot be prompted away. Guardrails act in real time before responses reach the user or a tool. (LLM observability guide)

Q5. How do you handle errors and retries against an LLM API? [medium] Catch typed error classes, not strings: retryable (429, 5xx, network) → exponential backoff with jitter, capped; non-retryable (400/404) → fail fast. Add fallbacks (alternate model, cached/partial answer) and graceful degradation. Most SDKs retry 429/5xx automatically; add custom logic only beyond that. Production gateways add circuit breakers, timeouts, and load balancing. (LLMOps guide 2026)

Q6. What should agent observability capture, and why isn't logging final responses enough? [hard] Trace every prompt, tool call + result, retrieval, guardrail trip, token count, cost, and latency, correlated by trace id. Final-response logging can't tell you why a trajectory failed, can't attribute cost across a multi-step multi-model run, and can't detect drift. Mature (phase-4) observability traces reasoning steps and routing behavior. (AI agent observability, Galileo: LLM monitoring)

Q7. How do you cut agent latency without changing the model? [medium] Parallelize independent tool calls (return all results in one turn), stream tokens (small time-to-first-token), and cache the stable prefix (skip prefill). Also: shorten trajectories (fewer tool round trips) and route latency-sensitive steps to a faster model.

Q8. How do you version and safely roll out a prompt/agent change? [hard] Treat prompts + agent config as versioned artifacts. Gate every release on an offline eval (gold set) in CI; ship new capabilities default-off behind a flag; enable only after the eval shows a measurable win; roll back (or flip the flag off) on regression. Default-off tiers gated by a score.py-style eval are exactly this. Pin config versions so running sessions aren't broken by an in-flight change.

Q9. Why is a step/budget cap a safety control and not just a cost control? [medium] Because an unbounded loop (runaway reflection, retry storm, self-fan-out) is a safety incident, not just a bill — it's the same failure class as an agent taking unbounded unsafe actions. A hard cap converts the divergent tail (c/q → ∞) into a bounded, known worst case (≤ c·S) and forces graceful termination.

Q10. Design the production wrapper around a bare tool-calling agent. [hard] Input guardrail → router (task-tier model selection) → bounded loop (step + budget caps, retries with backoff, parallel tools, streaming) with "LLM points, code decides" for consequential ops and a human gate on irreversible/low-confidence actions → output guardrail → all of it traced (prompt/tool/cost/latency by trace id) and shipped behind a versioned, eval-gated config with rollback. That's a non-bypassable guardrail + determinism/routing/eval-gate composed into one pipeline.

---

When to use / tradeoffs

Rules of thumb: cap every loop; cache every stable prefix; route every easy step to a cheap model; make every consequential op deterministic; guard every I/O boundary at the model layer; trace everything; gate every release on an eval and keep rollback one flag away.

---

Summary + related articles

A production agent is a non-deterministic core wrapped in deterministic scaffolding: routing + caching + caps for cost, parallelism + streaming for latency, guardrails + HITL for safety, typed retries + fallbacks for robustness, "LLM points, code decides" for reliability, tracing for observability, and versioned config behind an eval gate for safe rollout and rollback. One reference pipeline supplies the reliability half (determinism, routing, eval-gated flags, checksummed artifacts); a Bedrock guardrail supplies the non-bypassable safety half. Ship both.

Related articles:

• Agent Evaluation (6-5) — the eval gate that guards every release.
• Autonomous Agents (6-5) — the bounded loop, caps, and reversibility gates this wraps.
• Memory Management (6-5) — prompt caching, stable prefixes, and the stateless-worker + shared-store scaling pattern.
• Debugging Agents (6-4) & Orchestration (6-4) — tracing failing trajectories and coordinating multi-agent systems.