TL;DR
- You cannot debug an agent from its final answer — a wrong tool, a bad retrieval, or a loop can still produce a plausible output. You debug from the trace: every thought → action → observation, plus tool args, tokens, latency, and cost.
- Memorize the failure taxonomy: infinite loops, error compounding, wrong-tool selection, hallucinated arguments, context bloat/rot, cost/token blowups, and silent partial failure.
- Tooling: LangSmith and Langfuse give hierarchical traces (spans for each LLM/tool call), agent-graph views, tool-call analytics, cost dashboards, and eval hooks. In-house, you emit structured events per node and persist an errors.json + run manifest for reproducibility.
- Root-causing method: reproduce deterministically (fixed seed/temperature, saved inputs) → open the trace → find the first step that diverges (not the symptom) → inspect that step's exact prompt/args/observation → fix and add a regression test.
Simple explanation + analogy
Debugging an agent is like being an aircraft accident investigator, not a witness. A witness ("the answer was wrong") tells you that something failed. The investigator reads the flight recorder — the black box that logged every control input and instrument reading second by second — and finds the first deviation that started the chain. Agents need a black box too: a trace of every thought, tool call, argument, observation, and cost. Without it you're guessing; with it you scrub to the exact frame where things went wrong.
The subtle trap: agents often "land safely" (produce a confident answer) despite a mid-flight failure — they picked the wrong tool but the LLM smoothed over it. So a green final output does not mean a clean flight. You have to read the recorder.
Diagram
THE AGENT BLACK BOX (trace)
run ─┬─ span: LLM call ─── prompt | thought | token_in/out | latency | $$
├─ span: tool call ── name | ARGS | observation | error?
├─ span: LLM call ─── thought (react to observation) ...
└─ span: ... (nested for sub-agents / sub-graphs)
ROOT-CAUSE WORKFLOW FAILURE TAXONOMY
reproduce (seed, saved input) ┌───────────────────────────────┐
│ │ infinite loop wrong tool │
▼ │ error compound hallucin. args │
open full trace │ context bloat cost blowup │
│ │ silent partial failure │
▼ └───────────────────────────────┘
find FIRST divergent step ◀── not the symptom, the origin
│
▼
inspect its prompt / args / observation ─▶ fix ─▶ add regression test
How it works (deep)
Observability: what to capture
An agent trace is a tree of spans. For each span record: inputs (the exact rendered prompt or tool args), outputs (completion or observation), model + params (temperature, max_tokens), token counts, latency, cost, and any error. Sub-agents/sub-graphs nest as child spans so you can see which agent looped or blew the budget. This is what LangSmith and Langfuse give you out of the box; the key property is that control flow is visible — which subagents/handoffs/loop iterations ran, in what order, how often.
Three pillars:
- Tracing — the per-run span tree (debugging a single run).
- Metrics/monitoring — aggregate token/cost/latency/error-rate over time (spotting regressions and blowups).
- Evals — scored checks on captured traces (did it pick the right tool? cite the right doc?).
The failure taxonomy (know all of these)
- Infinite / runaway loops — the agent re-calls the same tool or ping-pongs between agents. Cause: no progress, no dedup, or an observation that keeps re-triggering the same plan. Guard: iteration caps, "once-only" directives, and cycle detection (the clinical RAG system caps tool iterations and flags
loop_capped). - Error compounding — a small early mistake propagates and amplifies down the chain (
P(success)=pⁿ). Guard: validate/gate between steps, don't let one bad step feed the next unchecked. - Wrong-tool selection — the LLM routes to the wrong specialist/tool. Cause: vague tool descriptions or overlapping tools. Guard: sharp, non-overlapping tool descriptions (the clinical RAG system's specialist tools have precise "Use for: ..." lists) and a routing eval.
- Hallucinated arguments — the model calls a real tool with invented/malformed args (a made-up ID, wrong JSON). Guard: schema-validate tool inputs before execution; coerce/repair or reject.
- Context bloat / rot — the window grows across hops until latency and cost spike and quality drops. Guard: summarize history, pass minimal structured context.
- Cost / token blowups — recursion, retries, or huge contexts explode spend (multi-agent already ~15× a chat). Guard: per-run token/cost budgets, alerts, and circuit breakers.
- Silent partial failure — a sub-step fails but the run returns a confident answer anyway. Guard: structured per-node error capture so failures surface even when the output looks fine.
Reproducibility
You can't fix what you can't reproduce. Non-determinism comes from temperature, tool/network variance, and changing model versions. Practices:
- Pin the run: fixed temperature (0 for debugging), saved inputs, recorded model + version.
- Persist a run manifest — model, embedding model/dim, RAG backend, key settings, IDs, timestamps — so a run can be re-created and compared (the grading pipeline writes
manifest.jsonper run). - Checkpoint state so a failed run can be resumed/inspected rather than lost (the grading pipeline's SqliteSaver checkpointer).
- Structured errors artifact —
errors.jsonwritten before re-raise, so post-mortems have the exact failure.
Root-cause method (the interview answer)
- Reproduce deterministically from saved inputs + manifest.
- Open the full trace — don't theorize from the final answer.
- Find the first divergence — scrub to the earliest step whose thought/args/observation is wrong. Symptoms cluster at the end; causes live earlier.
- Inspect that span's exact prompt / args / observation — was the tool description ambiguous? the arg hallucinated? the retrieval empty?
- Fix at the source (tool description, schema, prompt, gate) and add a regression test / eval so it can't silently return.
The math
Reliability compounding. Independent per-step success p over n steps:
P(run success) = pⁿ p=0.97, n=20 → 0.54
A 3% per-step error rate makes a 20-step agent a coin flip — this is why you gate and checkpoint rather than run long unguarded chains, and why observability must be per-step.
Loop / cost bound. With max iterations M, avg tokens/iteration t, cost/token κ, a single run's worst case is M · t · κ. Setting M and a token budget converts an unbounded blowup into a bounded, alertable failure. A circuit breaker on batch error rate (failures/processed > max_error_rate) bounds aggregate spend the same way.
MTTR and trace coverage. Mean-time-to-resolution drops roughly with how early you can localize the fault. If a trace lets you binary-search n steps, localization is O(log n) inspections instead of O(n) guesses — the practical ROI of tracing.
Code
Structured per-node error capture — persist before re-raising (a grading pipeline)
# node_wrap.py (example) — every node is wrapped so failures are never silent
def wrap_node(fn, *, node_name):
@functools.wraps(fn)
async def async_wrapper(state, *a, **k):
try:
return await fn(state, *a, **k)
except GraphInterrupt:
raise
except Exception as exc:
log.exception("node %s failed", node_name) # stack trace to logs
merged = append_structured_error(state, node=node_name, exc=exc)
_persist_errors_artifact(merged) # -> errors.json for post-mortem
raise
return async_wrapper
Progress/trace events per node — the black box (a grading pipeline)
# runtime.py (example) — emit {phase}_start / {phase}_done as each node runs
async for event in g.astream_events(initial_state, config=config, version="v2"):
kind, name = event.get("event", ""), event.get("name", "")
if kind == "on_chain_start" and name in _NODE_TO_PHASE:
_emit(progress_cb, f"{_NODE_TO_PHASE[name]}_start", {"node": name, "ts": time.time()})
elif kind == "on_chain_end" and name in _NODE_TO_PHASE:
_emit(progress_cb, f"{_NODE_TO_PHASE[name]}_done", {"node": name, "ts": time.time()})
Reproducibility manifest (a grading pipeline)
# runtime.py (example) — write_run_manifest(): snapshot everything needed to reproduce a run
manifest = {
"active_llm_provider": active_llm_provider(), "llm_model": active_llm_model(),
"embedding_model": embedding_model_name(), "embedding_dim": embedding_dimension(),
"rag_backend": s.rag.backend,
"settings": {"samples": s.grading.samples, "cv_threshold": s.grading.cv_threshold,
"rag_min_score": rag_min_score_for_provider(), "top_k": s.rag.top_k},
"run_id": state.get("run_id"), "started_at": ..., "ended_at": ...,
}
Path(run_dir, "manifest.json").write_text(json.dumps(manifest, indent=2))
Loop guard + hallucinated-tool defense — the clinical RAG system
# run_state.py — iteration telemetry + once-only directive stop runaway tool calls
@dataclass
class SupervisorRunState:
specialist_cache: dict[str, dict] = field(default_factory=dict)
tool_iterations: int = 0
loop_capped: bool = False # tripped when the model won't stop calling tools
Hallucinated-arg / malformed-output defense — schema coercion + retry (a grading pipeline)
# grade_attempt.py — malformed JSON is caught, retried once, then flagged (not silently passed)
last_raw, grade_data = await _grade_llm_call(...)
if grade_data is None:
last_raw, grade_data = await _grade_llm_call(...) # explicit single retry
if grade_data is None:
state = append_error_entry(state, node="grade_attempt",
type="json_parse_error", message="malformed JSON after retry")
emit_progress(run_id, "grade_parse_failed", {"node": "grade_attempt"})
grade_data = coerce(grade_data, GradeSummaryOutput, context="grade_attempt") # schema-coerce
Testing the graph without burning tokens — dry-run + node tests
A grading pipeline ships graph_dry_run.py and test_graph_dry_run.py that assert the topology (nodes/edges) without calling the LLM, plus per-node unit tests — the cheap regression layer that catches wiring bugs before an expensive end-to-end run.
Real-world example
A grading pipeline — full observability stack. Every node is wrap_node-wrapped (errors → errors.json), the runtime streams {phase}_start/_done events (live trace + SSE progress UI), each run writes a manifest.json (reproducibility), a SqliteSaver checkpointer lets a crashed run resume, self_consistency gates low-confidence outputs to human review, and grade_parallel's circuit breaker bounds cost when a batch goes bad. That's the seven failure modes covered in one pipeline.
The clinical RAG system — loop and cost control. Per-request SupervisorRunState tracks tool_iterations and loop_capped, and the once-only directive stops the supervisor from re-consulting a finished specialist — directly attacking runaway loops and token blowups. Sharp, non-overlapping tool descriptions reduce wrong-tool selection.
The concierge — errors as observations. Each specialist tool catches exceptions and returns a string, so a worker failure appears in the orchestrator's trace as an observation it can react to, rather than crashing the run silently.
The group-trip suggester — per-node error channel. Every node writes to state["error"] on failure and degrades gracefully (e.g., empty destination_context), so a failed retrieval is visible and non-fatal.
Interview questions companies actually ask
1. Why can't you debug an agent from its final output? [easy] Because a wrong tool, empty retrieval, or loop can still yield a plausible answer — the LLM smooths over mid-run failures. You need the full trace (thought/action/observation, args, cost) to see what actually happened. See LangChain: agent observability.
2. What does LangSmith/Langfuse give you for debugging? [easy] Hierarchical traces where every LLM call, tool invocation, and reasoning step is a span; agent-graph visualization; tool-call analytics; cost/latency monitors; and eval hooks on captured traces — so you can see which subagents/handoffs/loops ran and how often. See DigitalOcean: LangSmith and Langfuse: agent observability.
3. List the common agent failure modes. [medium] Infinite/runaway loops, error compounding, wrong-tool selection, hallucinated arguments, context bloat/rot, cost/token blowups, and silent partial failure. Be ready to name a guard for each (iteration caps, gates, sharp tool descriptions, input schema validation, summarization, budgets, structured error capture). See LangChain: agent observability powers evaluation.
4. Walk me through root-causing a misbehaving agent. [hard] Reproduce deterministically (temp 0, saved inputs, manifest) → open the full trace → find the first divergent step (not the symptom) → inspect its exact prompt/args/observation → fix at the source → add a regression test/eval. See apxml: LangSmith debugging & RCA.
5. How do you stop an agent from looping forever? [medium]
Iteration/step caps, "once-only" directives that mark completed tools done, dedup via shared-state cache, cycle detection, and a wall-clock/token budget that hard-stops the run. The clinical RAG system's tool_iterations + loop_capped + once-only cache is a concrete example.
6. How do you defend against hallucinated tool arguments? [hard]
Schema-validate tool inputs before execution (Pydantic), coerce/repair where safe, reject otherwise, and retry once on malformed output before flagging — never pass unvalidated args downstream. A grading pipeline parses/retries/coerces LLM JSON into GradeSummaryOutput. See LangSmith observability.
7. How do you make an agent run reproducible? [medium]
Pin temperature and inputs, record model + version, persist a run manifest (model, embeddings, settings, IDs), and checkpoint state so runs can be resumed and compared. A grading pipeline's write_run_manifest + SqliteSaver do this. See Langfuse: observability overview.
8. How do you keep agent costs from blowing up in production? [hard] Per-run token/cost budgets with alerts, iteration caps, circuit breakers on batch error rate, aggressive context summarization, and monitoring cost/token metrics over time to catch regressions — remembering multi-agent is ~15× a chat. See LangChain: agent observability.
9. How do you test agents without spending a fortune on tokens? [medium]
Layer it: topology/dry-run tests that assert graph wiring with no LLM calls, per-node unit tests with mocked tools/LLM, then a small set of end-to-end evals on golden traces. A grading pipeline ships exactly this (graph_dry_run, node tests). See apxml: LangSmith debugging.
10. What's "silent partial failure" and how do you catch it? [hard]
A sub-step fails but the run still returns a confident answer, so the failure never surfaces. Catch it with structured per-node error capture (record + persist even when the final output looks fine) and evals that check intermediate steps, not just the answer. A grading pipeline's wrap_node + errors.json is the pattern.
When to use / tradeoffs
- Always trace in production — the marginal cost of spans is tiny next to the debugging time saved (localization goes from
O(n)guessing toO(log n)). - Managed (LangSmith) vs. self-host (Langfuse): LangSmith is fastest to adopt in the LangChain ecosystem; Langfuse is open-source/self-hostable — pick per data-residency and budget constraints.
- Sampling: trace 100% in dev and for errors; sample high-volume prod traffic to control storage while keeping all failures.
- Don't over-instrument blindly: capture prompts/args/observations/cost, but redact PII and avoid logging secrets — traces are a data-governance surface.
- Evals > vibes: once you can trace, add scored checks on the traces so regressions fail a test instead of silently shipping.
Summary + related articles
- Debug from the trace, not the answer — agents fail silently and still sound confident.
- Know the failure taxonomy (loops, compounding, wrong tool, hallucinated args, bloat, cost blowups, silent partial failure) and a guard for each.
- Tooling = tracing + metrics + evals (LangSmith/Langfuse); in-house = structured events +
errors.json+ run manifest + checkpointer. - Root-cause = reproduce → open trace → find the first divergence → fix at source → add a regression test.
Related: Multi-Agent Patterns · Agent Orchestration · Agent Communication
Sources: LangChain: agent observability · LangSmith observability · Langfuse: agent observability · DigitalOcean: LangSmith · apxml: LangSmith debugging & RCA