Agent Orchestration

TL;DR

• Orchestration is the machinery that turns a request into coordinated agent work: routing (who runs), task decomposition (into subtasks), shared state (what everyone reads/writes), and result synthesis (merging outputs into one answer).
• Two dominant styles: an orchestrator agent (an LLM decides the flow at runtime — flexible, non-deterministic) vs. a graph orchestrator like LangGraph (Python owns the flow, the LLM does judgment at nodes — deterministic, testable).
• The load-bearing principle: anything that must be guaranteed lives in the orchestrator (Python); anything that needs judgment lives in the LLM. Or, put another way: "LLM points, Python reads."
• Partial failure is the hard part. Real orchestrators wrap every node in structured error capture, gate on confidence, retry on parse failure, and trip circuit breakers when the batch error rate spikes — instead of letting one bad step poison the run.

---

Simple explanation + analogy

Orchestration is the conductor of an orchestra. The conductor doesn't play an instrument — they decide who plays when, keep everyone reading the same score (shared state), cue sections in the right order (routing + decomposition), and blend the sound into one performance (synthesis). If the oboe misses a cue, the conductor doesn't stop the concert — they adapt (partial-failure handling).

There are two kinds of conductor:

• The improv conductor (orchestrator agent): an LLM that decides on the fly which section to cue next based on how the music is going. Flexible, but you can't fully predict what it'll do.
• The score-following conductor (graph orchestrator): the flow is written down as a graph of nodes and edges; the conductor follows it exactly, and only within a node does a musician improvise. Predictable and testable.

Most production systems use the score-following conductor for the backbone and let the LLM improvise inside nodes.

---

Diagram

                         ORCHESTRATION = 4 JOBS
   ┌──────────────────────────────────────────────────────────────────┐
   │  1. ROUTE            2. DECOMPOSE        3. SHARE STATE   4. SYNTH │
   └──────────────────────────────────────────────────────────────────┘

  ORCHESTRATOR AGENT (LLM-driven flow)        GRAPH ORCHESTRATOR (Python-driven flow)

  user ─▶ ┌──────────────┐                    START
          │ orchestrator │  "which tool?"       │
          │   (ReAct)    │◀─────────┐           ▼
          └──────┬───────┘          │        [node A] ──write──▶ ┌─────────────┐
     tool call │  ▲ observation     │           │               │ shared      │
               ▼  │                 │           ▼   ◀──read───── │ state dict  │
          [worker agent] ───────────┘        [node B]           │ (TypedDict) │
               (repeat until done)              │               └─────────────┘
          then SYNTHESIZE ─▶ answer             ▼ conditional edge
                                             [gate] ─▶ human_review │ persist ─▶ END
   flexible, non-deterministic,              deterministic topology, each node
   flow decided at runtime                   wrapped for error capture + progress events

---

How it works (deep)

The four jobs of an orchestrator

1. Routing — deciding who runs. Either the LLM chooses (emits a tool call / a label like "followup") or Python chooses (a conditional-edge function reads state and returns the next node). Best practice: let the LLM classify/point, let Python decide the branch so routing is inspectable and testable.

2. Task decomposition. Break the request into subtasks. In an orchestrator-agent this is implicit — the model calls tools in sequence. In a graph it's explicit — the graph topology is the decomposition, fixed at build time. Explicit decomposition is easier to reason about; implicit is more adaptive.

3. Shared state. Every orchestrator needs a place agents read inputs and write outputs. Two shapes:

• Typed state object threaded through the graph (LangGraph TypedDict). Nodes return partial updates that get merged. This is the grading pipeline and the team-chat suggester.
• Blackboard / scratchpad that agents post to and read from (the clinical RAG system's per-request SupervisorRunState caches each specialist's papers + text so a tool isn't re-run).

4. Result synthesis. Merge worker outputs into one coherent answer. In the concierge the orchestrator LLM synthesizes; in the clinical RAG system a dedicated consensus-extraction pass turns free prose into a validated schema; in a grading pipeline self_consistency aggregates multiple sampled grades into a confidence score.

Orchestrator agent vs. graph orchestration

The senior take: use a graph for the backbone (guaranteed steps, gates, persistence) and an LLM inside nodes for judgment. Reach for a pure orchestrator-agent when requests are too varied to enumerate as a graph.

Handling partial failures — the part that separates senior from junior

Long chains fail probabilistically (pⁿ), so orchestration must degrade gracefully:

• Structured error capture per node — wrap every node so an unhandled exception is recorded on state["errors"] and persisted to errors.json before re-raising (the grading pipeline's wrap_node).
• Retry on recoverable errors — the grading pipeline retries a grading call once on JSON-parse failure before giving up.
• Confidence gates — branch to human review when self-consistency variance is high, instead of shipping a low-confidence result.
• Circuit breaker — abort a parallel batch when the running error rate exceeds a threshold, so one systemic bug doesn't burn the whole run's budget.
• Errors as data — worker tools return error strings the orchestrator can react to, not exceptions that crash it (the concierge).

---

The math

Routing as classification. A router picks branch bᵢ given query q; quality is just classification accuracy P(correct route | q). A misroute is often unrecoverable, so many systems keep routing cheap and low-temperature and validate the branch in Python.

Synthesis and self-consistency. A grading pipeline samples the grade n times and measures dispersion with the coefficient of variation:

mean = (1/n) Σ sᵢ
CV   = sqrt( (1/n) Σ (sᵢ - mean)² ) / mean        # std / mean, scale-free
confident  = (CV < cv_threshold) AND (mean_confidence ≥ conf_min)

Low CV across independent samples ⇒ the model is stable ⇒ trust it; high CV ⇒ flag for a human. This is a concrete, testable synthesis rule — great to cite in an interview.

Batch reliability with a circuit breaker. For a batch of N items processed at concurrency c, abort once:

failures / processed  >  max_error_rate

This bounds wasted spend at roughly max_error_rate · budget instead of the full N · cost_per_item when something is systemically broken.

---

Code

Orchestrator agent — the concierge routes and synthesizes

# orchestrator.py  (example)
ORCHESTRATOR_PROMPT = """You are a trip-planning orchestrator coordinating
flights, hotels, activities, and budget specialists.
...
When a user asks a question:
1. Extract any dates, destinations, and budget mentioned
2. Determine which agent(s) are needed        # ROUTING (decided by the LLM)
3. Call the relevant agent(s) with focused queries
4. Synthesize responses into a comprehensive answer   # SYNTHESIS
5. Provide actionable recommendations with budget constraints noted"""

orchestrator_agent = Agent(
    model=bedrock_model,
    system_prompt=ORCHESTRATOR_PROMPT,
    tools=[flights_agent_tool, hotels_agent_tool,
           activities_agent_tool, budget_agent_tool],
    conversation_manager=SummarizingConversationManager(   # keep shared context bounded
        summary_ratio=0.3, preserve_recent_messages=5),
)

Graph orchestration — a grading pipeline builds an explicit topology

# grading_graph.py  (example)
def route_after_confidence(state: GradingState) -> str:      # Python owns the branch
    return "gate_human_review" if state.get("flagged_items") else "normalize_persist"

def build_grading_graph(*, checkpointer=None):
    g = StateGraph(GradingState)
    g.add_node("grade_attempt",    wrap_node(grade_attempt_node,    node_name="grade_attempt"))
    g.add_node("self_consistency", wrap_node(self_consistency_node, node_name="self_consistency"))
    # ... more nodes ...
    g.add_edge("grade_attempt", "self_consistency")
    g.add_conditional_edges("self_consistency", route_after_confidence)  # gate on confidence
    return g.compile(checkpointer=checkpointer)   # checkpointer = state survives restarts

Per-node structured error capture — partial-failure handling

# node_wrap.py  (example)
def wrap_node(fn, *, node_name):
    @functools.wraps(fn)
    async def async_wrapper(state, *a, **k):
        try:
            return await fn(state, *a, **k)
        except GraphInterrupt:
            raise                                   # let human-in-the-loop interrupts pass
        except Exception as exc:
            log.exception("node %s failed", node_name)
            merged = append_structured_error(state, node=node_name, exc=exc)
            _persist_errors_artifact(merged)        # write errors.json BEFORE re-raising
            raise
    return async_wrapper

Retry-on-parse-failure + confidence synthesis — a grading pipeline

# grade_attempt.py — retry once on malformed JSON, independent of sample count
last_raw, grade_data = await _grade_llm_call(system=system, user=user, run_id=run_id, errors=errors)
if grade_data is None:
    last_raw, grade_data = await _grade_llm_call(system=system, user=user, run_id=run_id, errors=errors)
if grade_data is None:
    state = append_error_entry(state, node="grade_attempt",
                               type="json_parse_error", message="malformed JSON after retry")

# self_consistency.py — synthesize N sampled grades into a confidence gate
var = sum((s - mean) ** 2 for s in scores) / len(scores)
cv  = (var ** 0.5) / mean
confident = cv < settings.cv_threshold and mean_conf >= settings.conf_min
flagged   = [] if confident else ["all"]     # high variance ⇒ route to human review

Circuit breaker on a parallel batch — a grading pipeline

# grade_parallel.py
for coro in asyncio.as_completed(tasks):
    r = await coro
    results.append(r)
    failures = sum(1 for x in results if x.get("status") != "graded")
    if results and failures / len(results) > max_error_rate:   # trip the breaker
        errors.append({"node": "grade_parallel", "type": "batch_aborted_high_error_rate"})
        break

---

Real-world example

The concierge — orchestrator agent. The LLM reads "a 5-day trip under $2000 with good weather," routes to the right subset of four specialists, and synthesizes. Flow is decided at runtime; a SummarizingConversationManager keeps the shared conversation from ballooning. This is orchestration where the model owns the flow.

A grading pipeline — graph orchestration, "LLM points, Python reads." A multi-node LangGraph pipeline. The LLM judges each answer; Python owns routing (route_after_confidence), gates (human review on high variance), retries (parse failure), persistence (SqliteSaver checkpointer so runs survive restarts), and a run manifest capturing model/embedding/settings for reproducibility. Batch mode fans out with a calibration phase and a circuit breaker. This is orchestration where Python owns the flow and the LLM is a node-level worker.

The clinical RAG system — routing + blackboard + consensus synthesis. The supervisor routes to a few subspecialties, a per-request SupervisorRunState blackboard caches each specialist's result (a "once-only" directive stops the model re-calling a finished tool), and a schema-validated consensus pass synthesizes agreement/debate/emerging findings.

---

Interview questions companies actually ask

1. What does "orchestration" actually cover? [easy] Routing (who runs), task decomposition (into subtasks), shared state (what's read/written), and result synthesis (merging into one answer) — plus failure handling across all of it. See Kore.ai: orchestration patterns.

2. Orchestrator agent vs. graph orchestration — when each? [medium] Orchestrator agent: LLM decides flow at runtime — flexible, handles varied requests, hard to test/guarantee. Graph (LangGraph): Python owns the topology — deterministic, testable, gate-able. Use graphs when you need guarantees and known steps; use an orchestrator agent when requests are too varied to enumerate. See LangChain: LangGraph.

3. Explain "LLM points, Python reads." [medium] The LLM makes judgments (classify intent, grade an answer, pick a route label); deterministic Python reads those outputs and controls flow, gates, retries, and persistence. It keeps the non-deterministic part small and inspectable — anything that must be guaranteed lives in Python. See Databricks: agent system design patterns.

4. How do you handle partial failure in a multi-step orchestration? [hard] Wrap each node for structured error capture (record + persist before re-raising), retry recoverable errors (e.g., JSON parse) once, gate on confidence to route uncertain results to humans, and trip a circuit breaker when the batch error rate exceeds a threshold. Return worker errors as data, not exceptions. All are standard practice in a robust grading pipeline. See TrueFoundry.

5. The orchestrator is a single point of failure — how do you mitigate? [hard] Keep routing cheap and low-temperature, validate the chosen branch in Python before executing, checkpoint state so a crash can resume rather than restart, and design workers to fail loudly-as-data. Consider a fallback/default route when classification confidence is low. See ByteByteGo: Anthropic's system.

6. How does shared state work in LangGraph, and what's a gotcha? [medium] A TypedDict state is threaded through nodes; each node returns a partial dict that's merged in. Gotcha: concurrent nodes writing the same key need a reducer or they clobber each other; and unbounded state (appending full docs each node) causes context bloat downstream. See LangSmith observability docs.

7. How do you synthesize results from multiple agents reliably? [medium] Don't just concatenate. Use a dedicated synthesis step with explicit criteria — a consensus-extraction LLM pass validated against a schema (the clinical RAG system), or a statistical aggregation like self-consistency's CV gate (a grading pipeline). Validation catches a synthesizer that hallucinates or drops citations.

8. How do you make an orchestrated run reproducible? [hard] Persist a run manifest — model + version, embedding model + dim, RAG backend, key settings (samples, thresholds, top_k), seeds where possible, and IDs — plus a checkpointer so state survives restarts. The grading pipeline writes manifest.json per run for exactly this. See Databricks.

---

When to use / tradeoffs

• Prefer a graph orchestrator when: the workflow is known, you need auditability/compliance, you want cheap unit tests of flow, or you must gate/checkpoint. Cost: less runtime flexibility, more upfront wiring.
• Prefer an orchestrator agent when: requests are open-ended and hard to enumerate, and adaptivity matters more than guarantees. Cost: non-determinism, harder testing, the orchestrator as SPOF.
• Always budget for partial failure — it's not optional at scale. The systems that survive production have per-node error capture, retries, confidence gates, and circuit breakers.
• Watch shared state size. Threading ever-growing state through many nodes is the #1 silent cause of latency and cost creep — summarize or pass references, not raw blobs.

---

Summary + related articles

• Orchestration = routing + decomposition + shared state + synthesis, plus failure handling.
• Two styles: LLM-driven (orchestrator agent) vs. Python-driven (graph). Real systems put a graph on the backbone and an LLM inside nodes.
• "LLM points, Python reads" keeps the guaranteed parts deterministic and the judgment parts small.
• Partial-failure machinery (error capture, retries, confidence gates, circuit breakers, checkpoints, manifests) is what makes orchestration production-grade.

Related: Multi-Agent Patterns · Agent Communication · Debugging & Observability for Agents

Sources: Kore.ai orchestration patterns · LangChain: LangGraph · Databricks: agent system design patterns · TrueFoundry: multi-agent architecture · LangChain: how & when to build multi-agent