Agent Evaluation

TL;DR

Evaluating an agent is not the same as evaluating a chatbot. A chatbot gets scored on its final answer; an agent has to be scored on the whole trajectory — did it pick the right tools, call them with the right arguments, stay grounded in real data, and finish the task without burning $4 and 90 seconds per request?

The professional stack looks like this:

1. Task success rate — did the end-to-end job actually get done? (the north-star metric)
2. Trajectory / step evaluation — was each intermediate step correct, not just the last one?
3. Tool-use accuracy — precision/recall/F1 on tool selection and argument correctness.
4. Grounding / hallucination rate — is every claim backed by a retrieved source or tool result?
5. Cost & latency — tokens, dollars, and wall-clock per task.

You measure these with gold sets (deterministic, checksummed), LLM-as-judge (scalable but biased — mitigate position/verbosity bias), and human review (the ground truth, reserved for low-confidence cases). You benchmark against τ-bench, SWE-bench, GAIA, AgentBench, WebArena. And you run both offline (pre-deploy, gold set) and online (production, A/B + live telemetry) evaluation.

If you remember one sentence: an agent that gets the right answer via the wrong trajectory is a latent production incident, not a success.

---

Simple explanation + analogy

Think of grading a new surgeon, not grading a multiple-choice test.

A multiple-choice test only checks the final letter you circled. But you would never certify a surgeon by only checking "did the patient survive?" You check every step: did they scrub in, pick the right instrument, cut in the right place, check vitals, and close correctly? A patient can survive despite a botched procedure (luck), and a great procedure can still lose a patient (bad luck). If you only score the outcome, you can't tell skill from luck — and you'll certify dangerous surgeons.

Agents are the same. An agent asked "what's the cheapest flight to Tokyo next Tuesday?" might return "$812" — correct! — but if it got there by hallucinating a flight number, calling the wrong API, and reasoning from stale cache, that "success" is a coin flip that will fail next week. Trajectory evaluation is watching the surgery, not just the survival rate.

---

Diagram

                         AGENT EVALUATION LAYERS
   ┌───────────────────────────────────────────────────────────────┐
   │  User task:  "Book the cheapest refundable flight to Tokyo"    │
   └───────────────────────────────────────────────────────────────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        ▼                       ▼                       ▼
   ┌─────────┐            ┌───────────┐           ┌───────────┐
   │ Step 1  │  ──────►   │  Step 2   │  ──────►  │  Step 3   │
   │search_  │            │ get_price │           │ book(...) │
   │flights()│            │  (id=..)  │           │           │
   └─────────┘            └───────────┘           └───────────┘
        │                       │                       │
   ┌────┴────┐            ┌─────┴─────┐           ┌─────┴─────┐
   │ TOOL    │            │ GROUNDING │           │  FINAL    │
   │ correct?│            │ price from│           │  ANSWER   │
   │ args ok?│            │ real tool?│           │  correct? │
   └────┬────┘            └─────┬─────┘           └─────┬─────┘
        │                       │                       │
        └───────────────────────┼───────────────────────┘
                                ▼
        ┌───────────────────────────────────────────────┐
        │  SCORECARD                                     │
        │  • task success@1 ...... ✅ 1/1                │
        │  • tool F1 ............. 0.91                  │
        │  • hallucination rate .. 0.0                   │
        │  • cost/task ........... $0.03                 │
        │  • p95 latency ......... 4.2 s                 │
        └───────────────────────────────────────────────┘

   OFFLINE (gold set, pre-deploy)  ||  ONLINE (prod A/B + telemetry)

---

How it works (deep)

1. Task success rate — the north star

Success rate is the fraction of tasks the agent actually completes. The subtlety is defining "completed." Three grading modes, from most to least reliable:

• Programmatic / execution-based (best). The gold answer is checkable by code. SWE-bench runs the agent's patch against a repo's real unit tests: pass = success. This is objective and cheap to re-run, but only exists for tasks with a verifiable oracle (code, math, structured extraction).
• Reference-match. Compare against a gold answer with exact match, F1, or a similarity threshold. Works for extraction and closed-QA; brittle for open-ended text.
• LLM-as-judge. A separate model scores the answer against a rubric. Scales to open-ended tasks but inherits the judge's biases (see the math and interview sections).

Success@k generalizes single-shot success: run the agent k times and count success if any run succeeds. High success@k with low success@1 means the agent is capable but unreliable — a signal to add retries, self-consistency, or a verifier, not to swap the model.

2. Trajectory / step evaluation

The single biggest mistake teams make is scoring only the final answer. An agent is a sequence of decisions, and each is a place to fail. Two ways to score trajectories:

• Exact/loose match against a reference trajectory — did the agent call the expected tools in an acceptable order? "Loose" match allows extra harmless steps and different valid orderings.
• Rubric-scored trajectory (LLM-judge over the transcript) — feed the whole tool-call transcript to a judge with a rubric ("Did every tool call have valid arguments? Did the agent avoid redundant calls? Did it recover from errors?").

Trajectory evaluation is what catches the right-answer-wrong-reason failure — the one that silently rots in production.

3. Tool-use accuracy

Break tool use into two sub-metrics:

• Tool selection — of the tools the agent chose to call, how many were correct, and of the tools it should have called, how many did it call? This is a precision/recall problem (see the math).
• Argument correctness — given the right tool, were the arguments right? A book_flight call with the correct tool but the wrong date is still a failure.

4. Grounding / hallucination rate

For any agent that touches retrieval or tools, every factual claim in the output should trace to a source. Hallucination rate = fraction of claims (or responses) with an unsupported assertion. Measure it by having a judge (or human) check each claim against the retrieved context / tool outputs and label it supported / unsupported / contradicted.

The production-grade design pattern that collapses this metric toward zero is "LLM points, code reads" (shown in the pipeline example below): the LLM is only allowed to select which evidence to use; the actual numbers are read by deterministic Python. The model can't hallucinate a value it isn't allowed to type.

5. Cost & latency

Every eval report needs a cost and latency column, or you will ship an agent that is correct and unaffordable. Track tokens per task, dollars per task (see the cost formula), tool calls per task (proxy for both cost and latency), and p50 / p95 latency. Report these alongside quality — the right model is a point on a cost/quality curve, not a single winner.

Offline vs online

• Offline — a frozen gold set, run before every deploy, in CI. Deterministic, reproducible, gates releases. Cheap and safe, but can drift from reality.
• Online — production telemetry: live success signals (did the user retry? thumbs-up?), A/B tests between agent versions, shadow traffic, and guardrail-trip rates. Reflects reality, but noisy and slow to attribute.

Mature teams run both: offline gates the release, online catches the drift the gold set didn't anticipate.

---

The math

Precision / recall / F1 for tool selection

Treat each candidate tool call as a prediction. For a task:

• TP = tool calls the agent made that were correct (right tool, appropriate moment)
• FP = tool calls the agent made that were wrong or unnecessary
• FN = tool calls the agent should have made but didn't

$$ \text{Precision} = \frac{TP}{TP + FP}, \qquad \text{Recall} = \frac{TP}{TP + FN} $$

$$ F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} $$

• Precision answers: when the agent calls a tool, how often is it the right call? Low precision = the agent is trigger-happy (over-calling, wasting cost).
• Recall answers: of the calls it needed to make, how many did it make? Low recall = the agent is lazy (answering from memory when it should have searched).
• F1 is the harmonic mean — a single number that punishes lopsided precision/recall. Use it to compare agent versions; use the individual components to diagnose.

success@k

Let p be the per-run success probability. If runs are independent:

$$ \text{success@}k = 1 - (1 - p)^k $$

Meaning: the probability that at least one of k attempts succeeds. Example: a flaky agent with p = 0.6 has success@1 = 0.60 but success@3 = 1 - 0.4^3 = 0.936. A large gap between success@1 and success@k is the fingerprint of a capable-but-unreliable agent — the fix is reliability engineering (verifier, retries, determinism), not a bigger model.

Cost per task

$$ \text{Cost}{\text{task}} = \sum{i=1}^{N} \Big( T^{in}i \cdot c{in} + T^{out}i \cdot c{out} + T^{cache}i \cdot c{cache} \Big) $$

where the sum runs over the N model calls in the trajectory, T^{in}/T^{out}/T^{cache} are input / output / cached-read tokens for call i, and c_{in}/c_{out}/c_{cache} are the per-token prices. The N matters: an agent that solves a task in 3 tool-calling turns costs far less than one that solves it in 12, even at the same per-token price. Cost per task, not cost per token, is the number that survives contact with production.

---

Real code

Below is a trimmed gold-set eval harness for a fixed multi-step clinical-guideline review pipeline. This harness scores the deterministic core — a Python digit reader, a reconciler, and a "pick from a list" comparator — offline, with no API keys. It is the gate that decides whether a new extraction tier is allowed to ship.

"""Score the Room C cross-verified extractors against a gold set.

Runs offline (no API keys). It exercises the deterministic core of each new box —
the Python digit reader, the deterministic reconciler, and the list-guarded
comparator picker (with a scripted LLM response per case) — and prints per-extractor
accuracy, the reconciled accuracy, and the cross-verification agreement rate.

This is the Phase-5 gate: enable a new tier's flag only after this shows it helps.
"""

class _ScriptedLLM:
    """Returns a fixed JSON payload — stands in for the comparator-picker LLM."""
    def __init__(self, payload: dict):
        self._payload = json.dumps(payload)
    async def ainvoke(self, messages, **kw):
        class _R: content = None
        r = _R(); r.content = self._payload
        return r

def score_reconcile(cases: list[dict]) -> tuple[int, int, int, int]:
    ok, agree_hits, multi = 0, 0, 0
    for c in cases:
        table = ExtractionResult(counts=_counts(c["table"]), provenance=PROV_CELL, confidence=0.7)
        results = [table]
        if c.get("text") is not None:
            results.append(ExtractionResult(counts=_counts(c["text"]), provenance=PROV_TEXT, confidence=0.6))
            multi += 1
        out = reconcile(results)                       # deterministic Python, not the LLM
        exp = c["expect"]
        correct = (list(out.counts_tuple()) == exp["counts"]
                   and out.provenance == exp["provenance"]
                   and bool(out.review) == bool(exp["review"]))
        ok += int(correct)
        if out.provenance == "cross_verified":
            agree_hits += 1
    return ok, len(cases), agree_hits, multi

def main() -> int:
    gold = json.loads(Path(sys.argv[1] if len(sys.argv) > 1 else "gold.json").read_text())
    p_ok, p_n = score_prose(gold.get("prose", []))
    r_ok, r_n, agree, multi = score_reconcile(gold.get("reconcile", []))
    c_ok, c_n = score_comparator(gold.get("comparator", []))
    e_ok, e_n = score_effect(load_effect_cases())

    print("── summary ─────────────────────────────")
    print(f"  text extractor accuracy      : {_pct(p_ok, p_n)}")
    print(f"  comparator picker accuracy   : {_pct(c_ok, c_n)}")
    print(f"  reconciled output accuracy   : {_pct(r_ok, r_n)}")
    print(f"  effect-measure math accuracy : {_pct(e_ok, e_n)}")
    print(f"  cross-verify agreement rate  : {_pct(agree, multi)}  ({agree}/{multi} multi-source)")

    total_ok = p_ok + r_ok + c_ok + e_ok
    total_n  = p_n + r_n + c_n + e_n
    print(f"  overall                      : {_pct(total_ok, total_n)}")
    return 0 if total_ok == total_n else 1   # non-zero exit fails the CI gate

Three things to internalize from this code:

1. The LLM is scripted (_ScriptedLLM). The eval tests the deterministic boxes around the model, so the score is reproducible and doesn't cost money. The model's job (pick a column from a list) is isolated and stubbed.
2. It prints a cross-verify agreement rate — a grounding metric: when two independent extractors (table + prose) agree, provenance is stamped cross_verified. Disagreement flags human review.
3. return 0 if total_ok == total_n else 1 — the harness is a CI gate. A regression on any gold case fails the build.

---

Real-world example

Gold-set eval as a release gate

A fixed multi-step review pipeline turns literature into guideline documents through a series of LangGraph nodes. Numeric extraction (pulling events/total out of a study's tables) is where a hallucinated digit becomes a clinical error, so the pipeline enforces "LLM points, Python reads": the model selects which cell/sentence, deterministic Python reads the number. The gold set (gold.json) has three sections — verbatim prose sentences with their true 2×2 counts, table+text reconciliation cases, and hand-computed effect-measure math (RR/RD/NNT/IRR/HR). New extraction tiers ship behind flags that are default-off and only flipped on after score.py shows the tier "measurably helps on this set." Combined with task-tier model routing (resolve_model_for_task picks a cheaper model for bulk screening, a stronger one for extraction) and human review on low-confidence reconciliations, the eval harness is the difference between a demo and a deployable clinical tool.

Group recommendation — fairness evaluation with a λ dial

A single-agent trip-recommendation workflow recommends a destination to a group with conflicting preferences and budgets. "Did the group get a good rec?" is not one number — it's a tradeoff between average happiness and minimum satisfaction. A fairness eval for this can use a λ dial:

$$ \text{Score} = \lambda \cdot \underbrace{\overline{h}}{\text{avg happiness}} + (1 - \lambda) \cdot \underbrace{\min{u \in G} h_u}_{\text{worst-off member}} $$

At λ = 1 you maximize the group average (and may strand the budget-conscious member). At λ = 0 you maximize the worst-off member (utilitarian floor). The evaluation also enforces a safe set F — recommendations that violate a hard constraint (exceeding a member's budget cap or an accessibility need) are disqualified regardless of happiness. This shows up directly in the code's conservative merge (get_combined_preferences): any member's accessibility constraint restricts the whole group, and budget caps are unioned across everyone. Evaluating a group agent on average happiness alone would hide exactly the failures (an under-served minority) that matter most — the min-satisfaction term is the metric that makes the agent fair, not just popular.

---

Interview questions companies actually ask

Q1. Why isn't final-answer accuracy enough to evaluate an agent? [easy] Because an agent is a sequence of decisions, and a correct final answer can be reached through a wrong trajectory (hallucinated intermediate, wrong tool, lucky guess). That "success" won't reproduce. You need trajectory/step evaluation to distinguish skill from luck. Modern frameworks explicitly separate final-response eval from trajectory eval for this reason. (LangChain: agent evaluation, Beyond Accuracy — multi-dimensional agentic eval)

Q2. Define precision, recall, and F1 for tool selection and say what each tells you. [medium] Precision = TP/(TP+FP): when the agent calls a tool, how often is it right (low = over-calling). Recall = TP/(TP+FN): of the calls it needed, how many did it make (low = under-calling / answering from memory). F1 is their harmonic mean, a single comparison number. You diagnose with the components and rank with F1.

Q3. What is success@k and when does a big success@1 vs success@k gap matter? [medium] success@k = 1 − (1−p)^k, the chance at least one of k runs succeeds. A large gap (low success@1, high success@k) means the agent is capable but unreliable — the fix is reliability (verifier, retries, determinism), not a stronger model. (SWE-bench, OpenLegion: agent benchmarks)

Q4. Walk me through the major agent benchmarks and what each measures. [medium] SWE-bench: 2,294 real GitHub issues, execution-based (does the patch pass the repo's tests). GAIA: real-world assistant tasks needing multi-step tool use (browser, code, docs). τ-bench (and τ²-bench): customer-service tasks that score policy adherence, not just task completion — a dimension the others miss. AgentBench: 29 LLMs across 8 environments (OS, DB, KG, games, embodied). WebArena / OSWorld: browser and computer-use tasks. (AI Agent Benchmarks 2026, OpenLegion)

Q5. LLM-as-judge is cheap and scalable — what are its failure modes and how do you mitigate them? [hard] Known biases: position bias (favoring the first option — one study found a judge with 72% first-position preference, and pairwise preferences flip ~13.6% of the time on average), verbosity bias (favoring longer answers), and self-preference bias (favoring the judge model's own style). Mitigations: randomize/swap option order and average, use a rubric with explicit criteria instead of a vibe score, calibrate the judge against a human-labeled subset, and reserve human review for low-confidence or high-stakes cases. (Judging the Judges: position bias, Justice or Prejudice? 12 biases)

Q6. What's the difference between offline and online evaluation, and why do you need both? [medium] Offline = frozen gold set in CI, deterministic, gates releases, but drifts from reality. Online = production telemetry (retry rate, thumbs, A/B, guardrail trips), reflects reality but noisy. Offline gates the deploy; online catches what the gold set didn't anticipate. Mature teams run both. (LangChain agent evaluation)

Q7. How do you measure hallucination / grounding in a tool-using agent? [hard] Per-claim: check each factual assertion in the output against the retrieved context / tool results, labeling supported / unsupported / contradicted; hallucination rate = fraction unsupported. The strongest architectural mitigation is "LLM points, code reads" — restrict the model to selecting evidence and let deterministic code produce the actual values, so the model can't emit a number it wasn't allowed to type.

Q8. Your agent scores 95% task success but costs $0.40 and 30s per task. Is it good? [medium] Unanswerable without the cost/latency budget — quality is a point on a cost/quality curve, not a scalar. Report cost per task (Σ over model calls of tokens×price), tool calls per task, and p95 latency next to success rate. A 92%-success agent at $0.03 / 4s often wins in production. Trim trajectory length and route cheap sub-steps to smaller models before touching the top-line model.

Q9. How would you evaluate a group recommendation agent where members conflict? [hard] Don't score average satisfaction alone — it hides an under-served minority. Use a λ-weighted objective trading average happiness against minimum (worst-off) satisfaction, and enforce a hard safety/constraint set (e.g. budget-cap or accessibility filters) that disqualifies violating recommendations regardless of score. This is exactly the group-recommendation fairness eval pattern (λ dial + safe set F). [medium/hard]

Q10. Why script the LLM in a gold-set harness instead of calling the real API? [medium] Determinism, reproducibility, and cost. Scripting the model isolates and tests the deterministic scaffolding (parsers, reconcilers, math) so the eval gives the same score every run, costs nothing, runs in CI, and can be a hard release gate (non-zero exit on any regression). You test the model's judgment separately, with its own smaller, sampled eval.

---

When to use / tradeoffs

Rules of thumb: always evaluate trajectory and outcome. Always report cost/latency next to quality. Use the cheapest reliable grader (programmatic > reference > LLM-judge > human) and escalate only on low confidence. Keep gold sets small, checksummed, and version-controlled so a score is reproducible. Gate flags default-off and flip them only after the eval shows a measurable win.

---

Summary + related articles

Evaluating an agent means scoring the whole trajectory — task success, step correctness, tool F1, grounding, and cost/latency — using a layered grader stack (gold sets → LLM-judge → human review), benchmarked against τ-bench / SWE-bench / GAIA / AgentBench, run both offline (release gate) and online (drift detection). The two hardest-won lessons: a right answer via a wrong trajectory is a latent incident, and cost-per-task is the metric that survives production.

Related articles:

• Production Agents (6-5) — cost control, guardrails, observability, and the "LLM points, code decides" pattern in depth.
• Autonomous Agents (6-5) — why evaluation gets harder as autonomy rises, and the human-review escape hatch.
• Debugging Agents (6-4) — tracing the trajectories your eval flags as failing.
• Memory Management (6-5) — evaluating memory retrieval quality and its cost/latency cost.