How to Evaluate an AI Agent (Before It Ships)

TL;DR

• Agent evaluation is not LLM evaluation. You are evaluating a sequence of decisions with real consequences, not the quality of a single text output.
• Three layers matter: component, trajectory, and outcome. You need all three. Skipping any one hides a category of failure.
• Two agents can reach the same correct answer with completely different production risk. The trajectory is where most agent bugs live.
• Most teams are at Level 0 or 1 on the agent evaluation maturity model and don’t realize it. The gap between “we tested it” and “we have a regression suite” is where production failures live.
• Start here: pick 20 real tasks, hand-author the expected trajectories, build a simple logging and replay harness, and run it on your next change.

Abstract

Most teams evaluate agents the way they evaluate LLMs: run prompts, score outputs with a rubric, check accuracy. That approach breaks down with agents.

Agents take actions. They call tools, loop, branch, write to real systems. The failure modes are different, the observability is different, and the stakes are higher because an agent doesn’t just generate text — it makes decisions that compound. A 90% per-step success rate gives you roughly 35% end-to-end success over a ten-step trajectory. Small per-step losses multiply.

Worse, agent failures are often silent. The output looks plausible. The action was executed. Nothing crashed. But the outcome was wrong, or the agent took twelve steps to do what should have taken four, or it called a write operation it shouldn’t have. If your evaluation only checks the final output, you are blind to all of it.

This post builds a practical evaluation framework that teams can actually run: what to measure, how to measure it, what breaks, and what to build first.

Why Agent Evaluation Is a Different Problem

LLM evaluation asks: did the model produce a good answer for this prompt?

Agent evaluation asks: did the agent make the right sequence of decisions to accomplish a goal — and can it do that reliably, safely, and at a reasonable cost?

The distinction is not semantic. It changes what you measure, how you measure it, and what you do with the results.

Consider the compounding error problem. If each step in an agent’s trajectory has a 90% success rate — the tool call works, the retrieval returns the right document, the planner picks the right next step — the probability that all ten steps succeed is 0.9^10, or about 35%. A system that looks 90% reliable at the component level is 35% reliable end-to-end. That math is the core reason agent eval cannot be component eval alone.

Then there is the silence problem. An LLM that gives a wrong answer gives a wrong answer — you can see it. An agent that takes a wrong action can execute that action successfully. The tool call completed. The database row was updated. The API returned 200. The agent reports success. The outcome is wrong, but nothing in the system flags it as wrong because the action itself succeeded.

If your evaluation only checks the final output, you catch some of these. If it only checks whether the agent “completed the task,” you catch even fewer. The failures that slip through are the ones where the agent reached the right answer through a dangerous, expensive, or fragile path — and you will not know until it breaks in production.

The Three Layers of Agent Evaluation

Agent evaluation operates at three layers. They are not alternatives. You need all three, and skipping any one creates a blind spot.

Layer 1: Component Evaluation

Each tool, retrieval step, and model call is evaluated in isolation.

• Does the search retriever return the right documents for a given query?
• Does the code interpreter run without error and return correct output?
• Does the planner select the right next step given the current state?

Component evaluation catches the obvious breakage: a tool that returns garbage, a retriever with bad recall, a planner that hallucinates steps that don’t exist in the toolset. It is the foundation, and it is where most teams start because it maps cleanly onto familiar LLM eval patterns.

But it is not sufficient. A system where every component passes its individual tests can still fail end-to-end, because component tests don’t capture how the components interact, how errors propagate, or how the agent behaves when a component returns a technically correct but contextually wrong result.

Layer 2: Trajectory Evaluation

Evaluate the full path the agent took, not just the final output.

• Did it take unnecessary steps?
• Did it recover from a failed tool call, or did it retry the same call five times?
• Did it loop?
• Did it call the right tool at the right time with the right arguments in the right order?

This is where most agent bugs live. Two agents can reach the same correct answer — one in four clean steps, one in twelve steps with three retries and a hallucinated tool call. If you only check the final output, both look identical. Same accuracy score. Completely different production risk.

The trajectory layer captures what the output layer cannot: efficiency, recovery, and the specific decision points where the agent went wrong. Metrics at this layer include step count, tool-call accuracy (right tool, right arguments, right order), recovery rate, loop frequency, and wasted-action ratio — steps that contributed nothing to the final outcome.

Layer 3: Outcome Evaluation

Did the agent accomplish the user’s goal?

This is the binary question stakeholders care about: did it work? It is also the bluntest instrument. A high outcome score can hide a dangerously inefficient trajectory. The agent completed the task, but it took twenty steps, called the wrong tool six times, and spent $3.00 on tokens to do what should have cost $0.05.

Outcome evaluation includes binary task success, partial credit (did it complete 3 of 5 sub-tasks?), and cost-adjusted success (did it accomplish the goal within an acceptable budget?). It is necessary — it is the ground truth of whether the agent is useful — but it is not sufficient on its own because it masks how the agent got there.

The Stacking Principle

The three layers stack. Component-level problems corrupt trajectories. Trajectory problems inflate cost and latency. Outcome scores alone mask both. If you can only invest in one layer, invest in trajectory evaluation — it is the layer that captures the failure modes unique to agents. But the goal is all three, because each layer catches a category of failure the others miss.

What to Measure

Resist the temptation to build one composite score. Agents fail in different ways for different reasons. A composite score hides the failure mode you most need to see. Instead, track a small set of metrics that each capture a distinct dimension of agent behavior.

Metric	What it captures	Why it matters
Task success rate	Did the agent accomplish the goal?	Ground truth of agent usefulness
Step efficiency	Actions taken vs. minimum needed	Fragility and cost; high ratio signals poor planning
Tool-call accuracy	Right tool, right arguments, right order	Core decision quality
Recovery rate	Does it self-correct after a failed action?	Resilience; low recovery means brittle agents
Cost per successful task	Tokens, API calls, wall-clock time	Production economics
Failure mode distribution	Where does it break?	Tells you what to fix next
Consistency	Same input, same trajectory?	Reliability; high variance means unpredictable agents
Safety surface	Rate of dangerous or out-of-scope actions	Production risk

Design note: consistency is the most undervalued metric. Run the same task five times. If the agent takes five different trajectories, your system is not reliable — it is lucky. Variance is the enemy of production agents because it means you cannot reproduce failures, which means you cannot debug them.

How to Measure It

No single method captures everything. Use a combination.

Golden trajectories. Hand-author the correct path for a core set of tasks. Compare agent paths against them. Not every task has one right path — but enough do to catch regressions. This is the highest-signal method because it evaluates the trajectory directly, not a proxy for it. The cost is authoring effort, which is why you keep the golden set small (20-50 tasks) and focused on the most common user goals.

LLM-as-judge. Useful for step-level reasoning quality and trajectory scoring when you cannot hand-author every path. But it needs a grounded rubric — not “rate this trajectory 1-5” but “did the agent select the correct tool at each step? did it provide the correct arguments? did it take unnecessary steps?” — and human spot checks. Never the only evaluator. LLM-as-judge is a scaling tool for human judgment, not a replacement for it.

Replay-based testing. Record real agent runs, replay them against updated tools, models, or prompts to detect regressions. This is cheap, high-signal, and underused. The insight: you don’t need to re-run the agent to evaluate a change — you can replay the recorded trajectory through the new version and see if the outcomes change. This turns evaluation from an expensive batch job into a fast feedback loop.

Red-team task suite. Adversarial tasks designed to trigger common failure modes: infinite loops, scope creep, tool misuse, hallucinated tool calls, premature termination. These are not your typical user tasks — they are stress tests. The goal is to find the edges where the agent breaks before users find them for you.

Human evaluation. Still necessary for outcome scoring on complex tasks where the definition of “correct” is ambiguous. Budget for it. Automate the rest. The trap is treating human eval as a fallback — it should be a deliberate, scheduled part of the pipeline for the subset of tasks where automated scoring is unreliable.

The Test Set: What to Put in It

Your eval is only as good as your test set. A test set of toy demos will produce a system that works on toy demos.

• Build from a representative distribution of real user tasks, not synthetic examples you made up to look good.
• Include failure-prone cases: multi-step reasoning, ambiguous instructions, tasks requiring tool chaining, and tasks with no valid solution (does the agent know when to stop?).
• Version the test set. Tag each task with difficulty, expected step count, tool requirements, and an expected cost ceiling. This metadata turns raw results into actionable signal — you can slice by any dimension.
• Size: 50-200 tasks for a meaningful signal. 10 is not enough — you are measuring failure modes, and rare failures need sample size to appear. 500 is a maintenance burden unless you have automation for test set expansion.

The most important test set design principle: include tasks that should fail. An agent that never refuses a task, never says “I can’t do this,” and never recognizes an impossible request is an agent that will confidently do the wrong thing in production. Tasks with no valid solution test the agent’s ability to recognize its own limits — and most agents fail this test.

Common Failure Modes (and How to Catch Them)

These are the failure patterns we see repeatedly in agent systems. Each one is invisible to output-only evaluation.

The confident wrong answer. The output looks correct. The action was taken. The result is wrong. This is the signature agent failure mode — the system did everything it was supposed to do, and the outcome is still wrong because the agent’s reasoning was flawed at a step that no component test would catch. Catch with outcome evaluation and human review of edge cases.

The unnecessary loop. The agent retries a failing tool five times, eventually succeeds (or switches strategy), and reports success. The task is complete, but the cost is 5x what it should be, and the underlying issue that caused the failure is still present. Catch with trajectory evaluation — step count, retry rate, and loop frequency metrics will flag this immediately.

The hallucinated tool call. The agent calls a tool with arguments that look right but reference nonexistent resources — a document ID that doesn’t exist, an API endpoint that isn’t part of the toolset, a parameter value that is syntactically valid but semantically wrong. The tool returns an error, the agent may or may not recover, and the failure is only visible if you log tool calls and their results. Catch with component evaluation (argument validation) and trajectory evaluation (tool-call accuracy).

The scope creep. The agent does more than asked. It “helpfully” modifies data it shouldn’t, calls write operations outside the task scope, or takes actions that are technically reasonable but were not requested. This is the most dangerous failure mode because the agent is succeeding — just at a task nobody asked it to do. Catch with safety-surface metrics and red-team tasks that test whether the agent stays within explicit boundaries.

The variance problem. Same task, different trajectory every time. The agent might succeed on 4 of 5 runs and fail on the 5th, with no obvious difference in input. High trajectory variance means you cannot reproduce failures, which means you cannot debug them, which means you cannot ship with confidence. Catch with consistency metrics — run each task N times and measure trajectory divergence. If your agent’s behavior is unpredictable, your evaluation results are unreliable regardless of the numbers.

Building the Evaluation Pipeline

Evaluation is infrastructure, not a one-time activity. The pipeline has four parts.

Logging. Every agent run emits a structured trajectory log: tool calls, arguments, results, timestamps, token counts. Without this, you have nothing to evaluate. This is the prerequisite for everything else — if your agent doesn’t log its trajectory, build that first. Everything downstream depends on it.

Evaluation harness. A script that replays logged trajectories through your scoring functions and produces a report. This doesn’t need to be sophisticated — a Python script that reads trajectory logs, runs your metrics, and outputs a summary table is enough to start. The goal is to make evaluation repeatable, not to build a platform.

Regression suite. Run the eval suite on every prompt change, tool update, or model swap. This is CI for agents. The discipline matters more than the tooling: if a change to the system prompt causes a regression in trajectory efficiency, you need to know before the change ships, not after.

Dashboard. Task success, step efficiency, cost, failure mode breakdown — trended over time, not just a snapshot. The dashboard answers two questions: “are we getting better?” and “where are we getting worse?” If you can only answer the first, you are missing regressions. The dashboard is what turns evaluation from a pre-launch gate into an ongoing feedback loop.

Cadence. Full eval suite on every meaningful change (prompt, tool, model). Smoke test (10-task subset) on every commit. The full suite catches regressions; the smoke test catches catastrophic failures before they reach the full suite. Neither is optional.

When the Numbers Look Fine But the Agent Is Wrong

This is the most important section. Aggregate metrics can be green while specific task types silently fail. You will hit this situation. Here is what to do.

Slice the data. Break down success rate by task type, step count, tool combination. The bug is in a slice. An overall success rate of 85% can hide a 40% success rate on multi-step tasks that require tool chaining — the slice that matters most for production reliability.

Read the trajectories. Pick failed tasks and read the full path. Metrics tell you where to look. Reading tells you why. There is no substitute for this. If you skip it, you are guessing at the cause.

Check distribution drift. Is your test set still representative of what users actually ask? A test set that was accurate six months ago may no longer cover the most common user tasks if the product has evolved. Stale test sets produce false confidence — your numbers look good because you are testing the wrong things.

The conviction: an evaluation you cannot debug is an evaluation you do not have. Debuggability is a first-class design goal for your eval pipeline. If your metrics tell you something is wrong but you cannot trace the failure to a specific step, tool call, or decision point, your evaluation is not actionable. It is a number on a dashboard. Build the pipeline so that a red metric leads to a trajectory you can read.

The Agent Evaluation Maturity Model

Most teams are at Level 0 or 1 and don’t realize it. The gap between “we tested it” and “we have a regression suite” is where production failures live. Use this model to figure out where you are and what to build next.

Level 0: Vibe-based. “Seems to work in demos.” You run a few examples by hand, the agent does something reasonable, and you ship it. There is no test set, no metrics, no logging. You are relying on luck and the fact that your demo inputs are easy.

Level 1: Ad hoc. Manual testing on a few examples before shipping. Slightly better than Level 0 — you are at least running the agent on more than a demo — but there is no systematization. No test set, no regression checks, no trajectory logging. Every release is a fresh roll of the dice.

Level 2: Systematic. Curated test set, automated scoring, regression checks on changes. You have 50+ tasks, you log trajectories, and you run the suite before every change. You can detect regressions. This is where most teams should aim first — it is the minimum viable evaluation system for a production agent.

Level 3: Continuous. Eval runs in CI, trajectory-level metrics, failure mode tracking, red-team suite. Evaluation is not a pre-launch gate — it is an ongoing system that runs on every change and surfaces trends over time. You know not just whether the agent works today but whether it is getting better or worse.

Level 4: Adaptive. Eval data feeds back into agent improvement. Failed trajectories inform prompt updates, tool fixes, and test set expansion. The system learns from its own failures — not the agent itself, but the team operating it. Each production failure produces a new test case, and the eval suite grows smarter over time. This is the goal, not the starting point.

The jump from Level 1 to Level 2 is the most important one. It is the difference between hoping your agent works and knowing whether it works. Everything past Level 2 is iteration on top of that foundation.

What to Build First

If you remember one thing from this post: evaluate the trajectory, not just the output.

If you build one thing this week: pick 20 real tasks — the actual things your users ask your agent to do — and hand-author the expected trajectories. Build a simple logging layer that records every tool call, argument, and result. Write a script that replays those trajectories through a basic scoring function and produces a summary table. Run it on your next change.

That is Level 2. It is not glamorous. It is not a platform. It is a Python script and a folder of markdown files. But it will catch regressions that output-only evaluation misses, and it will give you something most agent teams don’t have: visibility into how the agent is actually behaving, not just how its outputs look.

The teams that ship reliable agents are not the ones with the best models. They are the ones with the best feedback loops. Evaluation is the feedback loop. Build it before you need it, because by the time you need it, it’s too late.