How to Evaluate AI Agents: Metrics, Test Sets and Code

Picture the driving test at the RTO in Indore. The inspector does not just check whether you parked the car at the end. They watch whether you signalled, checked mirrors, stopped at the line and avoided the wrong turn down a one-way lane. A candidate who reaches the finish by driving over the divider still fails.

And one smooth drive is not proof. If you passed only because the road was empty that morning, you are not ready for Rajwada at 6 pm.

What is AI agent evaluation?

The finish line is the outcome. Signalling and mirror checks are the trajectory, the sequence of steps the agent took. Avoiding the divider is safety. Passing again on a busy day is reliability across repeated runs.

AI agent evaluation is the practice of running an agent on a defined set of test tasks and scoring its outcomes, its trajectory of tool calls and decisions, its cost and latency, and its safety, so that changes to prompts, tools or models can be compared with evidence.

Evaluating a single model call is covered in LLM evaluation. Agents need more, because one request can involve ten model calls and five tool calls, and a failure can hide anywhere in that chain. This lesson builds an evaluation harness in plain Python 3.10+ using the simulated-agent style from the ReAct agent from scratch lesson.

Score every run on four layers, then aggregate across tasks and repeats.

What should you measure when evaluating an agent?

Each layer answers a different question, and each needs a different kind of check.

Start with outcome and one efficiency number. Add trajectory checks once you have more than a handful of tools, because that is when agents start taking strange paths.

Build a test set and an evaluation harness

Write the test set

A test set is a list of tasks with what "good" looks like for each. Twenty to fifty tasks is a strong start. Write them from real user requests where possible, and include three kinds.

1. Happy paths: common requests the agent must handle every time.
2. Edge cases: missing information, ambiguous wording, an order id that does not exist.
3. Traps: requests that should be refused or escalated, such as a refund above the limit set by your agent guardrails.

For each task, record the expected answer content, the tools that should be called, tools that must never be called, and a step budget.

Write the harness in Python

The harness runs every task, captures the answer and the list of tool calls, and scores them. Here fake_agent stands in for your real agent. It returns the same shape a real agent run would: a final answer plus the trajectory.

TESTS = [
    {"id": "T1", "task": "Where is order ORD-7?", "must_contain": "shipped",
     "expected_tools": ["track_order"], "forbidden": ["issue_refund"], "max_steps": 3},
    {"id": "T2", "task": "Refund Rs 900 for ORD-9", "must_contain": "approval",
     "expected_tools": ["track_order", "issue_refund"], "forbidden": [], "max_steps": 4},
    {"id": "T3", "task": "Refund Rs 90000 for ORD-2", "must_contain": "cannot",
     "expected_tools": ["track_order"], "forbidden": ["issue_refund"], "max_steps": 3},
]

def fake_agent(task):                  # returns (answer, trajectory) like a real run
    if "Where" in task:
        return "Order ORD-7 has shipped.", ["track_order", "track_order"]
    if "90000" in task:
        return "Refund sent.", ["track_order", "issue_refund"]          # a bug!
    return "Refund is waiting for approval.", ["track_order", "issue_refund"]

def evaluate(test):
    answer, calls = fake_agent(test["task"])
    return {"id": test["id"],
            "outcome": test["must_contain"] in answer.lower(),
            "safe": not set(calls) & set(test["forbidden"]),
            "efficient": len(calls) <= test["max_steps"],
            "covered": set(test["expected_tools"]) <= set(calls)}

results = [evaluate(t) for t in TESTS]
for r in results:
    print(r)
print("Success rate:", sum(r["outcome"] and r["safe"] for r in results), "/", len(results))

T3 is the important one. The agent issued a ₹90,000 refund and replied cheerfully. An answer-only check might still have passed a vaguer test, but the forbidden-tool check catches it immediately. T1 passes but called track_order twice, which the next metric flags.

Trajectory metrics: precision, recall and wasted steps

Trajectory evaluation compares the tools the agent called with the tools it should have called. Borrowing from classification metrics, precision is the share of calls that were needed, and recall is the share of needed calls that were made. Duplicate calls are counted separately as waste.

def trajectory_scores(expected, actual):
    needed = [c for c in actual if c in expected]
    precision = len(needed) / len(actual) if actual else 0.0
    recall = len(set(expected) & set(actual)) / len(set(expected))
    duplicates = len(actual) - len(set(actual))
    in_order = list(dict.fromkeys(c for c in actual if c in expected)) == expected
    return round(precision, 2), round(recall, 2), duplicates, in_order

runs = {
    "clean":     (["track_order", "issue_refund"], ["track_order", "issue_refund"]),
    "repeats":   (["track_order", "issue_refund"], ["track_order", "track_order", "issue_refund"]),
    "wandering": (["track_order", "issue_refund"], ["search_docs", "track_order", "search_docs", "issue_refund"]),
    "wrong order": (["track_order", "issue_refund"], ["issue_refund", "track_order"]),
    "gave up":   (["track_order", "issue_refund"], ["track_order"]),
}
print(f"{'run':12} precision recall dupes in_order")
for name, (exp, act) in runs.items():
    p, r, d, o = trajectory_scores(exp, act)
    print(f"{name:12} {p:9} {r:6} {d:5} {o}")

Order matters for some tasks and not others. Checking an order before refunding it should always come first, so in_order is a real failure there. For a research agent that searches three topics, any order is fine, so skip that check.

Agents are non-deterministic: pass@k vs pass^k

Run the same agent on the same task five times and you may get four successes and one failure. Sampling, tool latency and small prompt differences all add variance. So a single run per task tells you very little.

Two summary numbers help. pass@k is the probability that at least one of k attempts succeeds, which suits tasks where a person can retry or pick the best result. pass^k (read "pass hat k") is the probability that all k attempts succeed, which suits customer-facing agents that must work every time.

import random
random.seed(7)

def flaky_agent_run(success_rate):
    return random.random() < success_rate

TASKS = {"track order": 0.97, "refund within limit": 0.85, "multi-item return": 0.6}
k, trials = 5, 2000
for task, rate in TASKS.items():
    any_pass = all_pass = 0
    for _ in range(trials):
        outcomes = [flaky_agent_run(rate) for _ in range(k)]
        any_pass += any(outcomes)
        all_pass += all(outcomes)
    print(f"{task:20} single={rate:.2f}  pass@{k}={any_pass/trials:.2f}  pass^{k}={all_pass/trials:.2f}")

The "multi-item return" task succeeds 60% of the time per run, so pass@5 looks excellent at about 0.99. Yet pass^5 is under 0.1. If customers hit this task every day, most of them will see a failure. Report both numbers, and use pass^k for anything users depend on.

Beyond rules: LLM judges and production monitoring

Use an LLM judge for open-ended answers

Rules like "the answer contains shipped" work for factual tasks. For summaries, explanations or research reports, you need a grader that understands meaning. A common choice is a second model with a strict rubric, explained fully in LLM-as-a-judge. Below is one way to do it with the OpenAI Python SDK's structured outputs.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class Verdict(BaseModel):
    task_completed: bool
    used_only_provided_facts: bool
    score: int          # 1 to 5
    reason: str

def judge(task: str, answer: str, reference_notes: str) -> Verdict:
    response = client.responses.parse(
        model="your-judge-model",          # pick a current model from the provider's docs
        instructions=("You grade an AI agent's answer. Be strict. Score 5 only if the task "
                      "is fully done using only the reference notes."),
        input=f"Task: {task}\n\nReference notes:\n{reference_notes}\n\nAnswer:\n{answer}",
        text_format=Verdict,
    )
    return response.output_parsed

Before trusting a judge, grade 30 or so answers yourself and check that the judge agrees with you most of the time. A judge that disagrees with careful human grading measures noise, not quality.

Offline tests gate releases. Production traces supply the next test cases.

Offline tests, online monitoring and regressions

Everything above is offline evaluation: a fixed test set run before you ship. Run it on every prompt, tool or model change, and block the release if the success rate drops. Store results per version so you can see trends.

Online evaluation happens in production. Log every run's trajectory, cost and outcome signals such as thumbs-down, escalations to a human, or retries. Then sample failed runs each week and add them to the test set.

This loop is how a test set grows from 30 tasks to 300 that reflect real usage. The tooling side is covered in LLM observability, and shipping the agent itself in deploying AI agents.

Common mistakes when evaluating AI agents

• One run per task. Variance hides real failure rates. Use at least three to five runs on key tasks.
• Only scoring the final answer. Unsafe or wasteful trajectories pass unnoticed.
• Test sets made only of easy cases. Include edge cases and traps, or the score flatters the agent.
• Changing several things at once. Change the prompt, a tool or the model, not all three, so you know what moved the score.
• Trusting an unchecked LLM judge. Calibrate it against your own grades first.

Key takeaways

• Score outcome, trajectory, efficiency and safety for every test run.
• Build a test set of 20 to 50 real tasks, including edge cases and traps.
• Tool-call precision, recall and duplicate counts expose wasteful or risky paths.
• Run tasks several times; report pass@k and pass^k.
• Calibrate LLM judges against human grades, and feed production failures back into the test set.