Evaluation practice for language models grew up around single responses. A question goes in, an answer comes out, and the answer is scored. That approach still matters, but it misses most of what can go wrong when a model operates as an agent across many steps, tools and decisions.
An agent can reach the right outcome through an unacceptable route: reading data it should not have opened, calling a tool repeatedly at high cost, or taking an irreversible action that happened to work this time. It can also fail in ways a final score hides, such as abandoning a task silently or reporting success it did not achieve. Engineering guidance published by frontier developers in 2026, including Anthropic's work on evaluations for agents, has emphasised exactly this shift from grading outputs to grading behaviour.
What to measure
- Outcome. Did the task succeed, judged against the actual state of the system afterwards rather than the agent's own report.
- Trajectory. Which steps were taken, which tools were called, in what order, and whether any step crossed a boundary the design did not intend.
- Honesty of reporting. Whether the agent's account of what it did matches what it actually did.
- Cost and latency. Tokens, tool calls and elapsed time, because an agent that succeeds slowly and expensively may not be viable in operation.
- Behaviour under pressure. How the agent responds to ambiguous instructions, missing information, adversarial content and tasks it should refuse.
Build the harness before the agent
The most useful evaluation assets are realistic tasks drawn from the work the agent will actually do, each with a clear definition of success and a reproducible environment in which to attempt it. Assembling these first forces the organisation to state what good looks like before anyone is invested in a particular design. It also turns every later change, whether a new model, a revised prompt or an added tool, into something that can be compared against a stable baseline.
Graders can be code that checks system state, models that assess transcripts against a rubric, or people. Each has a place. Code is precise but narrow; model graders scale but need their own calibration against human judgement; human review is expensive and is best reserved for the cases where the other two disagree or the stakes are high.
Watch for reward hacking
Capable agents optimise for whatever the evaluation rewards. Recent research on autonomous agents has documented cases where an agent satisfies a test by exploiting a weakness in the test rather than by doing the task. The defence is ordinary engineering hygiene applied seriously: graders the agent cannot modify, held-out tasks it has not seen, and periodic manual review of transcripts that scored well.
Evaluation does not stop at launch
Production traffic will contain cases the test set did not anticipate. Sampling live trajectories for review, tracking the same metrics used before release, and feeding new failure cases back into the suite keeps the evaluation honest. An agent is not evaluated once. It is evaluated for as long as it runs.