Build Your Agent Evals From Real Transcripts

Ask a team to write evaluation cases for its new agent and you'll get the cases the team already worried about. The agent will pass most of them, because the team built it to handle exactly those. The failures that reach customers come from somewhere else, like a phrasing nobody expected or a user who changed their mind halfway through. Those failures are already in your production transcripts, so build the eval suite from them.

A bare prompt won't write a good eval suite

It's tempting to have a coding agent write the evaluation. A study posted to arXiv in May, An Empirical Study of Automating Agent Evaluation, tested that. Without evaluation know-how built in, frontier coding models reached only a 30% execution success rate, and the evaluations they did produce were over-engineered, averaging more than 12 metrics per agent. The authors' own system, which encodes evaluation know-how as reusable skills and works from traces, got its evaluation code running and producing meaningful results on the first try 65% of the time.

From transcript to test case

Viacheslav Dubrov's guide to turning traces into test suites, published in June, lays out a sequence worth copying. Its short version: "Mine production failures before brainstorming additional eval cases."

Start by capturing complete traces: the user's input, every tool call with its arguments and output, and the final state. Strip personal data before anything leaves production.

Then label the failures. Give each one a category, such as wrong tool, right tool with wrong arguments, gave up too early or made something up. Group them, and keep one representative case per group, since fifty near-identical failures only slow the suite down.

Store the cases in a versioned dataset with the expected outcome next to each. When someone fixes a failure, the case stays in the set so the fix can't quietly come undone.

Check with code first, judge second

Most of what goes wrong in an agent run can be checked with ordinary code. Did it call the lookup before the refund? Were the arguments valid? Did it stop after three retries? Did the record end up in the right state? Write those checks as plain assertions. They're fast and repeatable.

Use an LLM judge only for what code can't decide, like whether a reply answered the question or kept to the right tone. Before trusting the judge, run it against a few dozen cases a person has already labelled, and see how often it agrees.

Run it where changes happen

The suite earns its keep in CI. Every change to a prompt or tool definition runs against it, and so does every model swap, and a drop in the pass rate blocks the merge.

In production, sample a share of live traffic and score it with the same checks. When a new kind of failure shows up there more than once, label it and move it into the offline set, so that after a few months most of the cases in the suite come from production.

A first week

Pull a week of transcripts from your busiest agent. Read fifty of them, label the failures, and pick the five categories that appear most. Write one test case for each, checked by plain assertions wherever possible, and wire them into the pipeline that ships prompt changes.