Build an Agent Evaluation Dataset from Real Work

An agent evaluation dataset should capture work the product must perform, including the conditions under which it must stop. A collection of clever prompts is not enough. Tool-using agents change external state, and their final messages can sound successful even when the underlying task failed.

For a support agent, “I updated your delivery address” should be checked against the order record. For a coding agent, a persuasive explanation should not compensate for a broken test. Begin with observable outcomes and build the conversation around them.

Write a task contract

Each case needs an initial state, a user goal, available tools, relevant policy, and a success definition. Also define forbidden outcomes. A task can finish the requested update while accidentally modifying another record; a success-only check would miss that damage.

An illustrative case might be:

id: address-change-before-dispatch
initial_state: fixtures/order-awaiting-dispatch.json
user_goal: Change the delivery address for my current order
required_outcome: The authorized order has the requested address
forbidden_outcomes:
  - A different customer's order changes
  - Payment details are disclosed
  - The agent claims a failed update succeeded

This is a proposed dataset format, not a framework configuration. Make the fixture and grader agree about what “current order” means, particularly if several orders exist.

Anthropic's agent evaluation guide distinguishes tasks, trials, graders, transcripts, and outcomes. That vocabulary helps avoid conflating a good-looking conversation with a correct final environment state.

Sample ordinary work and difficult boundaries

Start with real failure reports and common workflows, after removing sensitive data. Include ordinary cases because they represent product value. Add boundary cases because they define what the agent must not do.

For the address-change example, vary dispatch status, account ownership, incomplete addresses, ambiguous order references, and tool failures. Include a case where the correct action is asking a clarifying question and one where the correct action is refusing an unavailable change.

Do not generate dozens of paraphrases of one easy task and call the dataset diverse. Track scenario coverage explicitly. Different wording matters, but different state and policy conditions often reveal more consequential defects.

Use synthetic fixtures when production data is unsuitable. Clearly label them as synthetic and review them for realism. A fabricated customer record can be useful without pretending it was observed in production.

Grade outcomes and important constraints

Prefer deterministic checks when the result is concrete: database fields, file contents, tests, allowed tool arguments, and absence of unauthorized mutations. Use a model grader for aspects that need interpretation, such as whether an explanation accurately communicates a limitation.

Keep critical constraints separate from a weighted quality score. A friendly tone should not offset cross-account access. A task with a forbidden side effect should fail the relevant safety criterion even if it completed the requested work.

τ-bench evaluates tool-using agents through dynamic user interactions and checks final database state against the intended goal. Its design is useful inspiration for stateful product evaluations; its reported model results should not be treated as measurements of your agent.

When using a model grader, give it the necessary evidence and a clear “insufficient information” option. Calibrate it against human judgments on representative cases. Otherwise, you may automate a grader that simply rewards confident wording.

Reset the environment between trials

An agent should not inherit a completed task from an earlier run. Restore fixtures, isolate tool state, and control external dependencies. If a previous trial changed the order address, the next trial might appear successful without doing the required work.

Record the model, prompt, tool definitions, application version, fixture version, and relevant configuration for every run. These values define what you evaluated. A result labeled only with a model name is difficult to reproduce.

Simulated users also require review. If a user simulator supplies missing information too readily, or changes its goal during the conversation, it can distort results. Inspect transcripts where a surprising score may reflect the simulator rather than the agent.

Measure repeatability and failure shape

Run repeated trials on cases where inconsistent behavior matters. Report the number of trials and the distribution of outcomes. A task that passes once and fails four times needs a different interpretation from one that consistently succeeds.

Separate capability tests from regression tests. A challenging exploratory suite helps identify the next development target. A stable regression suite protects behavior the product already promises. Mixing both into one headline percentage can make a release look worse merely because you added harder tasks.

Preserve per-case results and failure categories: misunderstood goal, wrong tool, invalid arguments, unavailable data, policy violation, incomplete execution, or incorrect reporting. Those categories turn the evaluation into a repair queue.

Keep a holdout and review the grader

Reserve cases that are not used for routine prompt tuning. If developers repeatedly optimize against the same small set, improvements may become specific to its wording and fixtures. Refresh coverage as real workflows evolve.

Treat disputed results as data about the test. Sometimes the expected answer is incomplete, the environment is inconsistent, or the agent found a valid alternative. Correct the rubric transparently and retain its version history.

The RAG evaluation guide separates retrieval quality from answer quality; agent evaluation adds the outcome of actions. Include negative cases derived from the tool-permission boundaries so success requires authorized behavior as well as task completion.

A useful dataset becomes a living specification of your product. It says what successful work looks like, what uncertainty should trigger a pause, and what must remain protected even when the agent makes a mistake.