MyDot Consultancy

Featured image: AI-generated conceptual illustration. Sources checked 26 October 2026.

A promising demo answers a few chosen questions. An evaluation set asks whether the system can handle the work your users actually bring, including cases that should end with a clarification, refusal or handoff. You do not need a large benchmark program to begin. You need a clear task, realistic cases and a defensible definition of success.

The first set should be small enough for domain experts to inspect carefully and broad enough to expose different failure modes. Avoid choosing an arbitrary case count as a badge of rigor. Coverage and grading quality matter more than the size of an unreviewed spreadsheet.

Start with one service promise

Imagine a hypothetical agent that prepares customer-support reply drafts. It can read an assigned ticket and approved order information, but it cannot send messages or issue refunds. The service promise is to produce an accurate, policy-consistent draft and flag missing information for the support representative.

Write that scope at the top of the evaluation specification. A draft that correctly asks for a missing order number may pass. A draft that invents the number should fail. A run that sends the message should fail the authority check even if the wording is excellent. The scope determines what the grader must inspect.

Anthropic’s evaluation guide explains the relationship between tasks and grading logic. For a business team, the practical lesson is to define the expected behavior before reviewing the model’s output. Otherwise a plausible response can quietly move the goalposts.

Collect cases from the work, with permission

Ask experienced support representatives for examples of routine requests, confusing requests and incidents that required care. Use appropriately authorized data, remove unnecessary personal information and preserve the details that made the case difficult. An anonymized case should remain realistic enough to test the intended behavior.

Include language users actually use: shorthand, spelling errors, incomplete context and multiple questions in one message. Do not rewrite every case into a perfect instruction. An evaluation built only from polished prompts measures a different environment from a busy support queue.

Synthetic cases can fill coverage gaps, but have a domain expert review them. A fabricated policy conflict that could never occur may be useful for robustness testing, yet it should not be presented as representative traffic. Label the origin of each case so the team understands what the set does and does not represent.

Build a coverage map

• Routine: the required evidence exists and the request is within scope.
• Ambiguous: a name, order or intent needs clarification.
• Missing evidence: the available records cannot support a complete answer.
• Conflicting evidence: two sources disagree or apply to different periods.
• Access boundary: relevant information is outside the requester’s authority.
• Tool failure: a dependency times out or returns an error.
• Adversarial content: external text attempts to redirect the agent.

For each category, write the behavior you want and the behavior you must prevent. Do not combine everything into a single correctness label too early. A system can be good at routine replies and poor at handling unavailable data. That distinction should remain visible when deciding whether to launch.

Grade outcomes using the strongest available evidence

Some checks are exact: the draft must name the correct order, include the right status and avoid sending anything. Others require judgment: whether the explanation is clear or the tone is suitable. Use deterministic checks for exact properties and calibrated human review for judgments that cannot be reduced to a reliable rule.

Model-based grading can help scale review, but it needs validation against human decisions. Give the grader explicit criteria and inspect disagreements. A second model agreeing with the first is not independent proof that the answer is correct, especially when both receive incomplete evidence.

The OpenAI evaluation best-practices guide recommends task-specific tests and human calibration of automated scoring. Apply those principles to your workflow rather than copying a numerical threshold from someone else’s example.

Keep tuning cases separate from the final check

Teams naturally learn from failures and change prompts or tools to address them. Preserve some cases for later evaluation rather than repeatedly optimizing against the entire set. Otherwise improved results may show familiarity with the test examples rather than broader capability.

Keep the test environment controlled. Record model, prompt, tool and policy versions. Reset the relevant system state between runs so one attempt does not benefit from another’s changes. For variable behavior, repeat important cases and inspect the spread of outcomes instead of treating one successful run as conclusive.

The OpenAI Agents SDK testing documentation includes workflow testing and failure-injection recipes. These can help engineers verify application behavior separately from live-model evaluations. Both layers matter: a correct model response cannot repair broken approval or error-handling code.

Turn failures into a work queue

Review each failure and assign it to a likely layer: missing source, poor retrieval, confusing tool contract, incorrect model interpretation, weak policy enforcement or grading error. This diagnosis prevents the team from treating every issue as a prompt-writing exercise.

Retain the case and the corrected expectation after a fix. Over time, the evaluation set becomes a record of what the team has learned about the service. Add new production failures carefully, with appropriate privacy controls, rather than letting the set grow into an uncurated dump of conversations.

Use the result to make a bounded decision

An initial evaluation can justify a limited pilot, identify a blocker or show that the simpler workflow is sufficient. It cannot prove universal reliability. State which tasks and conditions were tested, which risks remain and what monitoring will accompany rollout.

The first useful evaluation set is a shared agreement about the work. It gives product owners, engineers and reviewers a way to disagree with evidence. That is a stronger foundation for deployment than a demo everyone enjoyed but nobody could grade.

Sources and further reading

Anthropic: Demystifying evals for AI agents. Task-and-grader concepts; published January 2026.

OpenAI: Evaluation best practices. Task-specific evaluation and human calibration; checked October 2026.

OpenAI Agents SDK: Testing. Workflow testing and failure-injection recipes; checked October 2026.