Featured image: AI-generated conceptual illustration. Sources checked 26 October 2026.
Imagine two agents in a controlled test returning the same sentence: “The customer’s renewal contact has been updated.” One changed the correct account. The other changed a similarly named account and left the requested record untouched. A response-only evaluation could score both as successful. A business cannot.
Once an agent can use tools, evaluation must include what happened outside the conversation. The final answer remains important, but it is only one artifact. Examine the destination state, the path to that state and any side effects that should not have occurred.
Build a test with a visible before and after
Consider a hypothetical CRM task that changes a renewal contact after approval. The test fixture contains two accounts with similar names, several contacts and a known initial relationship. The expected result identifies exactly which relationship may change and which records must remain untouched.
Run the agent in an isolated test environment. Afterward, inspect the stored relationship directly. Also check the audit trail for other writes. A correct final value does not excuse unrelated modifications, and a clean conversation does not prove that no hidden tool action occurred.
Anthropic’s agent evaluation guidance discusses grading work that changes an environment. The general principle is to match the evidence to the task. For a record update, the record matters more than a self-reported description of the update.
Use separate scorecards for separate risks
Task outcome: did the intended record reach the required state? For a draft-only task, did the right artifact exist without being sent? For a research task, did the output contain the required supported findings? Define success in terms of the actual service promise.
Authority: did the run stay within the user’s permissions and the granted scope? A correct update made without required approval is still a control failure. Keep this dimension visible rather than averaging it into a broad quality score.
Side effects: did the agent change anything unrelated, expose information or trigger a downstream action? Some side effects are delayed, so the test may need to inspect queues or callbacks as well as the immediate response.
Communication: did the assistant accurately explain the result and remaining uncertainty? A task can succeed while the final message is misleading, just as a polished message can accompany a failed task. Both conditions deserve attention.
Check sequence when sequence matters
Some business rules concern order, not just the final state. Approval must precede a consequential write. Identity must be verified before private information is returned. A cancellation should prevent new actions from starting. A final-state check alone may miss these violations.
Capture relevant events with timestamps or ordered identifiers. Link the proposed action to its approval and eventual execution. Do not rely on a narrative generated after the task to reconstruct the sequence; inspect the application’s observable events.
The OpenAI Agents SDK tracing guide describes traces and spans that can support execution inspection. Be deliberate about sensitive data: a trace can contain tool or model inputs and outputs, so evaluation logs need access and retention controls too.
Include the cases where no action is correct
An evaluation set biased toward successful changes may reward excessive initiative. Add requests that lack permission, have unresolved ambiguity or conflict with policy. The desired outcome may be a clarification or a safe stop, with no write at all.
For the CRM example, give the agent two plausible contacts named Priya. If the available evidence cannot identify the intended person, asking the user is better than choosing one. Grade that behavior explicitly. Otherwise the evaluation may penalize the cautious answer and encourage guessing.
OWASP’s excessive-agency guidance identifies risks from unnecessary functionality, permissions and autonomy. A practical evaluation should probe those boundaries directly rather than assuming that an agent will avoid an available action because the prompt says to be careful.
Test partial completion and uncertain writes
A multi-step task can fail after some work has succeeded. Arrange a test where a draft is saved but an attachment upload fails. The agent should report the partial state and preserve a safe recovery path. It should not describe the packet as ready or repeat completed actions without checking.
Also simulate an ambiguous timeout after a remote write. The agent’s recovery should distinguish “the request failed” from “the result is unknown.” If the system supports an operation identifier or status lookup, test that path. If it cannot establish the outcome, escalation may be the correct result.
- Capture the initial state and reset it between independent trials.
- Inspect the intended destination after each run.
- Check for unrelated writes and delayed side effects.
- Link approvals to the exact actions executed.
- Compare the final message with the verified state.
- Retain failures with enough evidence for diagnosis, while minimizing sensitive data.
Do not confuse a mock with the real system
Test doubles are useful for repeatable failures and precise control over edge cases. They can confirm that the application handles a timeout or permission denial as designed. They may not reproduce every behavior of a live CRM, mail service or document platform.
Pair controlled tests with authorized integration tests in a suitable sandbox. Verify identifiers, permissions, pagination and response semantics against the actual service. Keep production writes out of routine evaluation unless the organization has explicitly designed and authorized a safe testing process.
Report what the system did, not one reassuring number
A useful release report separates successful outcomes, authority violations, unwanted side effects and communication defects. It identifies the task categories tested and the limits of the environment. That gives decision-makers a basis for narrowing scope or requiring additional controls.
Agents earn trust through verified behavior. Evaluating their actions makes that trust concrete: the correct thing changed, the wrong things did not, and the assistant’s explanation matched the evidence. Anything less risks measuring fluency while overlooking the work itself.
Sources and further reading
Anthropic: Demystifying evals for AI agents. Evaluation of multi-turn and environment-changing tasks; published January 2026.
OpenAI Agents SDK: Tracing. Observable execution spans and sensitive-data controls; checked October 2026.
OWASP: Excessive Agency. 2025 risks from excessive functionality, authority and autonomy; checked October 2026.
Leave a Reply