MyDot Consultancy

Featured image: AI-generated conceptual illustration. Sources checked 26 October 2026.

A company may already have an agent builder but still lack an independent answer to a basic question: does the system perform its intended job reliably enough to deploy? That gap can support a specialist evaluation service. The deliverable is a reproducible assessment, not a badge claiming that the system is universally safe.

The proposed commercial model combines a scoped initial evaluation with optional regression testing after important changes. It suits customers that can define real tasks, provide authorized test access, and act on findings. It is a poor fit when the buyer wants a favorable score for marketing but refuses to expose failure cases or agree on what success means.

Sell a question you can answer

Begin with the deployment decision. A support team may need to know whether an agent can prepare accurate replies for a defined product category. A finance team may need to know whether invoice exceptions reach the correct reviewer. These are different evaluations, even if both systems use the same model.

Write a short evaluation charter naming the task, user population, available tools, environment, and permitted actions. Specify excluded uses and the period covered by the report. A result for one configuration does not automatically apply after a model upgrade, a new integration, or a major policy change.

Agree on failure severity before testing. An awkward greeting, an incorrect setup instruction, and an unauthorized refund should not receive equal weight. Some failures should block deployment even when the overall average looks strong. Make those gates visible so the evaluator cannot quietly change the standard after seeing the results.

Build evidence from real work

Obtain authorized examples that represent ordinary demand and costly edge cases. Remove unnecessary personal information and document retention. Historical records can reveal actual ambiguity, but they can also contain outdated policies or human mistakes. Validate expected outcomes rather than treating history as unquestionable truth.

Separate development examples from the final evaluation set. If the builder tunes directly against every test, the final score measures adaptation to that set as much as general task performance. Maintain a process for adding newly discovered failures without losing comparability with earlier runs.

Use synthetic cases deliberately. They are useful for testing missing fields, contradictory instructions, timeouts, and duplicate events. Label them as constructed cases and do not imply that their frequency matches production. A stress suite and a representative workload sample answer different questions.

Match the grader to the claim

Deterministic checks work well for observable facts: a record exists, the amount matches, a prohibited tool was not called, or a ticket has the expected owner. Human review is appropriate for nuanced relevance and policy application. Model-based graders can assist, but their agreement with qualified reviewers should be checked rather than assumed.

Inspect the resulting system state. A transcript that says “I updated the account” is not sufficient. The linked evaluation source distinguishes a task's final outcome from the agent's narrative. This is especially important for workflows that change external records.

Run repeated trials where variability matters. A single successful run can conceal an unstable process. Report the number of attempts, important configuration details, and uncertainty. If the sample is small, avoid dressing a rough estimate in excessive decimal precision.

Include recovery and authority tests

Test what happens when credentials expire, a tool returns an incomplete response, or a downstream action times out. Does the system stop, retry safely, or continue with an invented result? Recovery behavior often determines whether a workflow is practical to operate.

Include untrusted-content cases. A customer email or retrieved document may contain instructions aimed at the agent. The system should treat that content as task data rather than authority to change its operating rules. Test the boundary using harmless controlled scenarios and only within authorized environments.

Check approval handling separately. Can the agent distinguish preparing a recommendation from executing it? Does it preserve the approval requirement after a retry or a handoff? A technically correct action can still be a failure if it was performed without the required authority.

Make the report useful to a buyer

Deliver a concise decision summary, a methods section, detailed findings, and reproducible evidence. For each important failure, include the triggering conditions, observed outcome, business consequence, and suggested remediation. Protect sensitive traces and avoid distributing secrets in screenshots or exported logs.

Distinguish “not observed” from “cannot happen.” A finite test set cannot prove universal safety. Likewise, a voluntary risk-management framework is useful context but does not turn your report into a regulatory certification. State exactly what was tested and what remains unknown.

Price the work according to task complexity, environment setup, specialist review, and retest obligations. A fixed number of prompts is rarely a meaningful unit of professional effort. If the customer wants a recurring service, define which changes trigger a new test and how quickly findings will be delivered.

The strongest outcome is a better deployment decision: launch within a narrow scope, fix specific failures, keep a human approval step, or defer deployment. Independence is valuable only when the evaluator is willing to recommend any of those options.

Offer a retest with the same documented conditions after remediation. Preserve the original findings so the customer can distinguish a genuine fix from a changed test or a narrower claim.

Sources and evidence

Demystifying evals for AI agents.

NIST AI Risk Management Framework.

Research checked October 26, 2026. Business designs and illustrative calculations in this article are editorial proposals, not documented earnings or guarantees. Recheck current product terms before implementation.