Your AI Workflow Needs an Acceptance Test Before It Meets Customers
By Greg Nowak. Last updated 2026-07-19.
An AI workflow can look flawless in a demonstration and still be nowhere near ready for customer contact. Demos tend to be cooperative: the input is clean, the tools are available, the request is familiar, and someone is standing by to intervene. Production is less obliging. Customers are ambiguous. Records have gaps. APIs fail. Permissions vary. A plausible answer may hide the fact that the wrong action was taken.
An acceptance test gives the team an explicit, repeatable way to decide whether the whole workflow is safe and useful enough to release. This is not simply a test of whether the model writes a convincing response. An agentic workflow may interpret a request, retrieve information, choose tools, change business data, and report what happened. Each consequential step belongs inside the test boundary.
Check the result and how it was reached
With conventional software, a team can often assert that a particular input should produce an exact output. AI behaviour is less deterministic. The testing still needs to be disciplined, but the evidence looks different.
Anthropic's guidance separates the final outcome from the full trajectory: the outputs, tool calls, intermediate results, and other interactions recorded during a trial. That distinction matters. A workflow might tell a customer that a request has been completed even though the required record was never created. Checking the final state catches the failed task. Inspecting the trajectory can show why it failed, whether through an invalid parameter, an ignored constraint, an unnecessary tool call, or an unsafe route to an otherwise acceptable answer.
Trajectory checks should leave room for more than one valid approach. Start by verifying the business outcome, then add path-specific assertions wherever the route matters. Identity verification before a sensitive action is one example. Preventing access to an unapproved tool is another.
Different graders cover different parts of the job. Deterministic checks can confirm tool parameters, permissions, required fields, database state, latency, and cost. Human-scored rubrics are better suited to questions such as whether an escalation was understandable or a response showed appropriate caution. Model-based grading can help with larger test sets, provided its rubric has been calibrated against expert human judgment.
Start with business consequences
A useful evaluation suite starts with the workflow map, not a generic prompt benchmark. List what the system can read, decide, write, send, approve, or trigger. Then rank those actions by consequence. A poor internal draft is inconvenient. An incorrect customer message is more serious. A failed search is not in the same category as an unauthorised change to a customer record.
Those consequences become the basis for realistic scenarios. Cover normal requests, then add the conditions most likely to expose operational weaknesses: missing information, conflicting instructions, duplicate records, unavailable tools, misleading retrieved content, permission boundaries, and requests that genuinely require human judgment.
Anthropic recommends turning behaviours already checked manually into formal tests, then adding real failures when production evidence becomes available. Its guidance also calls for balanced test sets, covering both the cases where an action should happen and those where it should not. Google's ADK codelab takes a similarly structured approach, combining test cases, expected tool use, response-quality criteria, and repeatable evaluation runs to move an agent beyond proof-of-concept confidence.
| Test layer | Practical question | Evidence needed for release |
|---|---|---|
| Outcome | Was the business task completed correctly? | The required result exists and matches the scenario |
| Tool use | Were authorised tools used with valid parameters? | Required calls pass and prohibited calls are absent |
| Data and permissions | Did the workflow access and change only what it should? | Identity, scope, and field-level checks pass |
| Communication | Was the response accurate, clear, and properly qualified? | The agreed human-calibrated rubric threshold is met |
| Resilience | Did the workflow fail safely when an input or dependency broke? | No harmful action occurs, followed by a useful retry or escalation |
| Operations | Is performance acceptable under realistic conditions? | Latency, cost, and completion thresholds pass |
Recreate the conditions it will face
Passing a handful of curated examples in a development console says little about how a workflow will behave in production. OpenAI's deployment-simulation research evaluates model behaviour in environments designed to resemble real deployment contexts, including agents with access to tools. The point is to study behaviour before release, while failures are still contained.
For a business workflow, this means creating a sandbox with realistic tool schemas, permissions, data shapes, and error responses. Read and write operations should be tested without touching live customer records. Credentials should expire. Calls should time out. Results should arrive incomplete or malformed. Repeated requests should confirm that retry logic does not create duplicate actions. When authority or information is missing, the workflow should stop or escalate.
The simulation also needs to resemble the actual mix of work. A suite made entirely of adversarial prompts may overlook ordinary operational failures. Happy paths alone create false confidence. Include frequent tasks, high-impact exceptions, known failure patterns, and difficult boundary cases.
Important scenarios may need several runs because agent behaviour can vary between trials. One success proves only that the workflow can succeed. Acceptance is about whether it succeeds consistently enough for the intended use. Keep the task-level results visible; a reassuring average can easily conceal a serious failure.
Set the release gate in advance
An evaluation without a release rule is just an interesting report. Before the suite runs, decide what must pass, what can vary within an agreed tolerance, and what blocks deployment.
Some criteria should be absolute. There should be no tolerance for actions outside the workflow's permissions, disclosure of protected information, or prohibited changes to consequential records. Other measures can use agreed thresholds, including task completion, rubric scores, correct escalation, latency, and cost. Results should be split by scenario severity so that strong routine performance cannot mask failure where the stakes are highest.
A workable gate can combine four decisions: every critical safety and permission assertion passes; quality thresholds are met for each important scenario class; no consequential regression remains unresolved; and a named owner accepts any documented residual risk. When a release fails, the evidence should point to the part that needs work, whether that is the prompt, workflow logic, tool contract, permissions, model choice, or escalation path. The affected cases and the core regression suite can then be run again.
Acceptance expires
The enterprise AI assurance paper treats testing as continuous risk reduction, not a one-off certification. It covers retrieval and autonomous-agent systems, where failures may emerge from individual components or from the way those components interact.
That is a useful operating assumption because an acceptance result does not last forever. Models change. Prompts evolve. Tools acquire new fields. Business policies move on, and customer requests drift. The suite should be versioned alongside the workflow, with focused tests for routine changes and the full relevant suite before consequential releases.
Production monitoring should keep improving the test set. Tool errors, escalations, reversals, customer corrections, unusual costs, and changes in completion rates can all reveal cases worth reproducing. Once a meaningful failure has become a test, the team can confirm the fix and protect against its return. Evaluation, monitoring, and human review each answer different questions; all three are needed for a credible view of performance.
What the acceptance package should contain
The final package has to work for product owners and operators, not only AI specialists. It should include a map of consequential actions and boundaries, a scenario catalogue tied to business risks, deterministic assertions and scoring rubrics, an isolated tool-testing environment, explicit thresholds and blockers, reproducible regression runs, and reporting for quality, latency, cost, and errors.
Ownership needs the same clarity. Someone must review failures, someone must have authority to approve a release, and someone must be able to pause the workflow when production evidence contradicts the test result.
This work cannot promise that an AI system will never fail. It can replace confidence based on a polished demo with evidence grounded in how the workflow will actually operate. Greg can help turn that requirement into a practical testing package: mapping the workflow, building the scenarios and checks, setting the release gate, and leaving the team with a suite it can run again. Before the workflow speaks to customers or changes the data used to serve them, the business should know exactly what it had to prove.
Related on GrN.dk
- Agentic AI: what it is, how it works, and why it matters now
- Agent-ready APIs need a contract audit before MCP rollout
- ChatGPT apps need a permissions map before they touch company data
Need help with this kind of work?
Plan your AI workflow acceptance test Get in touch with Greg.