By Greg Nowak. Last updated 2026-11-18.
A revised prompt, a different model or a change to an API integration can leave an AI workflow sounding perfectly convincing while sending an enquiry to the wrong team. If you run the business or own the process, you need evidence that the work still gets done correctly.
There is also a deadline to check. OpenAI lists November 30, 2026 as the date its Evals dashboard, API and graders for evaluation workflows become read-only, and recommends migration to Promptfoo. If your tests depend on those features, review that dependency before the deadline.
That review is a useful occasion to put a repeatable release check around one important automation. Agree what it needs to achieve, test realistic inputs and compare the proposed change with the version you use today.
Start with the job the workflow needs to do
Take an illustrative workflow that reads incoming enquiries, assigns a category and creates a record for the appropriate team. A well-written acknowledgement helps, but the category, destination and saved details all need to be correct too.
Anthropic's evaluation guidance distinguishes an agent's recorded interaction from the actual outcome in its environment. For a business automation, this is a practical distinction: a message saying the record was created does not prove that the right record exists.
Write the acceptance criteria in terms the operations owner can approve. A complete sales enquiry should create a sales record containing the supplied contact details. An ambiguous request should go to a review queue. When a required detail is missing, the workflow should leave that uncertainty visible instead of filling the gap.
Get agreement on those expectations before scoring a new version. It is hard to make a release decision when the team is still debating what the workflow was supposed to do.
Choose examples that test real decisions
LangSmith describes offline evaluation as testing curated examples before deployment, including comparisons between application versions. Its backtesting approach uses historical production inputs to assess a newer version against realistic past work.
For the enquiry workflow, include a straightforward request, a message covering two services, incomplete contact details and a request that belongs outside the sales queue. Use suitable past records where they exist, removing details the test does not need.
Each example needs more than an input message. Give it an identifier, an expected destination, the required fields and a short explanation of the business rule. If the message is ambiguous, record the acceptable fallback. Someone picking up the tests later should be able to understand why an outcome passes.
The matrix below shows how those expectations could translate into release checks. These are illustrative rules; the business would need to agree them before implementation.
| Test case | Expected result | Evidence to inspect | Release decision |
|---|---|---|---|
| Complete sales enquiry | A sales record containing the supplied details | Saved category, destination and contact fields | Block if routing or details are wrong |
| Unclear service request | A review queue entry that preserves the uncertainty | Queue assignment and reason for review | Block if confidently misrouted |
| Missing required contact detail | The missing value remains explicit | Saved fields and follow-up status | Block if a value is invented |
| Correct routing, awkward acknowledgement | An accurate, usable acknowledgement | Human assessment against agreed writing criteria | Review severity before deciding |
Check the record as well as the response
LangSmith documents code evaluators for structure and business rules, model judges for qualities such as clarity, and pairwise evaluation for comparing two outputs. Each can answer a different question about the release.
Use code to check required fields and permitted categories. Inspect the destination record to confirm that the integration saved the right values. Then assess the acknowledgement for accuracy and usefulness. A valid category in the response does not establish that the enquiry reached the right team.
If a model grades the writing, give it explicit criteria and compare its verdicts with human assessments. Anthropic recommends calibrating model graders against expert judgment. Keep the results visible for each criterion so that polished wording cannot offset a wrong destination.
Some apparent failures will also need a closer look. If a test rejects an acceptable outcome, correct the test. Otherwise, you risk changing working behaviour to satisfy a check that does not reflect the business requirement.
Run both versions against the same cases
The Promptfoo configuration guide shows how prompts, model providers and example inputs can form a repeatable evaluation. Assertions check requirements automatically, while the matrix view lets you review outputs across configurations.
Keep prompts and test inputs in reusable files. Run the existing and proposed versions against the same cases, retaining the outputs and check results. The guide demonstrates checks for JSON content, custom JavaScript conditions and semantic similarity. Choose the ones that express your acceptance criteria.
For an integrated automation, the comparison also needs evidence from the destination system. Run the enquiry workflow against a test destination so you can inspect saved records without putting test requests into a live team's queue.
Repeat important cases. Anthropic notes that model outputs vary between trials, so one favourable run gives you limited evidence. Record the configuration used as well; a later reviewer needs to know exactly what was compared.
Decide what would stop the release
Agree the blocking failures before running the comparison. In this example, fabricated contact details would stop the release, even if the proposed version wrote better acknowledgements in other cases. Less serious wording problems might need a judgment call.
Look closely at the cases that changed. Which previously acceptable outcomes now fail? Which failures have improved? Does any new result need a decision from the operations owner? Keep the overall score, but review it alongside those details. An average cannot tell you how much a particular routing mistake matters to the business.
Capture the decision in a short release note, including unresolved limitations and the person responsible for approval. Preserve the previous configuration and agree how to restore it if necessary.
After release, LangSmith's production feedback approach offers a way to maintain the tests: review live failures and consider adding them to future offline evaluations. The examples can then reflect problems the workflow has actually encountered.
A manageable first piece of work
A first engagement with GrN.dk could cover one existing workflow and one proposed change. The work would produce agreed acceptance criteria, a reusable test dataset, automated checks, a comparison with the current version and a short handover showing how to rerun the evaluation.
At the outset, agree access, the test environment and who can judge borderline results. Those details determine whether the evaluation can support a release decision and who takes responsibility for it.
Bring Greg one automation your business already uses and the next change you have planned. That gives you a concrete starting point for agreeing what needs checking before release.
Related on GrN.dk
- Apache 2.4.68: What to Check Before You Call It Patched
- Your AI Workflow Runs on One Person’s API Key. Now What?
- AI Content and Google: Fix the Process Before You Publish More
Need help with this kind of work?
Discuss testing your AI workflow with Greg Get in touch with Greg.