Your AI Workflow Changed: How Do You Know It Still Works?

Illustrated infographic summarizing: Your AI Workflow Changed: How Do You Know It Still Works?

By Greg Nowak. Last updated 2026-11-18.

A revised prompt, a different model or a change to an API integration can leave an AI workflow sounding perfectly convincing while sending an enquiry to the wrong team. If you run the business or own the process, you need evidence that the work still gets done correctly.

There is also a deadline to check. OpenAI lists November 30, 2026 as the date its Evals dashboard, API and graders for evaluation workflows become read-only, and recommends migration to Promptfoo. If your tests depend on those features, review that dependency before the deadline.

That review is a useful occasion to put a repeatable release check around one important automation. Agree what it needs to achieve, test realistic inputs and compare the proposed change with the version you use today.

Start with the job the workflow needs to do

Take an illustrative workflow that reads incoming enquiries, assigns a category and creates a record for the appropriate team. A well-written acknowledgement helps, but the category, destination and saved details all need to be correct too.

Anthropic's evaluation guidance distinguishes an agent's recorded interaction from the actual outcome in its environment. For a business automation, this is a practical distinction: a message saying the record was created does not prove that the right record exists.

Write the acceptance criteria in terms the operations owner can approve. A complete sales enquiry should create a sales record containing the supplied contact details. An ambiguous request should go to a review queue. When a required detail is missing, the workflow should leave that uncertainty visible instead of filling the gap.

Get agreement on those expectations before scoring a new version. It is hard to make a release decision when the team is still debating what the workflow was supposed to do.

Choose examples that test real decisions

LangSmith describes offline evaluation as testing curated examples before deployment, including comparisons between application versions. Its backtesting approach uses historical production inputs to assess a newer version against realistic past work.

For the enquiry workflow, include a straightforward request, a message covering two services, incomplete contact details and a request that belongs outside the sales queue. Use suitable past records where they exist, removing details the test does not need.

Each example needs more than an input message. Give it an identifier, an expected destination, the required fields and a short explanation of the business rule. If the message is ambiguous, record the acceptable fallback. Someone picking up the tests later should be able to understand why an outcome passes.

The matrix below shows how those expectations could translate into release checks. These are illustrative rules; the business would need to agree them before implementation.

Enquiry-routing checks: what must happen, what to inspect and when to stop a release
Test case Expected result Evidence to inspect Release decision
Complete sales enquiry A sales record containing the supplied details Saved category, destination and contact fields Block if routing or details are wrong
Unclear service request A review queue entry that preserves the uncertainty Queue assignment and reason for review Block if confidently misrouted
Missing required contact detail The missing value remains explicit Saved fields and follow-up status Block if a value is invented
Correct routing, awkward acknowledgement An accurate, usable acknowledgement Human assessment against agreed writing criteria Review severity before deciding

Check the record as well as the response

LangSmith documents code evaluators for structure and business rules, model judges for qualities such as clarity, and pairwise evaluation for comparing two outputs. Each can answer a different question about the release.

Use code to check required fields and permitted categories. Inspect the destination record to confirm that the integration saved the right values. Then assess the acknowledgement for accuracy and usefulness. A valid category in the response does not establish that the enquiry reached the right team.

If a model grades the writing, give it explicit criteria and compare its verdicts with human assessments. Anthropic recommends calibrating model graders against expert judgment. Keep the results visible for each criterion so that polished wording cannot offset a wrong destination.

Some apparent failures will also need a closer look. If a test rejects an acceptable outcome, correct the test. Otherwise, you risk changing working behaviour to satisfy a check that does not reflect the business requirement.

Run both versions against the same cases

The Promptfoo configuration guide shows how prompts, model providers and example inputs can form a repeatable evaluation. Assertions check requirements automatically, while the matrix view lets you review outputs across configurations.

Keep prompts and test inputs in reusable files. Run the existing and proposed versions against the same cases, retaining the outputs and check results. The guide demonstrates checks for JSON content, custom JavaScript conditions and semantic similarity. Choose the ones that express your acceptance criteria.

For an integrated automation, the comparison also needs evidence from the destination system. Run the enquiry workflow against a test destination so you can inspect saved records without putting test requests into a live team's queue.

Repeat important cases. Anthropic notes that model outputs vary between trials, so one favourable run gives you limited evidence. Record the configuration used as well; a later reviewer needs to know exactly what was compared.

Decide what would stop the release

Agree the blocking failures before running the comparison. In this example, fabricated contact details would stop the release, even if the proposed version wrote better acknowledgements in other cases. Less serious wording problems might need a judgment call.

Look closely at the cases that changed. Which previously acceptable outcomes now fail? Which failures have improved? Does any new result need a decision from the operations owner? Keep the overall score, but review it alongside those details. An average cannot tell you how much a particular routing mistake matters to the business.

Capture the decision in a short release note, including unresolved limitations and the person responsible for approval. Preserve the previous configuration and agree how to restore it if necessary.

After release, LangSmith's production feedback approach offers a way to maintain the tests: review live failures and consider adding them to future offline evaluations. The examples can then reflect problems the workflow has actually encountered.

A manageable first piece of work

A first engagement with GrN.dk could cover one existing workflow and one proposed change. The work would produce agreed acceptance criteria, a reusable test dataset, automated checks, a comparison with the current version and a short handover showing how to rerun the evaluation.

At the outset, agree access, the test environment and who can judge borderline results. Those details determine whether the evaluation can support a release decision and who takes responsibility for it.

Bring Greg one automation your business already uses and the next change you have planned. That gives you a concrete starting point for agreeing what needs checking before release.

Related on GrN.dk

Need help with this kind of work?

Discuss testing your AI workflow with Greg Get in touch with Greg.

Sources

Latest articles

Before changing an AI workflow, check that it still does the job. Use realistic examples, clear release criteria and a comparison with the version running today.

Sending an invoice does not mean it has arrived. Understand delivery statuses, the system requirements coming in 2027, and how focused AI support can help finance teams handle errors and follow-up.

Before scaling AI content, check the pages you already have. Verify claims, resolve overlapping coverage, and give WordPress reviewers control over publication.

Build a refund approval workflow in n8n and Stripe that shows reviewers the exact action, expires old approvals and tracks refunds through to completion.

A timeout can hide a successful job creation. Use stable operation IDs, destination API rules and a clear recovery process to prevent duplicate work.

Does your ChatGPT subscription cover website translation? Understand API credits, access and a WordPress workflow that turns approved Danish content into translation drafts.

An internal AI can keep quoting a deleted policy. Check document updates, indexed passages and cached answers to find where outdated guidance survives.

Codex drew all ten pages of my Ninjai comic from the script Claude wrote in the first test. What made that handover work, and what it cost.

ChatGPT delivered ten painted comic pages; Claude drew one page in code. A practical look at AI tool selection, creative briefs and approval decisions.

Get missing supporting documents under control with a clearly scoped AI workflow that sorts replies, coordinates reminders and gives the bookkeeper a clear follow-up list.