By Greg Nowak. Updated November 7, 2026.
If your team relies on OpenAI’s hosted Evals to check an AI workflow before release, the immediate job is to keep those checks usable after the platform closes. That means preserving the test cases, understanding how they were graded and giving someone responsibility for running the replacement.
For a business owner or agency lead, the practical question is simple: after the next prompt or model change, can your team still explain why the workflow is ready for customers?
The read-only date has passed. Shutdown is November 30.
OpenAI’s deprecation schedule lists October 31, 2026 as the date existing evaluations become read-only. The Evals dashboard and API are scheduled to shut down on November 30, 2026. Graders documented for evaluation workflows are included in the transition.
Use the remaining access window to preserve available definitions and historical evidence. Plan replacement testing outside the hosted platform now; the read-only milestone is already behind us.
Check one adjacent dependency while taking inventory: reusable prompt objects and the v1/prompts API also have a November 30 shutdown date. OpenAI advises moving that prompt content into application code. A rebuilt evaluation needs the actual instructions it tests, including any content currently referenced by a hosted prompt identifier.
Start with the workflows your business depends on
List the evaluations used for current customer delivery, internal operations and release approval. Give each a business owner who can define success and a technical owner who will maintain the checks.
Prioritise tests that protect a live workflow or an upcoming release. Archive abandoned experiments separately. If time is tight, agree which essential behaviours must have working coverage before the deadline and document what remains uncovered.
| What to preserve | What the replacement must demonstrate |
|---|---|
| Inputs and reference answers | Cases still represent the customer requests and edge cases that matter. |
| Prompts and configuration | The intended instructions, model and settings are identifiable. |
| Grading rules and rubrics | Reviewed examples receive defensible pass or fail decisions. |
| Historical outputs and scores | Earlier decisions remain understandable, with missing evidence labelled. |
| Release thresholds and owners | Someone can explain what blocks deployment and approve exceptions. |
Keep the archive somewhere the team controls, with access appropriate to the customer data it contains. Record dates and configuration references alongside results. A spreadsheet of scores without the corresponding inputs, outputs or grading rules is a limited baseline; label those gaps explicitly.
Rebuild one useful evaluation first
The OpenAI Cookbook migration guide recommends Promptfoo and describes manually recreating an evaluation. It does not depend on an OpenAI Evals export feature. Prompts, providers, test cases and grading assertions move into a portable configuration.
Choose one active workflow with clear acceptance criteria. Recreate it in promptfooconfig.yaml, configure the provider credentials and check any custom tool or agent setup. With Promptfoo installed, these commands validate the configuration, run a fresh evaluation and open its local results viewer:
promptfoo validate config -c promptfooconfig.yaml
promptfoo eval -c promptfooconfig.yaml --no-cache
promptfoo viewThe new run is separate from completed hosted runs. Use the pilot to establish a file structure, report format and review process that another maintainer can follow before repeating the work across the remaining suite.
Validate the grading before trusting the score
The Cookbook warns that similarity scores can differ between systems and that recreated graders need validation. Copying an old numerical threshold does not establish equivalent behaviour.
Where archived outputs are available, check how the replacement grades those same outputs. This helps separate a grading change from a change in what the workflow produces. Then review fresh runs against the written requirements. If historical outputs are missing, establish a reviewed baseline and record that direct comparison was unavailable.
Include clear successes, clear failures and borderline cases. Ask the workflow owner to resolve disagreements, and record whether each resolution changes the configuration, rubric or acceptance threshold.
Anthropic’s agent evaluation guidance recommends multiple trials because outputs vary, and calibrating model judges against human judgement. For a disputed result, inspect the transcript and underlying evidence before deciding the workflow failed.
For automations, check the resulting record
Anthropic also distinguishes what an agent says from the final state it leaves behind. That matters when your workflow updates a CRM, creates a task or changes an order.
For example, a test for a CRM update should inspect the resulting test record: the intended field changed, the correct record was selected and unrelated fields stayed intact. A polite confirmation message cannot establish those facts. Run action tests in a controlled environment with a defined reset process so one trial does not contaminate the next.
Make a failed check stop the release
Promptfoo’s CI/CD documentation covers JSON, HTML and JUnit XML reports, plus quality gates for deployment pipelines. Use these to make results available where your team already reviews releases.
Agree which changes trigger evaluation, which failures block deployment and who investigates them. Give essential behaviours their own blocking checks; an overall pass rate can conceal a failure in the one action the business needs.
Before handover, deliberately introduce a failing case and demonstrate that deployment is blocked. Also check how the pipeline handles an unavailable provider or missing report. Keep evidence of a passing run and a blocked release, with instructions for finding both.
Give the migration a clear finish
A useful first project with Greg can cover one business workflow: an asset inventory, runnable replacement, reviewed grading comparison, documented release rules and a handover session. That provides a concrete basis for scoping the remaining work, including custom integrations and gaps in historical evidence.
If your team needs help getting this in place before November 30, contact Greg with the workflow you need to protect, who maintains it and how releases are currently approved. Those details are enough to start a practical scoping conversation.
Related on GrN.dk
- Your AI Workflow Needs an Acceptance Test Before It Meets Customers
- Your AI workflow has logs. Can they explain one bad decision?
- A Voice Agent Is Only Ready When the Human Handoff Works
Need help with this kind of work?
Discuss your evaluation migration with Greg Get in touch with Greg.