OpenAI Evals Is Closing: Keep Your Workflow Tests Running

Illustrated infographic summarizing: OpenAI Evals Is Closing: Keep Your Workflow Tests Running

By Greg Nowak. Updated November 7, 2026.

If your team relies on OpenAI’s hosted Evals to check an AI workflow before release, the immediate job is to keep those checks usable after the platform closes. That means preserving the test cases, understanding how they were graded and giving someone responsibility for running the replacement.

For a business owner or agency lead, the practical question is simple: after the next prompt or model change, can your team still explain why the workflow is ready for customers?

The read-only date has passed. Shutdown is November 30.

OpenAI’s deprecation schedule lists October 31, 2026 as the date existing evaluations become read-only. The Evals dashboard and API are scheduled to shut down on November 30, 2026. Graders documented for evaluation workflows are included in the transition.

Use the remaining access window to preserve available definitions and historical evidence. Plan replacement testing outside the hosted platform now; the read-only milestone is already behind us.

Check one adjacent dependency while taking inventory: reusable prompt objects and the v1/prompts API also have a November 30 shutdown date. OpenAI advises moving that prompt content into application code. A rebuilt evaluation needs the actual instructions it tests, including any content currently referenced by a hosted prompt identifier.

Start with the workflows your business depends on

List the evaluations used for current customer delivery, internal operations and release approval. Give each a business owner who can define success and a technical owner who will maintain the checks.

Prioritise tests that protect a live workflow or an upcoming release. Archive abandoned experiments separately. If time is tight, agree which essential behaviours must have working coverage before the deadline and document what remains uncovered.

Evaluation migration checklist: preserve the evidence behind each release decision.
What to preserve What the replacement must demonstrate
Inputs and reference answers Cases still represent the customer requests and edge cases that matter.
Prompts and configuration The intended instructions, model and settings are identifiable.
Grading rules and rubrics Reviewed examples receive defensible pass or fail decisions.
Historical outputs and scores Earlier decisions remain understandable, with missing evidence labelled.
Release thresholds and owners Someone can explain what blocks deployment and approve exceptions.

Keep the archive somewhere the team controls, with access appropriate to the customer data it contains. Record dates and configuration references alongside results. A spreadsheet of scores without the corresponding inputs, outputs or grading rules is a limited baseline; label those gaps explicitly.

Rebuild one useful evaluation first

The OpenAI Cookbook migration guide recommends Promptfoo and describes manually recreating an evaluation. It does not depend on an OpenAI Evals export feature. Prompts, providers, test cases and grading assertions move into a portable configuration.

Choose one active workflow with clear acceptance criteria. Recreate it in promptfooconfig.yaml, configure the provider credentials and check any custom tool or agent setup. With Promptfoo installed, these commands validate the configuration, run a fresh evaluation and open its local results viewer:

promptfoo validate config -c promptfooconfig.yaml
promptfoo eval -c promptfooconfig.yaml --no-cache
promptfoo view

The new run is separate from completed hosted runs. Use the pilot to establish a file structure, report format and review process that another maintainer can follow before repeating the work across the remaining suite.

Validate the grading before trusting the score

The Cookbook warns that similarity scores can differ between systems and that recreated graders need validation. Copying an old numerical threshold does not establish equivalent behaviour.

Where archived outputs are available, check how the replacement grades those same outputs. This helps separate a grading change from a change in what the workflow produces. Then review fresh runs against the written requirements. If historical outputs are missing, establish a reviewed baseline and record that direct comparison was unavailable.

Include clear successes, clear failures and borderline cases. Ask the workflow owner to resolve disagreements, and record whether each resolution changes the configuration, rubric or acceptance threshold.

Anthropic’s agent evaluation guidance recommends multiple trials because outputs vary, and calibrating model judges against human judgement. For a disputed result, inspect the transcript and underlying evidence before deciding the workflow failed.

For automations, check the resulting record

Anthropic also distinguishes what an agent says from the final state it leaves behind. That matters when your workflow updates a CRM, creates a task or changes an order.

For example, a test for a CRM update should inspect the resulting test record: the intended field changed, the correct record was selected and unrelated fields stayed intact. A polite confirmation message cannot establish those facts. Run action tests in a controlled environment with a defined reset process so one trial does not contaminate the next.

Make a failed check stop the release

Promptfoo’s CI/CD documentation covers JSON, HTML and JUnit XML reports, plus quality gates for deployment pipelines. Use these to make results available where your team already reviews releases.

Agree which changes trigger evaluation, which failures block deployment and who investigates them. Give essential behaviours their own blocking checks; an overall pass rate can conceal a failure in the one action the business needs.

Before handover, deliberately introduce a failing case and demonstrate that deployment is blocked. Also check how the pipeline handles an unavailable provider or missing report. Keep evidence of a passing run and a blocked release, with instructions for finding both.

Give the migration a clear finish

A useful first project with Greg can cover one business workflow: an asset inventory, runnable replacement, reviewed grading comparison, documented release rules and a handover session. That provides a concrete basis for scoping the remaining work, including custom integrations and gaps in historical evidence.

If your team needs help getting this in place before November 30, contact Greg with the workflow you need to protect, who maintains it and how releases are currently approved. Those details are enough to start a practical scoping conversation.

Related on GrN.dk

Need help with this kind of work?

Discuss your evaluation migration with Greg Get in touch with Greg.

Sources

Latest articles

Sending an invoice does not mean it has arrived. Understand delivery statuses, the system requirements coming in 2027, and how focused AI support can help finance teams handle errors and follow-up.

Before scaling AI content, check the pages you already have. Verify claims, resolve overlapping coverage, and give WordPress reviewers control over publication.

Build a refund approval workflow in n8n and Stripe that shows reviewers the exact action, expires old approvals and tracks refunds through to completion.

A timeout can hide a successful job creation. Use stable operation IDs, destination API rules and a clear recovery process to prevent duplicate work.

Does your ChatGPT subscription cover website translation? Understand API credits, access and a WordPress workflow that turns approved Danish content into translation drafts.

An internal AI can keep quoting a deleted policy. Check document updates, indexed passages and cached answers to find where outdated guidance survives.

Codex drew all ten pages of my Ninjai comic from the script Claude wrote in the first test. What made that handover work, and what it cost.

ChatGPT delivered ten painted comic pages; Claude drew one page in code. A practical look at AI tool selection, creative briefs and approval decisions.

Get missing supporting documents under control with a clearly scoped AI workflow that sorts replies, coordinates reminders and gives the bookkeeper a clear follow-up list.

Know what dmesg and journalctl can tell you after a reboot, how to find previous-boot logs, and how to check that retention covers your investigation window.