By Greg Nowak. Last updated 2026-08-21.
AI systems often become slow or expensive for a reason that has little to do with model quality: every task is sent through the same request path. A support reply needed in seconds, a research job that may run for minutes, and an overnight classification run have very different operating needs.
A better design starts by classifying the work. Keep interactive requests narrow, make repeated prompt content cacheable, run suitable long jobs asynchronously, and move delay-tolerant volume into Batch. The model may stay the same; the surrounding workflow becomes much easier to operate.
Choose the lane before the model
Begin with the business promise. Does someone need the answer while looking at the screen? Can the system acknowledge the request and deliver the result later? Or can the entire job finish within a day? Those questions usually determine the right architecture more reliably than another model comparison.
| Workload | Best-fit pattern | Typical examples | What you must design |
|---|---|---|---|
| User is waiting | Synchronous, cache-aware request | Copilot reply, form assistance, live extraction | Latency budget, fallback, concise context |
| May take minutes | Background response | Deep analysis, complex agent task | Job state, polling, cancellation, notification |
| Can finish later | Batch API | Evaluations, bulk classification, embeddings | Input validation, reconciliation, deletion |
One product may need all three lanes. A customer tool could answer simple questions synchronously, accept a complex investigation as a background job, and evaluate yesterday’s conversations overnight in Batch. Splitting those paths makes service expectations, costs, and failures visible.
Make repeated prompts cache-friendly
Prompt caching is available automatically for eligible requests, but prompt structure determines whether it helps. Cache matches depend on an identical reusable prefix. Put stable instructions, tool definitions, schemas, and shared examples first; place the customer-specific data and immediate request afterwards.
Do not inflate a short prompt simply to qualify for caching. The useful target is naturally long context that is repeated frequently. Also keep timestamps, request IDs, changing tool definitions, and other variable fields out of the reusable prefix.
The details now differ by model generation. For GPT-5.6 and later, cacheable prefixes have a strict 1,024-token minimum. The API supports implicit caching and explicit breakpoints marked with prompt_cache_breakpoint. Setting prompt_cache_options.mode to explicit prevents a changing suffix from creating unnecessary cache writes. A consistent prompt_cache_key improves matching for related requests.
This matters commercially because GPT-5.6 cache writes are billed at 1.25 times the uncached input-token rate, while cache reads receive the cached-input rate. Measure both cached_tokens and cache_write_tokens. A workflow that repeatedly writes prefixes without reusing them is not an optimization.
Use background mode when one connection is too fragile
Background mode is intended for Responses API tasks that may take several minutes. Start the request with background=true, retain the response ID, and retrieve it while its status is queued or in_progress.
The important change is not the parameter; it is the contract with the calling system. The initial result means “accepted,” not “completed.” Store an internal job record linking the response ID to its customer, purpose, status, and delivery channel. Add sensible polling backoff, cancellation, expiry, idempotent completion handling, and a human-readable failure state.
The retention guidance has changed since the original version of this article. Background jobs from Zero Data Retention projects now run with store=false, but response data is still written temporarily to disk for roughly ten minutes so asynchronous execution and polling can work. That temporary state should be included in the data review rather than described as full incompatibility with ZDR.
Move delay-tolerant volume into Batch
The Batch API is the better lane when an immediate answer has no business value. OpenAI currently documents 50% lower costs than synchronous APIs, a separate and substantially larger rate-limit pool, and completion within 24 hours. Suitable jobs include evaluation suites, repository embeddings, bulk enrichment, classification, and backlog reprocessing.
Batch is a file-based operating process, not merely a cheaper loop. Each JSONL request needs a unique custom_id. Output order may differ from input order, so results must be reconciled using that ID. Validate a small sample before submitting thousands of records, retain the input and output file IDs, and handle errors at the individual-record level.
Completed work also needs a deletion routine. OpenAI’s data-control table lists application state for /v1/batches as retained until deleted. Someone should therefore own cleanup, rather than assuming a completed batch disappears automatically.
Governance and quality belong in the routing decision
“OpenAI retention” is not one universal setting. Abuse-monitoring logs, stored Responses, temporary background state, cached tensors, uploaded files, and batch objects follow different rules. Controls may also depend on the organization’s eligibility and configuration.
For each lane, record what data is sent, why it is needed, which identifiers are stored locally, who can retrieve the result, and when remote and local state are deleted. Then test the lane against its real promise: latency and fallback for interactive calls; state transitions and duplicate delivery for background work; completeness and reconciliation for Batch.
A sensible rollout sequence
- Inventory existing AI calls and label them interactive, long-running, or delay-tolerant.
- Record the latency promise, data class, expected volume, failure consequence, and owner.
- Stabilize reusable prompt prefixes and establish a baseline for cache reads, cache writes, latency, and cost.
- Pilot one background workflow with durable job state and one Batch workflow with reliable ID reconciliation.
- Review retention settings and deletion routines with whoever owns security and compliance.
- Add lane-specific operational alerts and quality checks before increasing volume.
The goal is not to use every API feature. It is to give each workload the least expensive, most dependable lane that still meets its service and governance requirements.
Need a practical workload map?
If your AI automation still forces every task through one synchronous path, Greg can help classify the work, surface the operational trade-offs, and turn the redesign into an implementable plan. Talk to Greg about your AI workflow.
Related on GrN.dk
- OpenAI's Guardrails and Run State Make Internal Agent Rollouts a Paid Approval-and-Audit Job
- Not Every AI Job Needs an Instant Answer: Batch the Backlog
- AI automations need a spend dashboard before the first runaway bill
Need help with this kind of work?
Talk to Greg about your AI workflow Get in touch with Greg.
