Not Every AI Job Needs an Instant Answer: Batch the Backlog

By Greg Nowak. Updated 28 October 2026.

A customer waiting for an answer needs a fast AI response. A catalogue refresh, document classification run, or weekly report usually has a different deadline. Treating both alike can add processing cost without improving the work.

Batch processing lets you collect independent AI requests, submit them together, and retrieve the results later. For business owners and operations teams, the opportunity is straightforward: move suitable work into a cheaper processing route while keeping quality, deadlines, and ownership clear.

Start with the deadline, then choose the route

“Background work” is too vague to design around. Ask when someone actually needs the output, what happens if it is late, and whether each item can be processed independently.

Product tagging, content enrichment, evaluation runs, and archive summaries are sensible candidates. A task that needs a person's answer before it can continue needs a different workflow.

A crucial detail: submitting tonight does not necessarily mean receiving checked results before tomorrow's opening time. Schedule backwards from the business deadline, allowing time for import, validation, and recovery. If that buffer does not fit, keep a faster route.

What the batch discount actually buys

The main providers document similar pricing incentives, with different operating details:

• OpenAI: The Batch API offers a 50% discount against synchronous processing and a 24-hour completion window, with separate rate limits.
• Anthropic: The Message Batches API offers a 50% discount. Anthropic says most batches finish within an hour, but they can expire after 24 hours. Each batch is limited to 100,000 requests or 256 MB.
• Google: The Gemini Batch API runs at 50% of standard cost, with a target turnaround of 24 hours. It currently supports generateContent, with inline requests or file input.

Those discounts apply to provider processing. Your overall saving depends on queue maintenance, failures, retries, and human review. Compare the cost per accepted output using the same model and acceptance criteria before changing the model or prompt as well.

Build the queue around the business records

A scheduler and a small Python or PHP worker may be enough. The important parts are the records and recovery behaviour around the API call:

1. Define eligibility. Store the deadline, data class, source record version, and processing status. For agencies handling several clients, keep account access, payloads, and cost reporting separated.
2. Keep stable identifiers. Record the internal job ID, item ID, provider batch ID, model, and prompt version. Preserve the link between each request and the exact source content it used.
3. Validate before submission. Check required fields, encoding, supported model and endpoint, request size, and queue limits. Reserve selected records so overlapping scheduled runs cannot submit the same work.
4. Monitor and reconcile. Assign an owner for stalled or failed jobs. Compare expected items with successful outputs, errors, and missing results before marking the internal job finished.
5. Retry selectively. Retry temporary failures with bounded attempts. Fix invalid inputs first. Preserve successful results when a batch expires or is cancelled.
6. Import safely. Validate the output schema and allowed values, then sample quality. Make repeated imports harmless and reject stale results if the source record has changed.

For OpenAI, prepare a .jsonl file with one request per line and a unique custom_id. Upload it with purpose="batch", then create the batch with the input file ID, the chosen endpoint, and completion_window="24h". Match returned records by custom_id; output order can differ from input order. Retrieve both successful outputs and errors. The implementation guide covers the lifecycle and expiry behaviour.

Check retention before moving real data

Changing the processing route can change how inputs and outputs are stored. OpenAI's data controls documentation lists /v1/batches as ineligible for Zero Data Retention, with application state retained until deletion. Files have their own deletion and expiry controls; do not assume one cleanup action covers everything.

Anthropic documents storage of batch requests and responses for up to 29 days after creation, with deletion available after processing. For any provider, confirm the endpoint, region, project settings, and cleanup process for the actual dataset. Remove fields the task does not need.

Keep urgent work accountable

An order correction may need immediate processing even when routine catalogue work can wait. Define who can select the urgent route and record the reason. Coordinate that override with pending batch work so a late batch response cannot overwrite the correction.

If exceptions become frequent, revisit the deadline assumptions. The workload may need a faster service level.

Make the first pilot easy to judge

Choose one recurring workload with a clear owner and measurable acceptance criteria. Compare representative records across both routes: usable outputs, completion time, reconciliation, retries, review effort, and total cost. Include setup and ongoing support when deciding whether the saving justifies the change.

If your team needs help choosing the first workload or building the queue, I can review the existing integration and define a practical pilot. Contact me with the task, approximate volume, and deadline, and we can work through whether batching fits.

Related on GrN.dk

• Cache, Background, or Batch? How to Route AI Workloads
• AI automations need a spend dashboard before the first runaway bill
• From Supplier PDFs to Product Data: Where AI Needs a Second Check

Need help with this kind of work?

Discuss your AI batch processing workload with Greg Get in touch with Greg.

Sources

• OpenAI Batch API guide
• Claude batch processing documentation
• Gemini Batch API documentation
• Data controls in the OpenAI platform