By Greg Nowak. Last updated 2026-09-13.
A rising cache-hit count looks encouraging. Before expanding an AI workflow, though, you need to know what it costs to produce a usable result, including the attempts that went nowhere.
That matters whether you're running recurring briefs through an agency workflow or increasing volume as a founder. Start with the cost today, make a controlled change, and check whether the saving holds up under normal working conditions.
Decide what counts as completed work
Choose a clear finish line before opening the token report. In a document extraction workflow, that could be a record that passes validation. In a drafting workflow, it could be an output accepted for the next editorial stage. The right boundary depends on what your business needs from the output.
API cost per completed task = all API charges attributable to the workflow ÷ accepted completed tasks.
Include retries, rejected outputs and intermediate calls in the cost. Count each accepted business result once. Otherwise, extra attempts can make the workflow look busier while quietly making each finished task more expensive.
Track human review time alongside that figure. A lower API bill may be less attractive if someone now spends longer correcting the results. Agree on the quality threshold before the test and keep it fixed throughout.
Check the rules for the model you're using
OpenAI reuses matching prompt prefixes; leaving a session open doesn't guarantee a cache hit. Its current guide distinguishes GPT-5.6 and later, where cache writes cost 1.25 times ordinary input and reads cost 0.1 times, from earlier models with no additional write charge. These rates apply to their respective token categories; they aren't stacked on top of one another. Retention also varies by model. The details are in OpenAI's prompt caching guide.
Anthropic prices five-minute writes at 1.25 times base input and one-hour writes at twice base input. Its standard read multiplier is 0.1, with exceptions for named models. A longer retention window therefore needs enough additional reuse to cover the higher write cost. Check Anthropic's caching documentation for the applicable rates.
Keep ordinary input, cache writes, cache reads and output charges separate in your calculation. Apply the relevant rate to each, then add any other API charges attributable to the workflow. Check what each provider's usage fields include before reusing a formula from another integration.
The question is whether discounted reuse covers any extra write cost across the whole workload. Include the initial write and later misses. Looking only at successful reads will overstate what you know about the saving.
Look at the requests your application sends
Prompt order matters for OpenAI's prefix matching. Where the task allows it, put stable instructions first and variable material later, then check the usage data to see whether that helped. OpenAI's guide explains prefix reuse and the eligibility rules that vary by model.
Inspect representative outgoing requests, including tool definitions and any context your application assembles. A timestamp, identifier or reordered block near the beginning is worth investigating. Some of that variation may be necessary, so establish why it's there before changing it.
Keep the revision small enough to explain and test. Record a prompt version, preserve the information the task needs, and use the same acceptance criteria. You should be able to say what moved, why, and whether the output still does the job.
Test at the pace of normal work
Use a workload that includes routine tasks and difficult inputs. Compare the existing prompt construction with the revision while holding the model, evaluation rules and other settings steady. Record exceptions, including awkward results that would be convenient to leave out.
Request spacing and concurrency need to resemble normal operation. A rapid replay can help diagnose caching behavior, but label it as a separate test. Base the spending decision on the quiet periods and bursts you actually expect.
For OpenAI Responses, record input and output counts, usage.input_tokens_details.cached_tokens and, where applicable, usage.input_tokens_details.cache_write_tokens. These are the documented fields for measuring cache usage.
Add a task identifier, prompt version, completion status, retry count and elapsed time to your own records. Retain the provider's raw usage response so you can check the calculation later. Report total spend and the number of accepted tasks alongside the average; otherwise, a change in volume can obscure what happened.
| What you see | What to check | What to do next |
|---|---|---|
| More cache reads and lower cost per accepted task | Did quality hold, and did request timing reflect normal work? | Consider rolling out the change and keep measuring. |
| More cache reads, but no reduction in task cost | Are write costs, output charges or retries absorbing the saving? | Find the cause before expanding the change. |
| Repeated writes with little reuse | Are prompt differences or gaps between requests limiting reuse? | Reassess prompt construction and retention. |
| Lower API cost, but more review work | How much additional correction time is required? | Weigh the API saving against the extra delivery effort. |
| Application estimates don't match provider costs | Do both reports cover the same scope, period and charges? | Resolve the difference before claiming a saving. |
Check the result against provider accounting
OpenAI's Usage API includes organization usage reporting and a Costs endpoint. Cost results expose monetary amounts and support reporting dimensions including project and line item. That gives you an independent accounting view to compare with your application records. The reporting structures are documented in OpenAI's Usage API reference.
Match the time boundaries and project scope on both sides. If the totals differ, look for unrelated traffic, missing request records and charges your token estimate doesn't include. Leave any unresolved difference visible rather than adjusting the calculation to make the numbers agree.
A lower bill tells you that spending fell during that period. To attribute the reduction to caching, you also need comparable work and settings. Record other changes, such as a different mix of tasks, so you can explain how much the comparison actually establishes.
Use the findings to set operating limits
If you use LiteLLM, its budget controls include reset periods through budget_duration, and enforcement requires a connected database. Scope matters: team keys use team and team-member budgets rather than the owner's personal budget. LiteLLM's budget documentation sets out those requirements.
Give someone responsibility for reviewing spend and deciding what happens when a limit is reached. Test that behavior before depending on it. Budgets constrain spending; your cost-per-task comparison shows whether the work delivered for that money is economical.
A focused engagement with GrN.dk could cover one recurring workflow. Greg could inspect representative requests, identify changing prefixes, revise prompt construction and compare measured costs with provider reporting. The workload, acceptance criteria and spending boundary should be agreed at the outset.
You should come away with revised prompt construction, a repeatable comparison, measured cost per completed task, a reconciliation of provider charges and recommended operating limits. Bring one workflow and whatever usage records you have to a conversation with Greg about the scope.
Related on GrN.dk
- Background AI Tasks Need Queues—not Just Longer API Calls
- An AI Voice Agent Needs More Than a Phone Number and a Realtime Model
- Before You Buy a GPU: Test Your Team’s Local AI Workload
Need help with this kind of work?
Discuss a focused AI cost review Get in touch with Greg.
