Cloudflare AI Gateway Brings LLM Budgets Into the Request Path
By Greg Nowak. Last updated 2026-08-20.
LLM cost control is usually discussed after the invoice arrives. Cloudflare AI Gateway moves part of that job into live request handling: it can track estimated model spend, evaluate budget rules and stop traffic with a 429 response when a limit has been reached. For businesses running several AI features or providers, that creates a useful control point between application code and the model.
The feature does not replace financial reporting, procurement rules or good product decisions. It does give operations teams a practical way to prevent an experimental workflow, premium model or unusually active user from consuming an open-ended budget.
Why spend limits are different from rate limits
A request limit answers, “How often may this service be called?” A spend limit answers, “How much model cost may this traffic accumulate?” That distinction matters because two requests can have very different prices depending on the model, prompt length and generated output.
Cloudflare calculates an estimated cost from token usage and known model pricing. Before sending a new request upstream, AI Gateway checks every applicable spend rule. If any rule is already over budget, the request is blocked until the fixed or rolling window resets. Spend limits work with Unified Billing and with BYOK traffic where Cloudflare knows the model price.
This is enforcement based on recorded cumulative spend, not a perfectly reserved balance. The completed request is accounted for afterwards, and Cloudflare describes the system as eventually consistent. A burst of concurrent requests can therefore exceed the nominal ceiling briefly.
The difficult part is assigning ownership
Useful budgets follow business responsibility. A single gateway-wide ceiling may protect the company total, but it cannot explain whether support automation, an internal assistant or a customer-facing feature caused the increase.
AI Gateway rules can filter or split budgets by provider, model and custom metadata. “Filter” targets a particular value, such as the production environment. “Split” creates an independent budget bucket for each value, such as each user or team. Cloudflare accepts five custom metadata entries per request, so choose a compact schema and apply it consistently. A practical starting set is team, app, workflow, environment and user_id.
| Control objective | Suggested scope | Typical response |
|---|---|---|
| Protect the total AI budget | One shared gateway rule | Block and alert |
| Restrict premium-model use | Filter by model | Route to a cheaper model |
| Give teams separate allowances | Split by metadata.team |
Block or require review |
| Limit individual users or tenants | Split by metadata.user_id |
Degrade, queue or block |
Put the gateway and metadata into one test path
Cloudflare’s current REST API provides /ai/run, OpenAI-compatible chat and Responses endpoints, and an Anthropic-compatible Messages endpoint. The following abbreviated request selects a named gateway and attaches ownership metadata:
curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/v1/chat/completions" \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
-H "cf-aig-gateway-id: production" \
-H 'cf-aig-metadata: {"team":"support","app":"helpdesk","environment":"prod"}' \
-H "Content-Type: application/json" \
-d '{"model":"openai/gpt-4.1","messages":[{"role":"user","content":"Hello"}]}'Use a Cloudflare token with the required Workers AI permission, substitute your account and gateway values, and test this on one non-critical workflow before migrating everything. Confirm that requests, tokens, errors, cost estimates and metadata appear under the expected gateway.
There are two credential models to consider. Unified Billing lets the REST API call supported third-party models using Cloudflare billing and authentication. BYOK stores provider keys in Cloudflare Secrets Store so applications do not send provider credentials with every request. Gateway authentication is still required. BYOK also supports aliases for multiple provider keys, although Unified Billing endpoints consult only the default alias.
Decide what happens when money runs out
Blocking is appropriate for batch jobs, experiments and optional internal tools. It is often too blunt for a customer journey. Cloudflare Dynamic Routing can evaluate conditions, apply rate or budget nodes and switch to a fallback model. A sensible fallback might use a cheaper model, reduce an optional enrichment step or return the application’s non-AI experience.
There is an important implementation caveat in the current documentation: dynamic routes are called through Cloudflare’s OpenAI-compatible /compat/chat/completions path and are not yet available through the newer REST API endpoints. The compatibility endpoint is deprecated for ordinary single-model chat completions but remains required for dynamic routing. Check this boundary before planning a wholesale endpoint migration.
Treat the numbers as controls, not accounts
Cloudflare says its cost figures are best-effort estimates. Providers can change prices, negotiated rates may differ and some endpoints cannot expose cost metrics unless the response contains model and token data. Use the provider dashboard or invoice as the billing authority.
There is also a maximum of 20 spend-limit rules per gateway. Avoid spending that allowance on dozens of exceptions. Start with a global ceiling, a premium-model rule and a small number of ownership-based budgets. Review blocked requests, errors, token usage and cache effectiveness regularly, then adjust limits from observed traffic rather than guesses.
A practical rollout sequence
- Inventory model calls by application, owner, provider, model and environment.
- Choose Unified Billing, BYOK or a deliberate mix.
- Route one representative workflow through a non-production gateway.
- Add a five-field-or-smaller metadata contract and verify it in logs.
- Observe normal spend before setting initial fixed or rolling limits.
- Test the
429path and any cheaper-model fallback. - Document who may change budgets, routes and provider keys.
The result should be more than another dashboard. It should be an operating model in which product owners understand their allowance, developers know the failure path, and finance can trace spend to a meaningful workload.
If your model calls are spread across scripts, products and agency-built integrations, Greg can help audit the traffic, design the metadata contract and implement a controlled AI Gateway rollout. Discuss your AI operations setup with Greg.
Related on GrN.dk
- AI automations need a spend dashboard before the first runaway bill
- The risky part of AI workflow pilots is often the OAuth screen
- Cloudflare Service Keys Stop in September: Find Every Caller
Need help with this kind of work?
Discuss your AI Gateway rollout Get in touch with Greg.