By Greg Nowak. Updated 27 October 2026.
A request calls itself Google-Agent. Your firewall challenges it, and someone asks you to let it through. What evidence would justify changing the rule?
The user-agent string is supplied by the caller. Copying a familiar name is easy. Web Bot Auth adds cryptographic evidence, giving your CDN or web application firewall (WAF) a way to verify a signed agent request.
For a business, the aim is straightforward: keep useful automated access working while controlling impersonation, server load and sensitive routes. That requires both identity verification and a clear access policy.
What Web Bot Auth proves—and what you still decide
Web Bot Auth builds on RFC 9421, the HTTP Message Signatures standard. A verifier checks a signature against a key and the selected request components it covers. Those components matter: a valid signature does not protect values outside its coverage.
A signature alone does not establish a familiar brand. The verifier also needs a trustworthy association between the signing key and the claimed identity. Otherwise, an unknown operator can sign its own requests perfectly well.
Once identity is established, permission remains your decision. You might allow a recognised agent to read public product pages with rate limits while keeping account pages, checkout actions and internal APIs behind their existing controls. Verified operator identity does not authenticate a customer or authorise a purchase.
Use a decision table before writing an exception
I would start by agreeing these outcomes with whoever owns the website and whoever maintains its edge rules. A shared policy prevents a support request from becoming an unnecessarily broad firewall exemption.
| Request outcome | What it means | Suggested treatment |
|---|---|---|
| Verified, recognised identity | Signature checks pass and the key is associated with an identity you recognise. | Apply that agent’s route permissions and rate limits. |
| Familiar name, no signature | The name remains an unverified claim. | Use the operator’s documented fallback checks. |
| Invalid signature | Cryptographic, coverage or freshness checks fail. | Withhold verified privileges; investigate the failure. |
| Unknown key or unavailable directory | There is insufficient evidence to establish the claimed identity. | Keep it unverified; apply fallback policy and bounded retries. |
| Other unsigned traffic | No signature evidence is available; this alone does not identify a bot. | Keep normal access and bot controls in place. |
The distinction between invalid and unverified follows the current IETF Web Bot Auth working draft. It remains work in progress, rather than a finished RFC. A directory outage should not be treated as proof of impersonation.
Google still needs a fallback
Google’s current guidance calls its implementation experimental. Not all Google agents participate, and participating agents do not sign every request. Requiring signatures across all Google traffic could therefore block legitimate access.
For teams implementing verification themselves, Google currently specifies:
- Fetch public keys from
https://agent.bot.goog/.well-known/http-message-signatures-directory. - Respect
Cache-Controlwhen caching keys, and remove cached keys absent from a successfully refreshed directory. - Recognise
Signature-Agent: g="https://agent.bot.goog"and verify the correspondinggentries inSignatureandSignature-Input. - Retain Google’s documented IP-based verification fallback for unsigned requests.
These are verification steps, not a header-matching allow rule. The presence of those headers is not evidence that verification succeeded.
Choose the verifier, then test the whole request path
If your edge provider supports the agent, managed verification is usually the first option to assess. Confirm which trusted identity signal reaches your rules and exactly what an allow or skip action bypasses. Scope an exception to the protection causing the problem.
Cloudflare’s documentation illustrates why compatibility checks matter. Its documented integration supports Ed25519 and a subset of RFC 9421. It currently describes a quoted-string Signature-Agent format and says later dictionary forms fail verification. Google documents a labelled form. Do not assume examples from different implementations are interchangeable; confirm support for the specific agent with your provider.
Cloudflare also says it does not maintain a database of seen nonces and recommends short signature expiry periods. A nonce appearing in a header is therefore no guarantee that replay has been prevented.
Map the production route through the CDN, load balancer, reverse proxy and application. Preserve Signature, Signature-Input and Signature-Agent until verification, and check whether any intermediary changes covered values.
If the origin consumes an edge verification result, prevent clients from supplying that trusted signal themselves or reaching an unprotected origin directly. Otherwise, the application may trust a claim that bypassed the verifier.
For custom verification, key retrieval needs limits too. The working draft warns about server-side request forgery through caller-controlled directory URLs. Restrict fetch destinations, block private network addresses, and bound redirects, response sizes and timeouts.
Roll out with evidence and a way back
Start by observing verification outcomes without changing access. Compare them with existing classifications and investigate unexpected failures. Then introduce narrow exceptions on public routes, with an owner and a rollback procedure.
Test valid, tampered, expired, malformed and missing signatures, plus unknown keys and directory failures. Include the real proxy chain. A valid request passing is insufficient: a request with a broken signature must lose any privileges reserved for verified agents.
Log the claimed identity, verified identity where available, outcome, route, action and policy version. Keep directory failures distinct from signature failures so operations can investigate the right component.
For an agency handover, the deliverable should include the request-path map, policy table, tested rules, fallback behaviour and named maintenance owner. That gives the next person enough context to change a rule safely.
If you need help connecting those pieces, Greg can review your CDN, proxies and origin controls and turn the findings into a scoped implementation plan. Talk to Greg about your AI-agent edge policy.
Related on GrN.dk
- Your AI Gateway Can Name the User. Decide What That Log Is For
- A stray Set-Cookie can waste your CDN: audit the cache at the edge
- AI Crawler Control for Business Websites: Protect Content Without Losing Search Visibility
Need help with this kind of work?
Talk to Greg about your AI-agent edge policy Get in touch with Greg.
