By Greg Nowak. Last updated 2026-09-27.
A request arrives calling itself Google-Agent. Another says it comes from ChatGPT. Do you let it through, challenge it or block it?
The user-agent string won’t settle the question because the caller supplies it. Anyone can copy a familiar name. IP checks provide more evidence, but signed requests offer a stronger option: the automated client proves its identity cryptographically.
Web Bot Auth applies HTTP message signatures to bot traffic. A verifier at the CDN, WAF or origin checks whether a recognised key signed the request. That gives your access policy something more dependable than a self-declared header.
There is a catch. Google describes its implementation as experimental, providers don’t yet sign every relevant request, and current implementations support different versions and subsets of the developing protocol. So the immediate job isn’t to create a blanket “allow signed bots” rule. It’s to distinguish verified agents from unsigned claims, failed verification and unknown automation—and decide how each category should be handled.
A bot name is a claim. A signature is evidence.
Google’s Web Bot Auth guidance explains the basic model. Participating agents sign their requests, and the receiving site verifies those signatures with public keys associated with the agent. The request is tied to a cryptographic identity instead of relying on its stated name or IP address alone.
The underlying mechanism is HTTP Message Signatures, defined in RFC 9421. A signature covers selected parts of an HTTP message. Verification checks the signature against the identified key, algorithm, time boundaries and expected covered components.
Which components are covered matters. A signature may be mathematically valid while leaving a header or request value used by your policy unsigned and open to modification.
Verified identity also doesn’t grant permission by itself. It may establish that a recognised operator signed the request, but your policy must still decide whether that agent may reach a checkout path, account area or expensive search endpoint. RFC 9421 treats message signatures as one part of a wider security design.
| What the edge sees | What you actually know | Practical response |
|---|---|---|
| Recognised identity with a valid signature | The key associated with that identity signed the request | Apply the permissions and limits assigned to that verified agent |
| Recognised user-agent without a signature | The request makes a familiar claim, but its cryptographic identity is unproven | Use the provider’s documented fallback checks and a more cautious rule |
| Signature present but invalid | Verification failed because of the key, syntax, timing, covered data or request handling | Withhold verified privileges; log the failure and investigate |
| Unknown automation | No trusted identity has been established | Apply the site’s normal bot controls, rate limits and access rules |
Verify the request where access is decided
For many sites, the best verifier is the edge service already deciding whether a request can continue. Cloudflare’s Web Bot Auth documentation describes a process based on a bot’s HTTPS key directory, signed requests and provider-managed bot registration. Cloudflare can then expose the verification result to edge policy.
OpenAI’s cloud-browser allowlisting guidance follows the same principle. ChatGPT’s cloud browser sends Signature, Signature-Input and Signature-Agent headers. Supported edge providers verify them automatically. Other environments can retrieve the public key and perform RFC 9421 verification themselves.
Provider-managed verification avoids a fair amount of custom cryptographic work. It doesn’t remove the need to inspect the resulting rule. Check which product feature performs verification, which field carries the verified identity and what an “allow” or “skip” action bypasses. Exempting a known agent from one bot rule shouldn’t quietly disable rate limits or protections on sensitive routes.
Audit the request path, not just the WAF screen
-
Map every intermediary. Document where TLS terminates and which CDN, load balancer, reverse proxy, service mesh and application component touches the request. RFC 9421 accommodates HTTP messages passing through intermediaries, but a change to a covered component can still invalidate the signature.
-
Give verification a clear owner. If the CDN or WAF already recognises the agent and verifies the signature, use its documented identity signal. If the origin must verify requests, define how keys are retrieved and cached, when expired or revoked keys leave the cache, and what happens when the key directory is unavailable. Google advises caching its key set according to Cache-Control and removing keys that are no longer published.
-
Keep the evidence intact. OpenAI recommends checking whether intermediate proxies preserve Signature, Signature-Input and Signature-Agent when legitimate traffic is unexpectedly blocked. Test the full production route. A staging request that skips a proxy tells you little about what reaches the production verifier.
-
Exercise the failure cases. Test valid, invalid, expired, missing and malformed signatures. RFC 9421 warns that successful requests alone cannot prove verification happened: a correctly signed request may pass even if the verification step was skipped. An invalid signature must never receive privileges reserved for a verified identity.
A partial rollout needs a gradual policy
Google says that some of its user agents don’t yet use Web Bot Auth and that participating agents don’t sign every request. Its guidance keeps user-agent, IP-address and reverse-DNS checks as fallbacks during the rollout.
This rules out two tempting shortcuts. Requiring a signature for every request could reject legitimate traffic. Trusting any request with the right user-agent simply preserves the original spoofing problem.
Start in observation mode. Record signature status without changing access, then compare verified traffic with existing bot classifications and fallback checks. Once the results make sense, give verified identities narrowly scoped treatment on low-risk public routes. Tighten the rules for unsigned claims later, while retaining the fallback method documented by the relevant provider.
The logs need enough detail to explain a decision: claimed identity, verification result, route, action and policy version. That record becomes especially useful when a provider changes its implementation or a proxy update begins altering signed data.
It also helps the operations team respond in proportion to the problem. A missing signature may be part of a provider’s partial rollout. An invalid signature could come from an expired message, unsupported syntax or an intermediary modifying covered data. None qualifies as verified, but each calls for a different investigation.
The implementation details are easy to underestimate
Web Bot Auth is still moving. The approved IETF architecture document is an expired Internet-Draft that newer work has replaced, while RFC 9421 remains the established standard for signing HTTP components. Treat the bot-specific layer as something to monitor rather than fixing today’s draft syntax permanently into your infrastructure.
Support also differs by provider. Cloudflare documents a limited set of accepted algorithms, components and parameters. It recommends short expiry periods and notes that its current nonce handling doesn’t keep a database of previously seen nonces. Its documented Signature-Agent syntax reflects a particular draft version. Google’s experimental guide uses a labelled form for participating requests.
Those differences make native verification appealing when your edge provider offers it. If you’re verifying elsewhere, compatibility tests are essential.
Checking whether a signature header exists is nowhere near enough. The verifier has to resolve the right key, reconstruct the signature base, validate the signature and timing, and require the components that matter to the access decision. As RFC 9421 makes clear, parts of the message outside the signature can still be changed.
Build the policy around three decisions
A workable policy answers three questions: Which identity can the edge verify? What is that identity allowed to do? What happens when verification fails or isn’t available?
Keeping those decisions separate makes WAF rules easier to review and logs easier to interpret. It also reduces the chance that one broad exception gives a crawler more access than intended.
The business deliverable extends beyond signature-validation code. It covers CDN configuration, proxy behaviour, origin rules, key handling, documented fallbacks, logging and a staged route to enforcement. Done properly, the control recognises legitimate signed agents without trusting every automation claim or giving a verified agent unrestricted access.
Greg can audit that full request path, find the right verification point and check that required headers survive every proxy. He can then turn the identity result into practical allow, limit or block rules, with fallback checks and logs that make the policy maintainable as the protocol changes.
Related on GrN.dk
- An AI Voice Agent Needs More Than a Phone Number and a Realtime Model
- Your AI Gateway Can Name the User. Decide What That Log Is For
- Apache 2.4.68 Isn’t the Only Number That Proves You’re Patched
Need help with this kind of work?
Review your AI-agent edge policy Get in touch with Greg.
