By Greg Nowak. Last updated 2026-06-17.
On June 4, 2026, Tom's Hardware reported that Cloudflare's latest figures put automated HTTP requests at 57.5% of traffic, versus 42.5% for humans. Strictly speaking, that figure is about bot traffic overall, not a clean AI-only subtotal. But if you run a website, the practical conclusion is the same: crawler policy is no longer something you can leave vague. Once bots become the majority condition on the web, you need a deliberate position on who can access your content, where, and on what terms.
That is why Cloudflare's AI Crawl Control matters. This is no longer just a block list. Cloudflare documents it as a working control layer for seeing which AI services access your site, setting crawler-specific allow or block rules, monitoring robots.txt compliance, and exploring monetized access. Business Insider's April 7, 2026 reporting on Cloudflare's partnership with GoDaddy makes the broader point: this is moving out of specialist publisher workflows and into normal website operations for millions of site owners.
If your site contains valuable editorial, documentation, product content, or structured data, you now need a policy by section. Which crawlers do you want to allow? Which do you want to block? Which areas might deserve a paid-access model later? If you have not answered those questions, you still have a policy. It is just happening by default.
| Site area | Practical default stance | Business reason | What to monitor |
|---|---|---|---|
| Ad-supported editorial | Managed robots.txt plus selective blocking | If referral value is weak, heavy crawling can undercut pages that depend on view-based revenue | Allowed requests, referrals, bandwidth, and most-requested paths |
| Documentation and help content | Allow selected crawlers, then review outcomes | Some sections may still earn useful discovery or referral traffic | Operators, referrers, destination patterns, and response codes |
| Product, pricing, and structured pages | Review section by section | These pages carry sales value but can also leak reusable content or data | Path patterns, status codes, and per-crawler data transfer |
| Premium or licensed archives | Block by default or assess paid access | High-reuse content may justify tighter control or monetization | Blocked requests, 402 responses, and operator demand |
The old indexing bargain has changed
For years, most site owners accepted crawler access because the trade was easy to understand: index my pages and send people back. Cloudflare's July 1, 2025 post on AI training controls argues that AI training crawlers break that older bargain. Technically, they crawl in familiar ways. Economically, they do something else. They can absorb content into answer products that keep the user inside the AI platform instead of returning them to the original site.
Cloudflare attached hard numbers to that argument. In June 2025, the company said OpenAI's crawl-to-referral ratio was 1,700 to 1, while Anthropic's was 73,000 to 1. That does not mean every crawl is harmful. It does mean the old assumption that crawling naturally pays back in traffic is no longer safe. The value exchange now varies sharply by operator, by content type, and by business model.
The same post also noted that only about 37% of the top 10,000 domains had a robots.txt file at all. That matters because it shows how many organizations still treat AI crawling as an abstract future concern, even while the traffic is already there. Basic governance is still missing on a large share of prominent sites.
What changed in the tooling
Cloudflare's documentation makes clear that AI crawler governance is becoming operational work, not theory. AI Crawl Control is available on all plans and is designed to work automatically, but the bigger shift is visibility. The overview page describes per-crawler controls, robots.txt compliance monitoring, and beta monetization options alongside zero-configuration deployment. That is a very different setup from dropping a few disallow lines into a text file and hoping for the best.
The analytics side is where this becomes genuinely useful. Site owners can inspect total request volume, changes over time, common status codes, popular paths, high-volume crawlers, and grouped patterns such as /blog/*, /api/*, or /docs/*. They can filter by date range, crawler, operator, hostname, or path. They can review allowed requests, unsuccessful requests, and data transfer. Paid-plan customers can also see referrer analytics, referral trends over time, and destination patterns that show which parts of the site actually receive AI-driven visits.
That path-level view matters because most businesses should not want one sitewide answer. You may want public help documentation visible, ad-supported sections protected more aggressively, and structured product data reviewed carefully. Cloudflare's status-code reporting even separates different policy outcomes inside 4xx traffic, including blocked requests and payment-required responses. In other words, the tooling is built for policy design, not just traffic suppression.
That changes the conversation. You no longer have to debate AI bots in the abstract. You can look at which operators are hitting which parts of the site, how much bandwidth they consume, what response mix they receive, and whether any measurable referral traffic comes back.
Robots.txt still matters, but it is not enough
Cloudflare is direct about the limitation: robots.txt is an important signal, but compliance is voluntary. That is why the company paired managed robots.txt with enforcement options. According to the post, customers can let Cloudflare create a robots.txt file or prepend managed directives to an existing one, so AI-training preferences stay current without constant manual upkeep.
The same post also describes a more selective control for ad-supported sites: block AI bots only on hostnames where ads are detected. That is a useful middle ground. A publisher, for example, may want utility sections or support content accessible while protecting the inventory that directly supports advertising revenue.
The broader lesson is that crawler governance now needs layers. One layer communicates preference through robots.txt. Another enforces policy when a crawler ignores that preference. A third measures whether any allowed access produces worthwhile referrals or simply extracts value without much return.
Why Pay Per Crawl changes the brief
The biggest shift may end up being commercial rather than technical. Cloudflare's Pay Per Crawl documentation describes a beta feature that lets a site owner set a price per zone for AI crawler access. The docs say a crawler can present payment intent for successful HTTP 200 access or receive HTTP 402 Payment Required with pricing information. Cloudflare also notes that existing WAF or Bot Management blocking rules override charging rules, which means monetization sits inside a broader control stack rather than replacing it.
Even if many businesses never charge for crawler access, the existence of that feature changes the planning model. The question is no longer just allow versus block. It becomes: which content should stay open, which should be restricted, and which might eventually justify priced access? That is especially relevant for high-effort documentation, research archives, premium editorial libraries, and structured content that is expensive to produce and easy to reuse elsewhere.
What sensible governance looks like now
A workable policy usually starts with classification, not tooling. Break the site into content groups with different business logic: ad-supported pages, lead-generation pages, public documentation, support content, premium assets, and machine-readable endpoints. Then give each group a default crawler stance and a measurement plan.
From Cloudflare's analytics alone, you can build a practical review loop. Look at which operators and crawlers are touching the site. Review the most-requested paths and URI patterns. Compare allowed request volume with referral volume where that data is available. Check whether some sections attract heavy crawling with little commercial upside. Then decide whether those sections should remain open, be blocked, or be reserved for future monetization experiments.
This is the kind of audit GrN can help with: reviewing AI crawler traffic, splitting a site into sensible allow, block, and potential monetization zones, implementing Cloudflare rules and robots policies, and tying the result back to referral data so decisions stop relying on guesswork.
The larger point is straightforward. Cloudflare's recent product changes, the June 2026 reporting that bots now outweigh human traffic, and the GoDaddy rollout all point in the same direction. AI crawler control has moved from edge-case bot blocking to a mainstream operating discipline. Doing nothing is still a policy. It is simply an unexamined one.
Related on GrN.dk
- Cloudflare Turnstile only works if you validate and tune it, making lead-form abuse a paid ops job
- AI Shopping Surfaces Make Product Data Integrity a Technical Ops Job
- ChatGPT Apps and Full MCP Access Turn Internal Tool Exposure Into a Paid Governance Project
Need help with this kind of work?
Discuss AI crawler governance Get in touch with Greg.
Sources
- Overview · Cloudflare AI Crawl Control docs
- Analyze AI traffic · Cloudflare AI Crawl Control docs
- What is Pay Per Crawl? · Cloudflare AI Crawl Control docs
- Control content use for AI training with Cloudflare's managed robots.txt and blocking for monetized content
- Cloudflare and GoDaddy team up to help websites fend off Big Tech's AI bot swarm
- 'Bots have now passed human traffic online,' Cloudflare boss laments