By Greg Nowak. Updated 22 August 2026.
Automated traffic has crossed a meaningful threshold. In June, Cloudflare data reported by Tom’s Hardware put bots at 57.5% of HTTP requests and humans at 42.5%. That does not mean AI systems now represent most internet users: the measurement counts requests, not people, attention or revenue. It also covers bot traffic more broadly rather than providing a clean AI-only subtotal.
The business implication still matters. Websites are increasingly being read by search crawlers, training crawlers, AI assistants, shopping agents and less welcome automation. A blanket “allow everything” or “block all AI” response is unlikely to serve every part of a commercial website.
A useful crawler policy answers three questions: which automated visitors may access which content, what value the business expects in return, and how the decision will be enforced and reviewed.
Start with business value, not bot names
Do not begin by copying somebody else’s crawler block list. Begin by classifying the site. Public documentation, lead-generation pages, product data and licensed research have different economics even when they share one domain.
| Content area | Reasonable starting position | Evidence to review |
|---|---|---|
| Marketing and service pages | Allow reputable search and assistant crawlers | Qualified referrals, citations and conversions |
| Documentation and support | Allow selectively; monitor volume | Referral destinations, support outcomes and bandwidth |
| Product, price or inventory data | Review path by path | Commercial discovery versus reuse or excessive polling |
| Original research or premium archives | Block by default or assess licensing | Crawl demand, content cost and existing agreements |
| Login, search and account areas | Restrict unnecessary crawling | Security logs, server load and failed requests |
Record an owner and rationale for every exception. “Allowed because it may be useful” is not a durable policy; “allowed on public documentation while monthly referrals and request volume are reviewed” is.
Run a crawler audit before changing the rules
Cloudflare’s AI Crawl Control is available across its plans and can show crawler activity by operator, hostname and path. Current documentation also highlights an important limitation: free-plan detection relies on self-declared user-agent strings, while more advanced detection requires Bot Management. Treat the dashboard as operational evidence, not a perfect census of every automated request.
- Confirm that every production hostname serves a valid
/robots.txt. - Export or record the busiest crawlers, requested paths, response codes and transferred bytes.
- Separate AI training crawlers, AI assistants, search crawlers and unidentified automation where the available data permits it.
- Compare allowed crawling with referrals, leads or other useful outcomes.
- Investigate sudden request growth, repeated fetching and access to expensive or commercially sensitive endpoints.
Free-plan AI Crawl Control metrics currently cover a maximum 24-hour window, so businesses needing a longer baseline should supplement them with existing CDN, web-server or analytics records. Thirty days is a practical first review period when those records are available.
Robots.txt is the notice; edge rules are the enforcement
A robots.txt directive communicates your preference to a named crawler. It does not authenticate the requester or force compliance. Cloudflare can monitor requests to disallowed paths and can enforce decisions through its Web Application Firewall, including path-specific exceptions.
Check both the published file and the response an identified crawler receives:
curl -i https://example.com/robots.txt
curl -I -A "CrawlerUserAgent" https://example.com/research/sample-pageReplace the user agent and path with real values from the relevant operator’s current documentation and your policy. A simulated user agent only tests the response associated with that declaration; it does not prove how stronger bot-verification signals will classify a genuine request.
Cloudflare’s violation reporting needs similarly careful interpretation. It compares current directives with historical requests, so changing a rule today can make older, previously permitted requests appear as violations. Check timing before escalating an apparent breach.
Measure what allowed access gives back
Request totals alone do not tell you whether access is worthwhile. Review the most-requested paths, bandwidth, status codes and crawler operators alongside referrals and downstream business activity. Paid Cloudflare plans can provide AI referral analytics, but normal web analytics and CRM attribution remain useful for checking whether visits become enquiries, sales or product usage.
This avoids two costly mistakes: blocking an assistant that helps prospective customers discover your services, or serving large volumes of valuable content to a crawler that produces no observable return. Not every benefit will appear as a conventional click, so combine quantitative data with commercial agreements, citation visibility and the strategic purpose of the content.
Keep Pay Per Crawl in the experiment column
Cloudflare’s Pay Per Crawl expands the choice from allow or block to allow, charge or block. As of August 2026, however, it remains a closed beta—not a dependable revenue stream that should drive a crawler strategy.
For eligible sites, a sensible test would start with a clearly defined, high-value section while leaving discovery pages accessible. Establish the content covered, price hypothesis, crawler eligibility, accounting owner and success criteria before enabling payment responses. Existing WAF blocks still take precedence, so charging must be designed as part of the wider ruleset.
Make crawler governance a maintained business policy
Assign one accountable owner, keep rules under change control and review them quarterly or after a material change in traffic. Marketing should define discovery value, editorial or product teams should classify content, and technical operations should implement and monitor the controls. Legal input may be needed for licensing language, but it should not replace technical enforcement.
If the dashboard is full of bot names but nobody can explain what should be allowed, Greg can help turn the evidence into a path-level policy, implement the Cloudflare and robots.txt controls, and establish a review process tied to business outcomes. Discuss the site and its crawler policy.
Related on GrN.dk
- ChatGPT Visibility Without Open Access: robots.txt Is Only the Start
- AI Crawler Control for Business Websites: Protect Content Without Vanishing from Search
- AI crawler policy now has verbs: separate search, RAG, and training
Need help with this kind of work?
Discuss your crawler policy with Greg Get in touch with Greg.
