By Greg Nowak. Last updated 2026-11-07.
You may want potential customers to find your services through AI search while declining permission to use your content for model training. You might also want an assistant to fetch a page when someone specifically asks for it. Your website’s crawler rules need to reflect all three decisions.
That distinction matters when someone changes a setting on your behalf. Cloudflare’s crawler policy documentation, updated on November 2, 2026, separates Search, Training and Agent controls. Anthropic has also begun recognizing a new training preference signal. This is a good time to check whether your published rules and access settings still match what the business wants.
Agree on what the business wants to allow
For a service business, a reasonable starting point might be to allow search discovery, decline training and permit selected user-requested visits. A publisher may make different choices. Either way, the person configuring the website needs a clear decision and the business reason behind it.
Use this worksheet to separate those choices. The checks are a suggested audit approach; the settings you choose will depend on the policy you approve.
| Use | Decision to approve | What to check | Evidence to keep |
|---|---|---|---|
| Search discovery | Which search crawlers should reach our public pages? | Do the crawler directives and access settings both support that choice? | Verified requests to priority pages, with their outcomes. |
| Model training | Should our content be available for training? | Are we publishing a preference, enforcing a block, or doing both? | Served directives, configuration settings and agreed exceptions. |
| User-requested retrieval | Which assistants should fetch pages when a user asks? | Do the agent rules allow the visits we have approved? | Observed retrieval results and relevant security events. |
Give the completed worksheet an owner and a review date. A handover note that says “allow AI” leaves the next administrator to interpret a commercial decision that should already have been made.
Check what “Disallow” and “Block” actually do
Cloudflare distinguishes a Training Disallow preference in robots.txt from a block on matching requests. Its documentation describes robots.txt directives as voluntary. With Bot Preference Sync enabled, Disallow also blocks training crawlers at Cloudflare’s edge, but excludes mixed-purpose search and training crawlers from that edge block. The details are in Cloudflare’s access policy guidance.
Mixed-use crawlers make this a business decision as well as a configuration task. Cloudflare’s September announcement explains that Training Block and Block on pages with ads now include mixed-use crawlers. Those settings can therefore affect search access. Search Allow, on its own, does not establish that every search crawler you want can reach the site. Cloudflare’s explanation of mixed-use crawlers sets out the consequences.
The same announcement identifies a gap: the no-training preference is not automatically conveyed to Bing through robots.txt until Microsoft’s planned support arrives, targeted for early 2027. Record that as a planned capability when reviewing the available controls.
Before approving a change, ask the implementer to explain its expected effect in plain English. If continued discovery matters, name the crawlers that must remain accessible and agree which restrictions the business will accept.
Each operator treats search and retrieval differently
Anthropic identifies ClaudeBot for potential training collection, Claude-SearchBot for search and Claude-User for user-directed retrieval. Its guidance says restricting the latter two can reduce visibility in their respective contexts.
On October 30, 2026, Anthropic began recognizing a new machine-readable training preference. It describes a gradual rollout with an implementation that is still evolving. Its crawler and training preference guidance also retains explicit robots.txt instructions for ClaudeBot. Keep that distinction in the implementation notes so the new signal is assessed alongside the existing instructions.
Perplexity describes PerplexityBot as a search crawler that does not collect content for foundation-model training. It recommends permitting the crawler in robots.txt and allowing its published IP ranges. Perplexity-User handles user-requested visits and generally ignores robots.txt rules. When configuring firewall access, its crawler documentation recommends matching each bot’s user-agent against that bot’s own IP ranges.
Add an entry for each relevant operator to your worksheet. Record its stated behavior, your chosen control and any checks still outstanding. One robots.txt rule cannot be assumed to govern every kind of assistant visit.
Compare the approved policy with the live configuration
Save the current robots.txt response and relevant Cloudflare settings before making changes. Record the domain, date and intended policy alongside them. That baseline gives everyone something concrete to review and provides a reference if a change needs to be reversed.
Then compare the approved decisions with what the website actually serves. Check the main site and each relevant subdomain separately. Anthropic explicitly tells owners to apply its opt-out rules to every subdomain they want covered, so include those subdomains in the work.
Describe discrepancies precisely. “Training preference missing from the served file” tells an administrator what needs attention. “AI configuration needs improvement” creates another round of questions. Each finding should have a proposed correction, a responsible person and a check that will show whether the correction worked.
If the business wants to allow Perplexity, inspect the relevant firewall conditions and rule order too. Perplexity’s documentation warns that Cloudflare Bot Fight Mode cannot be skipped through WAF custom rules. Test any exception against the complete site configuration before treating it as effective.
Test the pages that matter to the business
Choose a small set of representative public pages. A homepage, core service page, important article and public document could be a useful starting sample. Have the owner confirm that the selection covers the content whose accessibility matters most.
Write down the expected result for each page before testing. An approved search crawler may need to receive the content. A training crawler may need to be blocked, or receive a preference it is expected to honor. A selected assistant’s user-requested fetch may need to succeed. Agreeing on those outcomes first makes disagreements easier to resolve before settings change.
Where request records are available, inspect actual crawler visits. Keep the timestamp, requested URL, available evidence of the crawler’s identity, response outcome and security action. For Perplexity, use the bot-specific identity checks described above.
A manually simulated request can help test the configuration. Label it clearly: it does not prove that the operator itself successfully visited the page.
Be equally precise about the conclusions. A served directive shows that you published a preference. A recorded blocked request shows the access decision for that particular request. Neither, by itself, establishes how previously collected material is used. Leave unresolved questions visible in the handover.
Leave the next administrator a usable record
The audit should leave the business with an approved policy matrix, a tested configuration record and a review schedule. Include exceptions, outstanding checks and reversal instructions. Another administrator should be able to understand the decisions without having to reconstruct the project.
Set a review date, along with reasons to check sooner: a crawler-policy update, a firewall change or a new public subdomain, for example. At each review, compare the business’s choices with the current implementation and repeat the agreed page checks.
When these decisions sit between marketing, your website supplier and IT, Greg can help scope and coordinate a crawler governance audit through GrN.dk. Bring the business priorities and access to the relevant configuration. Agree on the pages to test, the decisions that need approval and the evidence required for a clear handover.
Related on GrN.dk
- AI Content on Google: Which Pages Are Worth Keeping?
- AI Search Visibility: Are Your Snippet Rules Working Against You?
- Apache 2.4.68: What to Check Before You Call It Patched
Need help with this kind of work?
Discuss a crawler governance audit Get in touch with Greg.
