AI crawler policy now has verbs: separate search, RAG, and training
By Greg Nowak. Last updated 2026-08-02.
“Allow AI” and “block AI” sound decisive, but they no longer describe the choice a publisher is making. The same public article might belong in conventional search, be acceptable as a source for a grounded answer, and remain off-limits for model training. An advertising landing page may need to be available for review without opening the door to unrelated reuse.
The useful questions are more specific: What may this content be used for? Which paths does the decision cover? And which part of the technical stack is responsible for carrying it out?
This is becoming an operational issue, not a theoretical one. The IETF AI Preferences Working Group lists August 2026 milestones for sending an AI preference vocabulary and association mechanisms to the IESG. Meanwhile, major platforms already offer controls that distinguish among search, real-time AI input, training, grounding, advertising, and retrieval initiated by a user. The terms do not line up neatly, but the direction does: crawler policy is becoming purpose-specific.
The vocabulary is getting better, but it is not settled
The active IETF draft expresses a preference as allow, disallow, or unknown for a category of use. Its current core vocabulary has two categories: search and train-ai.
In the draft, search means selecting assets and directing people back to their original location through links and relevance-focused excerpts. Generated summaries are explicitly outside that definition. Training means using an asset to produce or refine a generative AI model.
Those definitions are useful, but the draft remains a work in progress and warns that important sections do not yet represent working-group consensus. It also treats the vocabulary separately from the mechanism used to attach a preference. The group is considering content metadata, delivery protocols, robots.txt, and HTTP response headers. Technical enforcement is outside its charter.
Commercial controls follow similar ideas without using the same categories. Cloudflare Content Signals distinguish conventional search, ai-input for RAG or answer-time grounding, and ai-train. Google-Extended controls whether crawled content may be used for future Gemini training and for grounding at prompt time; it does not affect inclusion or ranking in Google Search.
That last detail matters. A publisher may want to permit grounding and refuse training, but a platform control that bundles the two cannot express that decision precisely.
| Content use | What it means in practice | Controls covered by the sources | Decision to record |
|---|---|---|---|
| Search and discovery | Indexing, links, and short excerpts that send people to the original source | IETF search; Cloudflare search; product-specific search crawlers |
Which public paths should remain discoverable? |
| RAG and grounding | Retrieving content at answer time to help generate a response | Cloudflare ai-input; Google-Extended grounding; search-augmented assistant retrieval |
Is real-time use acceptable, and can the platform express that choice separately? |
| Model training | Using content to produce or refine a generative model | IETF train-ai; Cloudflare ai-train; Google-Extended training |
Which content, if any, may be reused for training? |
| Advertising review | Validating and assessing a submitted landing page | OAI-AdsBot, separate from OAI-SearchBot | Which campaign paths must remain reachable for review? |
A preference is not the same as a restriction
This distinction is easy to miss. The IETF vocabulary can tell a recipient what a publisher prefers, but it cannot make the recipient comply. Cloudflare draws the same line in practical terms: its managed robots.txt feature expresses preferences, while AI Crawl Control can enforce a block. Organisations can use both.
A workable policy matrix therefore needs two separate columns. One should record the declared preference, such as search allowed and training disallowed. The other should record the infrastructure response: allow, challenge, rate-limit, authenticate, or block.
Combining those columns creates false confidence. A robots.txt rule may say that access is allowed while a WAF blocks the request. Conversely, a preference against a particular use does not itself create an access barrier.
The delivered result can also differ from the file someone edited. Cloudflare may prepend managed content to the origin’s robots.txt. A CMS plugin may generate its own version. CDN rules, application-level challenges, or bot mitigation may stop a crawler even when robots.txt permits it. OpenAI’s advertiser guidance lists 403 responses, bot mitigation, CAPTCHAs, JavaScript challenges, authentication, geographic rules, and rate limits as possible barriers. The origin file is only one link in the chain.
Why one robots.txt rule cannot settle the issue
A crawler identity does not always correspond to one use. Google-Extended, for example, is a control token rather than a separate HTTP request user agent, and it covers both training and grounding.
OpenAI’s advertising guidance makes a different split. OAI-AdsBot is required to review landing pages, while OAI-SearchBot is recommended for understanding public web content. A blanket block could interrupt a wanted advertising workflow and still fail to describe the specific form of reuse the publisher wants to prevent.
Nor is a configuration screenshot proof that the policy works. A controlled study published in July 2026 tested ten AI assistants across 200 trials using server logs and secret codes. Results varied substantially. Some assistants followed the expected allow and disallow patterns. Others reached restricted resources without first requesting robots.txt or used generic user agents that made attribution harder. Successful retrieval and a visibly correct answer did not always coincide.
That changes what counts as a useful test. Teams need to inspect requests, responses, and logs, not just ask an assistant a question and judge what appears on screen.
A workable governance process
1. Classify content by path
Begin with content groups the business already recognises: editorial archives, product pages, campaign landing pages, documentation, media libraries, account areas, and licensed material. For each group, decide separately whether it should support discovery, real-time AI answers, training, and specialist functions such as advertising review.
Keep “unknown” as a real policy state. Missing instructions should not quietly be treated as either consent or prohibition.
2. Map the whole control chain
Record the robots.txt response served at the edge, not only the file stored at the origin. Add CMS plugins, HTTP headers, CDN-managed directives, WAF rules, bot-management products, rate limits, authentication, and application challenges.
For every control, name the owning system and note whether it merely expresses a preference or actually restricts access. That makes conflicting rules much easier to spot.
3. Translate business decisions into platform controls
Match each purpose in the policy matrix to the controls a platform currently provides. If one control bundles separate business decisions, document the mismatch. Google-Extended’s combined coverage of training and grounding is a clear example. The policy should not claim a level of precision the implementation cannot deliver.
4. Test representative paths and crawlers
Choose a small set of URLs from every policy class. Check the robots.txt content actually delivered, the HTTP status returned to the relevant crawler, and the corresponding CDN or WAF event.
OpenAI’s guidance calls out 403 and 429 responses in particular. It also recommends combining user-agent identification with verified-bot programmes, firewall rules, robots.txt behaviour, and provider verification instead of depending only on short-lived IP observations.
5. Leave an evidence trail
Each policy decision needs an owner, rationale, implementation location, test result, and review date. Standards drafts and commercial controls will continue to change. The matrix should therefore live as operational documentation, not as a one-off SEO setting that nobody revisits.
Make the policy clear enough to operate
The goal is not to guess which vocabulary will eventually win. It is to state editorial and commercial intent clearly enough that the organisation can adapt as standards and platform controls change.
Greg can help turn that intent into working practice: inventory the live control chain, translate stakeholder decisions into a path-level matrix, test representative crawlers, and document where a preference ends and an enforceable restriction begins.
The result is easier for everyone to use. Content teams can see what has been approved, infrastructure teams know what must be implemented, and commercial teams can revise a decision without treating search, RAG, training, and advertising as one blunt switch.
Related on GrN.dk
- If the Facts Need JavaScript, AI Search May Miss the Full Page
- AI agents need a browser policy before they start clicking around
- Background AI Tasks Need Queues, Not Just Longer API Calls
Need help with this kind of work?
Review your crawler policy with Greg Get in touch with Greg.