By Greg Nowak. Updated 2 November 2026.
A business can want its service pages found in search, its expertise cited in AI answers and its paid research kept out of model training. Those are separate commercial decisions. They are hard to manage when the only record is a few lines in robots.txt.
An AI crawler licensing register is a shared record of those choices: which content, which use, which permission, who approved it and how it is implemented. A spreadsheet is enough. Its value is giving marketing, operations, developers and agency teams a clear answer when someone asks, “Why are we allowing this bot?”
Decide what each type of content is for
Start with content that has a clear business purpose: service pages, public guides, paid reports, product feeds and client material. Check who owns the rights before considering licensing. Permission to publish a commissioned image or customer contribution may have limits.
For each content group, decide separately about search discovery, AI answers, training, user-requested retrieval and licensed reuse. The matrix below frames the conversation; it does not prescribe a policy.
| Use | Business question | What to record or check |
|---|---|---|
| Search discovery | Should prospects find these pages? | Indexing settings, crawler rules and representative URLs |
| AI answers | Is citation or summarisation useful here? | Provider-specific search or grounding controls and their limits |
| Model training | Do we accept this use or require separate terms? | Relevant training controls and the approved decision |
| User-requested retrieval | Can an assistant open the content for a user? | Authentication, edge rules and verified request evidence |
| Licensed reuse | Which assets and uses can we offer commercially? | Rights ownership, terms, agreements and renewal dates |
Give each decision an owner and a working reference
Each entry needs the content scope, intended use, decision and business reason. Add the provider or crawler token, the live configuration or license reference, an approver, an implementation owner, the last test result and a review date. Record exceptions with an expiry date.
For example, a hypothetical entry for public guides could say: “Permit ChatGPT search discovery; decline GPTBot training access.” Link it to the relevant rules and a tested guide URL. A paid report needs its own entry, including how access is controlled.
For an agency-managed site, agree who changes the CMS, hosting and content delivery network (CDN), and who approves commercial terms. Keep the register accessible to the client. It should explain a deployment decision without requiring someone to reconstruct an old email thread.
Match the crawler control to the intended use
OpenAI documents independent controls: OAI-SearchBot supports ChatGPT search, while GPTBot crawls content that may be used for model training. ChatGPT-User handles certain user-initiated visits; OpenAI says robots.txt rules may not apply to those requests. Allowing search access does not guarantee a citation.
Google’s controls differ. Google-Extended covers specified Gemini training and grounding uses. Blocking it therefore also declines those grounding uses. Google says it does not affect Google Search inclusion or ranking. It is a control token, with no separate HTTP user-agent string to find in logs.
If that combination matches the approved policy, these groups illustrate declining GPTBot access and the uses controlled by Google-Extended:
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /Merge this into the existing file after checking for overlapping rules. It does not itself allow or block Googlebot or OAI-SearchBot. Under the Robots Exclusion Protocol, a matching crawler group takes precedence over the wildcard fallback. Adding a specific group with Allow: / can leave restrictions from the wildcard group unapplied. Carry forward any restrictions that should still govern that crawler.
Keep access, indexing and licensing distinct
Google’s robots.txt guidance explains the limits: crawler instructions do not secure content, and a blocked page URL can still appear in search. Put private material behind authentication. For public pages excluded from Google Search, use an appropriate noindex directive and allow crawling so Google can read it.
Licensing describes permitted reuse and its terms. If machine-readable licensing fits your publishing model, the RSL 1.0 specification defines an XML format and a robots.txt discovery directive:
License: https://example.com/ai-license.xmlReplace the example with a valid RSL document. Place the directive before any User-agent group if it should be global; RSL also supports group-specific references. The directive does not alter Allow or Disallow rules or ensure every crawler supports RSL. Keep the asset scope, approved terms and signed agreements in the register. Have the appropriate commercial and legal owners review licensing terms.
Check what the live site actually does
Before changing anything, capture the live robots.txt file, relevant CDN or firewall rules and a sample of request logs. Then test a public page, a restricted URL and a downloadable asset against the intended policy.
Check both declared rules and actual responses. A permitted search crawler may still meet a firewall challenge. A user-agent name alone does not establish identity; compare requests with published provider IP information where available. OpenAI publishes IP lists in its crawler documentation. A request made with a copied bot name only tests how your site treats that name.
Save the result, deployment reference and previous configuration. Review after a content launch, licensing agreement, infrastructure change or provider documentation update. Record “not yet verified” when evidence is missing.
If these decisions currently sit between marketing, operations and your agency, Greg can help map the content, agree ownership and turn the policy into tested settings and a maintainable register. Discuss a crawler permissions review with Greg.
Related on GrN.dk
- AI crawler policy now has verbs: separate search, RAG, and training
- ChatGPT Visibility Without Opening Every Door: robots.txt Is Only the Start
- Bot traffic now exceeds human traffic—and crawler rules are no longer optional
Need help with this kind of work?
Discuss a crawler permissions review with Greg Get in touch with Greg.