Why AI crawler permissions belong in a licensing register

Illustrated infographic summarizing: AI crawler permissions now belong in a licensing register

By Greg Nowak. Updated 2 November 2026.

A business can want its service pages found in search, its expertise cited in AI answers and its paid research kept out of model training. Those are separate commercial decisions. They are hard to manage when the only record is a few lines in robots.txt.

An AI crawler licensing register is a shared record of those choices: which content, which use, which permission, who approved it and how it is implemented. A spreadsheet is enough. Its value is giving marketing, operations, developers and agency teams a clear answer when someone asks, “Why are we allowing this bot?”

Decide what each type of content is for

Start with content that has a clear business purpose: service pages, public guides, paid reports, product feeds and client material. Check who owns the rights before considering licensing. Permission to publish a commissioned image or customer contribution may have limits.

For each content group, decide separately about search discovery, AI answers, training, user-requested retrieval and licensed reuse. The matrix below frames the conversation; it does not prescribe a policy.

Use Business question What to record or check
Search discovery Should prospects find these pages? Indexing settings, crawler rules and representative URLs
AI answers Is citation or summarisation useful here? Provider-specific search or grounding controls and their limits
Model training Do we accept this use or require separate terms? Relevant training controls and the approved decision
User-requested retrieval Can an assistant open the content for a user? Authentication, edge rules and verified request evidence
Licensed reuse Which assets and uses can we offer commercially? Rights ownership, terms, agreements and renewal dates
Create a separate register entry wherever the content, intended use or permission differs.

Give each decision an owner and a working reference

Each entry needs the content scope, intended use, decision and business reason. Add the provider or crawler token, the live configuration or license reference, an approver, an implementation owner, the last test result and a review date. Record exceptions with an expiry date.

For example, a hypothetical entry for public guides could say: “Permit ChatGPT search discovery; decline GPTBot training access.” Link it to the relevant rules and a tested guide URL. A paid report needs its own entry, including how access is controlled.

For an agency-managed site, agree who changes the CMS, hosting and content delivery network (CDN), and who approves commercial terms. Keep the register accessible to the client. It should explain a deployment decision without requiring someone to reconstruct an old email thread.

Match the crawler control to the intended use

OpenAI documents independent controls: OAI-SearchBot supports ChatGPT search, while GPTBot crawls content that may be used for model training. ChatGPT-User handles certain user-initiated visits; OpenAI says robots.txt rules may not apply to those requests. Allowing search access does not guarantee a citation.

Google’s controls differ. Google-Extended covers specified Gemini training and grounding uses. Blocking it therefore also declines those grounding uses. Google says it does not affect Google Search inclusion or ranking. It is a control token, with no separate HTTP user-agent string to find in logs.

If that combination matches the approved policy, these groups illustrate declining GPTBot access and the uses controlled by Google-Extended:

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

Merge this into the existing file after checking for overlapping rules. It does not itself allow or block Googlebot or OAI-SearchBot. Under the Robots Exclusion Protocol, a matching crawler group takes precedence over the wildcard fallback. Adding a specific group with Allow: / can leave restrictions from the wildcard group unapplied. Carry forward any restrictions that should still govern that crawler.

Keep access, indexing and licensing distinct

Google’s robots.txt guidance explains the limits: crawler instructions do not secure content, and a blocked page URL can still appear in search. Put private material behind authentication. For public pages excluded from Google Search, use an appropriate noindex directive and allow crawling so Google can read it.

Licensing describes permitted reuse and its terms. If machine-readable licensing fits your publishing model, the RSL 1.0 specification defines an XML format and a robots.txt discovery directive:

License: https://example.com/ai-license.xml

Replace the example with a valid RSL document. Place the directive before any User-agent group if it should be global; RSL also supports group-specific references. The directive does not alter Allow or Disallow rules or ensure every crawler supports RSL. Keep the asset scope, approved terms and signed agreements in the register. Have the appropriate commercial and legal owners review licensing terms.

Check what the live site actually does

Before changing anything, capture the live robots.txt file, relevant CDN or firewall rules and a sample of request logs. Then test a public page, a restricted URL and a downloadable asset against the intended policy.

Check both declared rules and actual responses. A permitted search crawler may still meet a firewall challenge. A user-agent name alone does not establish identity; compare requests with published provider IP information where available. OpenAI publishes IP lists in its crawler documentation. A request made with a copied bot name only tests how your site treats that name.

Save the result, deployment reference and previous configuration. Review after a content launch, licensing agreement, infrastructure change or provider documentation update. Record “not yet verified” when evidence is missing.

If these decisions currently sit between marketing, operations and your agency, Greg can help map the content, agree ownership and turn the policy into tested settings and a maintainable register. Discuss a crawler permissions review with Greg.

Related on GrN.dk

Need help with this kind of work?

Discuss a crawler permissions review with Greg Get in touch with Greg.

Sources

Latest articles

Sending an invoice does not mean it has arrived. Understand delivery statuses, the system requirements coming in 2027, and how focused AI support can help finance teams handle errors and follow-up.

Before scaling AI content, check the pages you already have. Verify claims, resolve overlapping coverage, and give WordPress reviewers control over publication.

Build a refund approval workflow in n8n and Stripe that shows reviewers the exact action, expires old approvals and tracks refunds through to completion.

A timeout can hide a successful job creation. Use stable operation IDs, destination API rules and a clear recovery process to prevent duplicate work.

Does your ChatGPT subscription cover website translation? Understand API credits, access and a WordPress workflow that turns approved Danish content into translation drafts.

An internal AI can keep quoting a deleted policy. Check document updates, indexed passages and cached answers to find where outdated guidance survives.

Codex drew all ten pages of my Ninjai comic from the script Claude wrote in the first test. What made that handover work, and what it cost.

ChatGPT delivered ten painted comic pages; Claude drew one page in code. A practical look at AI tool selection, creative briefs and approval decisions.

Get missing supporting documents under control with a clearly scoped AI workflow that sorts replies, coordinates reminders and gives the bookkeeper a clear follow-up list.

Know what dmesg and journalctl can tell you after a reboot, how to find previous-boot logs, and how to check that retention covers your investigation window.