AI Crawlers: Separate Your Search, Training and Agent Rules
Keep AI search visibility, training preferences and assistant access separate. Use a practical policy matrix to check Cloudflare settings and test key public pages.
Keep AI search visibility, training preferences and assistant access separate. Use a practical policy matrix to check Cloudflare settings and test key public pages.
Cloudflare Free provides useful website protection, but its rate limiting and bot controls have limits. Here is how to assess them for a WordPress site.
Verify AI crawler identities with Web Bot Auth, handle unsigned Google traffic, and build edge rules that keep useful agents moving without broad firewall exceptions.
A DNS record can exist in your dashboard and still be invisible publicly. Check subdomain delegation, glue, caching and DNSSEC to find the right fix.
Identity-aware AI Gateway logs can help control spend and investigate unusual activity. Decide what to record, who can inspect it, and when to delete it.
Cloudflare’s September bot defaults give publishers more control, but one training block could also cut search crawling and AI-driven discovery.
Cloudflare’s DoH JSON changes can break DNS scripts without breaking HTTP requests. Learn what changed, where to look, and how to prioritise a reliable repair.
Passing a challenge is only one signal. Session analysis, server-side validation and endpoint-specific controls help reduce bot abuse without blocking customers.
Cloudflare Cache Response Rules can recover wasted CDN capacity, but first you need a route-level audit of public, personal and authenticated responses.
Cloudflare Service Keys stop working on September 30, 2026. Here is how to find every caller, move to scoped API tokens and avoid a late outage.
Written recommendations from Trafik og Veje, Aarhus Municipality (2011) and AgroTech (2010).