AI crawler control

By Greg Nowak, 17 June, 2026

AI bot traffic just beat humans, and crawler rules are no longer optional

Bots now make up more web traffic than humans, and Cloudflare's AI crawler controls give site owners a practical way to audit, block, allow, or price access.

AI Crawler Control for Business Websites: Protect Content Without Vanishing from Search

A practical AI crawler policy for business websites: separate search visibility from model training, protect private content, and test every control layer.

Tags

AI crawler control

Review Greg on Google

A stray Set-Cookie can waste your CDN: audit the cache at the edge

2026-07-27

Cloudflare Cache Response Rules can recover wasted CDN capacity, but first you need a route-level audit of public, personal and authenticated responses.

Shorter TLS certificates expose every renewal you never automated

2026-07-26

Shorter TLS lifetimes leave less room for manual handoffs and faulty deploy hooks. Build a renewal path that protects service availability.

One Timeout, Two Orders: Make AI Actions Safe to Retry

2026-07-25

A timed-out AI action may already have succeeded. Stable keys, durable ledgers, queues and stored results prevent a routine retry from duplicating real work.

Your AI Visibility Dashboard Needs a Methodology, Not More Charts

2026-07-24

A practical framework for measuring AI-search visibility with fixed prompts, repeated tests, separate metrics, retained evidence, and honest reporting.

AI Admin APIs Are Here—But Your Directory Is Still the Source of Truth

2026-07-23

New AI admin APIs can automate access and spend controls, but reliable governance still starts with authoritative directory data and clear ownership.

OpenAI Presence Arrived—But Is Your Workflow Ready for an Agent?

2026-07-22

Before an AI agent can take on real work, its workflow needs clear scope, permissions, handoffs, evaluation cases, and production monitoring.

Chatbot Transcripts Quietly Became a Retention and Redaction Problem

2026-07-21

Chatbot transcripts spread across providers, logs and support tools. Here is how to map each copy, redact sensitive data and test deletion properly.

Cloudflare Service Keys Stop in September: Find Every Caller

2026-07-20

Cloudflare Service Keys stop working on September 30, 2026. Here is how to find every caller, move to scoped API tokens and avoid a late outage.

Your AI Workflow Needs an Acceptance Test Before It Meets Customers

2026-07-19

A practical way to test AI workflows using realistic scenarios, tool checks, human rubrics, regression suites, and clear release gates.

Three cover candidates for The Goats Were Load-Bearing fanned on a dark background: an ember-lit door, three slow knocks, and a founders' ledger

The Goats Were Load-Bearing: a fantasy where the bill always comes due

2026-07-19

A teaser for the upcoming darkly comic fantasy novel The Goats Were Load-Bearing — a village, a door that must stay poor, and the worst possible time to sell the herd. Readers pick the cover.