crawler governance

By Greg Nowak, 2 June, 2026

ChatGPT Visibility Without Open Access: robots.txt Is Only the Start

Want ChatGPT visibility without opening every door? Align robots.txt, indexing, WAF rules, canonicals and measurement around a clear policy.

Tags

Review Greg on Google

Cloudflare Service Keys Stop in September: Find Every Caller

2026-07-20

Cloudflare Service Keys stop working on September 30, 2026. Here is how to find every caller, move to scoped API tokens and avoid a late outage.

Your AI Workflow Needs an Acceptance Test Before It Meets Customers

2026-07-19

A practical way to test AI workflows using realistic scenarios, tool checks, human rubrics, regression suites, and clear release gates.

Three cover candidates for The Goats Were Load-Bearing fanned on a dark background: an ember-lit door, three slow knocks, and a founders' ledger

The Goats Were Load-Bearing: a fantasy where the bill always comes due

2026-07-19

A teaser for the upcoming darkly comic fantasy novel The Goats Were Load-Bearing — a village, a door that must stay poor, and the worst possible time to sell the herd. Readers pick the cover.

Vegan Power game: the yellow player catches falling fruit while a chicken and a cow look on

Vegan Power: The Little Game About Eating Fruit, Not Friends

2026-07-19

Vegan Power is a free browser game where you catch fruit, dodge the animals, protect seven hearts, and chase a better high score.

KotobaMon: Shipping a 3D Browser Game With No Build Step and Self-Hosted Voice

2026-07-19

A look at fantasy.grn.dk, a browser-based 3D game that teaches Japanese with no build step, procedural art and self-hosted AI voice, and what its constraints show about shipping interactive products fast and cheap.

WordPress Forced an Emergency Update. Did Every Site Take It?

2026-07-18

WordPress pushed an emergency security update, but teams still need to confirm the right patched version and core integrity on every installation.

IndexNow: Wire corrections and deletions into the CMS

2026-07-17

IndexNow works best when CMS workflows report updates, redirects and removals as well as new pages. Here is how to cover the full content lifecycle.

The EU’s August AI Deadline Reaches Bots and Synthetic Content

2026-07-16

Article 50 applies from 2 August 2026. Businesses need to map AI touchpoints, clarify ownership, and put workable transparency controls in place.

A Voice Agent Is Only Ready When the Human Handoff Works

2026-07-15

A practical guide to voice agents that recognise failure, pass useful context to staff, protect customer data, and improve resolution after launch.

If the Facts Need JavaScript, AI Search May Miss the Full Page

2026-07-14

A practical guide to finding and fixing JavaScript rendering gaps that can hide services, prices, contact details and metadata from AI search crawlers.