Join my community / free newsletter — sign up here

crawler governance

By Greg Nowak , 2 June, 2026

ChatGPT Visibility Without Open Access: robots.txt Is Only the Start

Want ChatGPT visibility without opening every door? Align robots.txt, indexing, WAF rules, canonicals and measurement around a clear policy.

Tags

Review Greg on Google

Written recommendations from Trafik og Veje, Aarhus Municipality (2011) and AgroTech (2010) — read them on LinkedIn.

From Supplier Invoice to Bookkeeping: AI with a Control Checkpoint

2026-08-18

AI can reduce the work involved in processing supplier invoices, but reliable bookkeeping requires validation, duplicate checks, approval and a clear audit trail.

Nginx 1.30 Changed the Upstream Defaults—Test Before You Upgrade

2026-08-17

Nginx 1.30 defaults upstream proxying to HTTP/1.1 with keepalive enabled. Here is what to inspect, model and test before upgrading.

OpenAI’s Assistants API Shuts Down in Ten Days. Is Your App Ready?

2026-08-16

OpenAI’s Assistants API shuts down on August 26, 2026. Learn what to inventory, how to preserve state and how to cut over without breaking the product.

WordPress 7.1 Forces the Editor Into an iframe—Test Your Custom Blocks

2026-08-15

WordPress 7.1 removes the non-iframe editor fallback. Learn how to audit custom blocks, test real workflows and fix compatibility issues before launch.

GitHub will stop sending jobs to stale self-hosted runners

2026-08-14

GitHub starts enforcing runner versions on August 24, 2026. Audit and upgrade self-hosted runners before builds and deployments start stalling.

Your AI Agent Has Shell Access. What Can It Reach?

2026-08-13

A practical guide to mapping what a shell-enabled AI agent can reach, then containing its access to files, credentials, networks, tools, and high-impact actions.

Cloudflare Changed DoH JSON. What Else Is Parsing DNS as Text?

2026-08-12

Cloudflare’s DoH JSON change exposes brittle DNS parsing. Find affected scripts, test both formats, and choose a safer integration contract.

Your Website Can Answer Questions Now. Should It?

2026-08-11

NLWeb makes conversational website search practical to deploy. The real question is whether your content, users and team are ready to support it.

AI Search Finally Has Reports. Now Connect Visibility to Revenue

2026-08-11

Google and Bing now expose first-party AI search data. The real task is connecting citations and impressions to analytics, CRM outcomes, and revenue.

The Bot Passed Your CAPTCHA. What Did It Do Next?

2026-08-11

Passing a challenge is only one signal. Session analysis, server-side validation and endpoint-specific controls help reduce bot abuse without blocking customers.