Join my community / free newsletter — sign up here

robots.txt

By Greg Nowak , 2 August, 2026

AI crawler policy now has verbs: separate search, RAG, and training

AI crawler rules now need separate decisions for search, RAG, and training, backed by practical testing across robots.txt, CDNs, WAFs, and CMS controls.

Bot traffic now exceeds human traffic—and crawler rules are no longer optional

Bot requests now exceed human web traffic. Learn how to audit AI crawlers, set path-level rules, enforce robots.txt, and measure the business return.

Tags

robots.txt

Review Greg on Google

Written recommendations from Trafik og Veje, Aarhus Municipality (2011) and AgroTech (2010) — read them on LinkedIn.

Search Console Can See TikTok Now. Your Reporting Has to Catch Up

2026-08-23

Google can now report how social profiles appear in Search. Here is how to measure cross-channel discovery without mistaking visibility for business results.

Your AI workflow has logs. Can they explain one bad decision?

2026-08-22

Logs can show that every service worked while leaving a bad AI decision unexplained. See how connected traces and careful redaction close the gap.

Security Questionnaires Eat Into Selling Time—Let AI Find the Evidence

2026-08-21

NIS 2 is generating more supplier questionnaires. A controlled AI assistant can find approved answers and sources—and route uncertain cases for review.

Locked out of your Apple developer account? Fix it before October 1

2026-08-20

Apple's updated developer agreement must be accepted by October 1, 2026, and many small app owners cannot even log in. Here is where Apple's two-factor codes really go, and how to fix your access before the deadline.

Cloudflare Workflows Now Charges by the Step—Price the Outcome

2026-08-20

Cloudflare Workflows now bills paid plans for steps and stored state. Here is how to track cost per completed outcome without weakening reliability.

Google’s AI Search Toggle Is a Publishing Decision, Not an SEO Setting

2026-08-19

Google’s AI Search toggle forces a commercial choice about visibility, attribution and content use. Here’s how to make that choice responsibly.

From Supplier Invoice to Bookkeeping: AI with a Control Checkpoint

2026-08-18

AI can reduce the work involved in processing supplier invoices, but reliable bookkeeping requires validation, duplicate checks, approval and a clear audit trail.

Nginx 1.30 Changed the Upstream Defaults—Test Before You Upgrade

2026-08-17

Nginx 1.30 defaults upstream proxying to HTTP/1.1 with keepalive enabled. Here is what to inspect, model and test before upgrading.

OpenAI’s Assistants API Shuts Down in Ten Days. Is Your App Ready?

2026-08-16

OpenAI’s Assistants API shuts down on August 26, 2026. Learn what to inventory, how to preserve state and how to cut over without breaking the product.

WordPress 7.1 Forces the Editor Into an iframe—Test Your Custom Blocks

2026-08-15

WordPress 7.1 removes the non-iframe editor fallback. Learn how to audit custom blocks, test real workflows and fix compatibility issues before launch.