Join my community / free newsletter — sign up here

crawler governance

By Greg Nowak , 17 June, 2026

Bot traffic now exceeds human traffic—and crawler rules are no longer optional

Bot requests now exceed human web traffic. Learn how to audit AI crawlers, set path-level rules, enforce robots.txt, and measure the business return.

ChatGPT Visibility Without Opening Every Door: robots.txt Is Only the Start

Allow ChatGPT search discovery without opening your whole site. A practical guide to crawler rules, WAF controls, canonicals and measurement.

Tags

crawler governance

Review Greg on Google

Written recommendations from Trafik og Veje, Aarhus Municipality (2011) and AgroTech (2010) — read them on LinkedIn.

Drupal Relaunch: What Happens to Your Content and Old URLs?

2026-09-09

Planning a Drupal relaunch? Set clear rules for content, translations, media and old URLs, with a practical checklist for approving the migration and launch.

AI alt text: How to tackle your online store’s image backlog

2026-09-08

Use AI for your online store’s alt text with a manageable pilot: map the images, generate suggestions in Danish, and check the results in WordPress and WooCommerce.

From Supplier PDFs to Product Data: Where AI Needs a Second Check

2026-09-07

Supplier files need more than extraction. Here’s how to check coverage, match SKUs, resolve unclear units and prices, and test product data before a catalogue import.

Shorter TLS Certificates: Will Your Renewal Setup Keep Up?

2026-09-06

Shorter TLS certificates leave less room for renewal problems. Check domain validation, scheduling, deployment and the certificate your customers actually receive.

Your AI Image Has Content Credentials. Will Your Website Keep Them?

2026-09-05

AI image credentials can disappear during routine website processing. Learn how to test your CMS, optimizer, CDN, and publishing workflow end to end.

What Are Customers Asking? Let AI Find the Patterns in Support Tickets

2026-09-04

AI-based ticket analysis can uncover recurring complaints, product defects and gaps in documentation—without the company needing yet another chatbot.

OpenAI Has Machine Identity Now. Which Jobs Should Lose API Keys?

2026-09-03

OpenAI’s X.509 workload identity can replace API keys for the right workloads. This practical framework helps teams decide where to start safely.

WordPress 7.1 Exposes AI-Ready Actions. Who Gets to Run Them?

2026-09-02

WordPress 7.1 helps AI agents discover and invoke site abilities. Here is how to keep exposure, authentication and permission firmly separate.

From Sales Meeting to CRM: Automate Follow-Up Without Compromising Data Quality

2026-09-01

How to use AI for meeting notes and follow-up while fixed rules protect CRM data, customer matching and the sales pipeline from errors and premature changes.

Your AI Gateway Can Name the User. Decide What That Log Is For

2026-08-31

Identity-aware AI Gateway logs can sharpen security and cost control, but only when attribution, access, retention, guardrails, and response are clearly defined.