By Greg Nowak. Last updated 2026-07-07.
AI search has made an old SEO bargain feel much less reliable. The old bargain was clear enough: let search engines crawl the site, win visibility, and turn the resulting visits into revenue, leads, subscribers, support deflection, or product discovery.
Cloudflare's July 2026 AI-search posts argue that this bargain is breaking down. Answer engines can read a page, summarize it, and satisfy the user without sending the visit back. Pew Research Center's data makes the risk less theoretical: when Google showed an AI summary, users clicked a traditional result in 8% of visits, compared with 15% when no AI summary appeared. Links inside the AI summary were clicked in just 1% of visits.
That does not mean every important page needs an answer-engine rewrite this quarter. It means the first serious project is measurement. Which queries trigger AI summaries? Which pages are cited, summarized, fetched, ignored, or crawled over and over? Which automated traffic is search, agent activity, training, SEO tooling, ads verification, feed fetching, social preview, or something else?
Without that map, content teams can spend money polishing pages that already get summarized but rarely visited, while missing pages that still influence revenue.
Clicks are no longer the only useful signal
Pew's research is useful because it shows where the risk clusters. Longer searches were much more likely to produce AI summaries: only 8% of one- or two-word searches did so, while 53% of searches with 10 words or more did. Searches phrased as questions produced AI summaries 60% of the time, and full-sentence searches did so 36% of the time.
For B2B, ecommerce, and publisher sites, those longer and more specific queries often carry buying intent, research intent, or support intent. They are exactly the searches where a user may get enough information from the results page and never arrive on the site.
Cloudflare adds the infrastructure view. Its 2026 posts separate search crawlers, user-directed agents, and training crawlers. That distinction matters. A search crawler may help people discover the site. A user-directed agent may be acting on behalf of a real person. A training crawler may be collecting content to improve a model. Treating all three as the same traffic creates bad decisions: the site either blocks useful discovery or leaves valuable content more open than the business model can support.
| Dashboard question | Evidence to collect | Decision it supports |
|---|---|---|
| Which queries are most exposed? | Search Console exports, query length, question wording, landing page, and comparison with Pew's AI-summary patterns. | Prioritize long, question-led, and full-sentence queries before rewriting high-value content. |
| Which pages are used by AI search? | Cloudflare traffic data, server logs, AI-search reporting where available, snippets, ranking position, and URL-level crawl patterns. | Decide whether a page needs fresher content, clearer attribution, a stronger conversion path, or tighter access rules. |
| Which bots need different rules? | Cloudflare classifications for Search, Agent, Training, SEO, Ads Verification, Feed Fetching, and other automated traffic. | Set crawler permissions by business purpose instead of using one blanket allow or block rule. |
| Where is crawl waste happening? | Repeated requests to unchanged pages, freshness signals, last-modified behavior, cache behavior, and high-volume bot paths. | Reduce unnecessary crawl load and make freshness part of content operations. |
| Where do licensing signals matter? | Robots.txt content signals, use-level preferences such as immediate, reference, or full, and RSL-related policy choices. | Align technical permissions with commercial policy before licensing or compensation talks begin. |
Start with queries, not page rewrites
The easy move is to start editing pages: add direct answers, restructure headings, create FAQ blocks, refresh dates, and make the copy easier for answer engines to quote. Some of that may help. It should not be the first move.
The better sequence is narrower. Segment queries by likely AI-summary exposure. Map those queries to pages and business outcomes. Compare automated access with human traffic. Then decide whether a page needs a rewrite, a clearer conversion path, a crawler-permission change, or no action.
This is where answer-engine optimization diverges from familiar SEO content work. A page can become more useful to an AI answer and still produce fewer human sessions. That may be acceptable if the page supports brand visibility, product trust, licensing value, or downstream demand. It may be a problem if the page depends on advertising, affiliate clicks, lead forms, or product comparisons that require the visit.
The dashboard should make that tradeoff visible at query and page level. Average organic traffic will hide too much.
Crawler policy is now content policy
Cloudflare's newer AI traffic options make this operational. The company now frames automated traffic around behaviors such as Search, Agent, and Training, and it is adding finer handling for content use. One post describes use levels that range from immediate interaction through reference use to fuller reproduction, with a content signal that can express preferences such as allowing search, not allowing AI training, and setting use to reference.
For site owners, the important part is not just the syntax. Crawler policy is becoming part of content governance.
The Verge's report on RSL 1.0 points in the same direction. RSL is presented as an official licensing standard that lets publishers state licensing and compensation rules to crawlers, extending the role of robots.txt-like signals. The same report notes the hard limit: a signal does not physically stop an AI scraper that ignores it, while infrastructure providers that support the standard can help with enforcement.
That pushes legal, editorial, analytics, and technical decisions into the same room. Crawler settings can no longer sit off to the side as a small SEO preference.
Cloudflare's agentic Internet report explains the urgency. It says more than 50% of Internet traffic is now non-human, that 52% of crawler requests were for AI training as of June 2026, and that mixed-use crawlers represented more than 36% of activity. Mixed-use crawlers are the awkward case: discovery, agent use, and training can be hard to separate. If a business cannot see the purpose of automated access, it cannot make a clean decision about what to allow, block, price, or monitor.
Build a dashboard people will actually maintain
A useful AI-search dashboard should be small enough to run every month and specific enough to change decisions. Start with the content paths where the click matters most: commercial category pages, comparison pages, support articles, research pages, editorial explainers, pricing pages, and any page that earns value from a visit rather than from a mention.
For each group, connect five views: query type, AI-summary exposure risk, bot access by purpose, freshness or recrawl behavior, and business dependency on the click.
That last view is often missing. A publisher article, B2B service page, ecommerce category, and documentation page can all appear in AI answers. The business reading is different in each case. Some pages should be optimized so they are cited accurately. Some should push harder toward owned conversion paths. Some should expose only reference-level use. Some may need licensing or compensation discussions. Some may simply need better freshness signals so crawlers stop refetching unchanged content.
Cloudflare says more than half of good-bot crawl traffic goes to re-fetching pages that have not changed, so this is not only a visibility issue. There is an operational cost as well.
For GrN.dk, this is the kind of project where a digital project manager can keep the work grounded. Greg could audit Cloudflare bot and search data, server logs, Search Console exports, CMS metadata, and high-value content paths, then define KPIs that marketing and technical teams can maintain. The useful deliverable is a working measurement loop: query groups, page cohorts, crawler permissions, freshness routines, governance owners, and a review rhythm.
The practical response
AI search is eating some of the click. A blind rewrite is still the wrong response.
The practical response is to measure where the click is disappearing, where AI visibility still creates value, and where crawler access no longer matches the business model. Once that evidence exists, page rewriting becomes targeted work instead of a content-wide reflex. The teams that move first will know which queries, bots, pages, and policies deserve attention.
Related on GrN.dk
- Web Search and Citation Controls Turn AI Research Assistants Into a Source-Governance Project
- AI bot traffic just beat humans, and crawler rules are no longer optional
- Support bots need a deletion test before they learn the old help center
Need help with this kind of work?
Talk to Greg about AI-search visibility Get in touch with Greg.