If the Facts Need JavaScript, AI Search May Miss the Full Page

By Greg Nowak. Last updated 2026-07-14.

Open a company website in a browser and everything may appear to be in place: service descriptions, prices, navigation, contact details and supporting metadata. An AI crawler can request the same URL and receive far less.

The gap often comes down to JavaScript. A visitor’s browser runs the code that adds content after the initial page loads. A crawler that reads only the original HTML response may never see those additions, even when they contain the facts someone would need to understand or contact the business.

JavaScript itself is not the problem. Interactive calculators, live chat, view counters and similar features can sensibly remain client-side. The risk comes from making essential information depend on JavaScript too: what the company offers, who it serves, where it operates and how a prospective customer can get in touch.

The gap can be measured

A July 2026 study investigated this issue using a purposive sample of 405 North American small-business and B2B websites. Researchers captured each homepage twice. One capture came from a browser that executed JavaScript; the other used the raw HTML returned by an HTTP request with a GPTBot user agent.

Of the 368 sites that produced analysable captures, 25% had at least one “answer-critical” element in the rendered page that was absent from the raw response. Contact information was missing from the raw HTML on 14.7% of those sites.

The study’s definition of answer-critical content reflects the information used in ordinary business discovery: body copy, headings, internal navigation, contact details, forms and structured data. It also found a notable descriptive difference between platform groups. Wix and Webflow, grouped as managed builders that server-render by default, had a lower observed rate of missing answer-critical content than the other platforms in the sample.

There are sensible limits to what can be concluded from these numbers. The sample was not representative, the analysis covered homepages and direct-crawl visibility, and the author disclosed operating commercial website-audit and AI-visibility tools. The study makes a good case for testing individual websites. It does not provide a market-wide forecast or prove that incomplete HTML automatically leads to lost revenue.

A browser check does not tell the whole story

Google’s documentation explains the technical distinction. A traditional or server-rendered page includes its meaningful content in the initial HTML. An “app shell” may initially contain little beyond a framework, leaving JavaScript to construct the useful page later.

Googlebot can queue eligible pages for rendering, but Google still recommends server-side rendering or pre-rendering because not every bot can run JavaScript. That qualification matters when assessing AI search visibility.

Vercel and MERJ reported that the major AI crawlers they measured did not execute JavaScript, even when some of them fetched JavaScript files. Their findings also showed that crawler behaviour is not uniform. Google’s Gemini used Googlebot infrastructure, while AppleBot rendered JavaScript.

So the issue is not that every AI system receives the same thin page. It is that client-side rendering introduces a dependency: visibility now rests on whether a particular crawler can and will execute the code needed to reveal the facts.

Raw HTML versus rendered page: what to check first

Business layer	Compare	Warning sign	Practical response
Offer	Services, products, prices and locations	Important facts appear only after JavaScript runs	Server-render or pre-render the critical content
Page meaning	Title, description and primary heading	Raw metadata is empty, generic or inconsistent	Repair the CMS template and initial HTML
Discovery	Navigation and internal links	Links appear only after an interaction	Include crawlable links in the server response
Business identity	Phone number, email and address	Contact details are absent from the raw response	Place essential contact data in the HTML
Machine-readable facts	Relevant structured data	Markup is injected only into the rendered DOM	Generate consistent structured data server-side
Access	robots.txt, status codes and WAF rules	The crawler is blocked, challenged or redirected	Align access controls with the business’s crawler policy

Start with the pages that matter commercially

There is little value in cataloguing every difference between raw and rendered output. Begin with the pages and facts that affect whether a potential customer can find, assess or contact the business.

For a service company, that could mean capabilities, sectors, geographic coverage, evidence of expertise and contact routes. For an ecommerce business, the priority set may include product names, availability, specifications and prices. This produces an audit tied to business value rather than an abstract technical score.

Compare the response, not only the screenshot

For each priority URL, capture both the raw HTTP response and the fully rendered DOM. Check visible copy, headings, links, metadata, canonical information, contact details and structured data. Record status codes and redirects too.

A page can look correct after ten seconds in a browser while returning an empty shell, incomplete metadata or an error to an automated request. That is why visual testing alone is not enough.

Greg can run this comparison with relevant crawler user agents and then check the findings against server logs. User-agent tests show what the server returns under controlled conditions. Logs reveal whether crawlers actually arrived, which URLs they requested and which responses they received. Used together, they help separate a rendering problem from an access, routing or infrastructure problem.

Check crawler access beyond robots.txt

OpenAI advises publishers not to block OAI-SearchBot if they want their content to be available for summaries and snippets in ChatGPT search. It also says permitted publishers can identify referral traffic through the UTM parameter added to outbound links from ChatGPT. This makes crawler access relevant to a measurable acquisition channel, although allowing access does not guarantee inclusion.

Perplexity describes PerplexityBot as the crawler it uses to surface and link websites in search results. Its documentation recommends allowing the bot in robots.txt and permitting requests from its published IP ranges. It also notes that a web application firewall may need explicit configuration.

In practice, a permissive robots.txt file proves only that the stated policy allows access. A WAF, bot-management service or security challenge can still reject the request before the page is delivered.

Fix the underlying source

The durable solution is usually to place essential information in the initial response through server-side rendering, static generation or appropriate pre-rendering. Depending on the site, that may involve the application framework, CMS theme, content template, metadata component or deployment process.

There is no need to remove every client-side feature. Vercel’s guidance makes the same distinction: critical content should be available without JavaScript, while non-essential enhancements can remain client-rendered.

Work through the fixes in commercial order. Start with pages closest to customer intent and with elements that establish identity, relevance or a route to action. Shared templates deserve attention before individual pages. Correcting a template can repair headings, metadata, navigation or structured data across a whole section; accumulating page-level exceptions creates more maintenance work later.

Keep crawler visibility in the release process

A repaired page can regress when a CMS theme changes, a new component library is introduced or an editor chooses a different content block. Raw-versus-rendered checks should therefore be repeated after material releases and across representative page types.

The acceptance criterion can stay straightforward: agreed business-critical facts must be present, accurate and consistent before client-side JavaScript executes.

Meeting that standard cannot guarantee an AI citation, recommendation or visit. Discovery systems also rely on other signals, caches and third-party sources. It does, however, remove a preventable source of uncertainty. A business should not need every crawler to behave like a modern browser simply to understand what the company does.

If a polished website is exposing only a thin version of the business to automated systems, a focused crawler-visibility audit can show where the facts disappear, whether access controls are contributing to the problem and which rendering or CMS changes should come first.