By Greg Nowak. Last updated 2026-08-22.
It takes minutes to create a bad llms.txt file. Building one that is accurate, selective, and worth maintaining is a content-governance project.
The distinction matters because llms.txt is sometimes presented as a shortcut to AI visibility. It is not. Google now states explicitly that Google Search—including its generative AI features—ignores the file. Publishing one will neither improve nor damage your Google rankings.
That does not make the idea useless. It makes the business case narrower and more honest: llms.txt can provide a compact route into your website for agents and tools designed to look for it. For documentation-heavy businesses, software products, universities, agencies, and complex service organisations, that may be useful. For a five-page brochure site, it may be unnecessary overhead.
What llms.txt actually does
The current proposal describes a Markdown file placed at /llms.txt, or within a specific path such as /docs/llms.txt. It contains a site or project name, an optional summary and guidance, then grouped links to more detailed resources. The file is intended to remain concise; the detail lives on the linked pages.
Version 2 of the proposal, updated in August 2026, also recommends clean Markdown alternatives for important pages and standard link relations that help clients discover them. This makes the concept more useful for documentation systems than a lone text file containing links to ordinary, navigation-heavy web pages.
It still has a different job from established web controls. A sitemap supports discovery of indexable URLs. robots.txt communicates crawler access rules. Canonical tags help consolidate duplicate pages. llms.txt is an optional editorial guide for compatible agents; it replaces none of those foundations.
| Situation | Recommendation | Reason |
|---|---|---|
| Large documentation or support estate | Build or pilot | Agents benefit from a curated route to current references, policies, and examples. |
| Complex services, products, or eligibility rules | Pilot selectively | A short approved index may reduce ambiguity, provided the underlying pages are authoritative. |
| Small site with clear navigation | Usually defer | The maintenance cost may exceed the practical benefit. |
| Weak, duplicated, or outdated content | Fix the website first | A machine-readable index only makes existing content problems easier to find. |
| Goal is higher Google visibility | Do not prioritise it | Google says the file has no effect on Search visibility or rankings. |
Start with the questions the file must support
Before selecting a plugin or assigning development time, write down the tasks an agent should be able to complete. A buyer might need to compare services, confirm geography, understand pricing boundaries, or find an escalation route. A developer might need the current API version, authentication instructions, SDK guidance, and error references.
Those tasks determine scope. Include canonical product information, current documentation, approved policies, and genuinely useful support material. Exclude expired campaigns, duplicate landing pages, thin taxonomy archives, draft material, obsolete documentation, and pages whose claims nobody owns.
Do not export every published CMS item simply because automation makes it possible. A complete content dump is usually a poor guide: it increases noise, exposes inconsistencies, and creates a larger maintenance surface.
Make it a publishing workflow, not a one-off file
A workable implementation has four parts:
- Ownership: assign a business or editorial owner for selection and descriptions, plus a technical owner for delivery, caching, and validation.
- Generation: produce the file from approved CMS content where practical. Manual overrides should be visible and reviewable rather than buried in server configuration.
- Update triggers: regenerate or review the file when linked pages are unpublished, redirected, materially changed, or moved between product versions.
- Testing: give an agent only the file as its starting point and test real buying, support, and implementation questions. Check whether it reaches the correct source—not merely whether it produces a fluent answer.
For WordPress, evaluate generators on curation controls, exclusions, previewing, regeneration, multisite behaviour, caching, and ownership—not on promises of “AI rankings.” Check for conflicts if an SEO plugin, custom code, or another extension already serves the route.
Drupal teams have a current contributed module that separates deployable structure from site-specific content and integrates with caching. For supported Drupal 10.3 and 11.1 installations, its current Composer command is:
composer require 'drupal/llms_txt:^1.1'Module installation is only one step. Confirm that the web server routes /llms.txt to Drupal correctly, then review permissions, language handling, cache invalidation, and behaviour across development, test, and production environments.
Validate delivery before debating visibility
Basic command-line checks catch surprisingly common problems:
curl -fsS https://example.com/llms.txt
curl -I https://example.com/llms.txtLook for a successful response, readable UTF-8 text, one clear H1, concise descriptions, valid absolute links, and no unexpected redirects or authentication. Test every linked destination and verify that restricted or non-public material has not slipped into the output.
Chrome’s Lighthouse agentic-browsing audit treats llms.txt as optional: a missing file is marked not applicable, while a server error is flagged. That is a useful operational distinction. If you choose not to provide the file, a clean 404 is better than a broken endpoint.
Measure usefulness, not wishful visibility
There is no credible universal “llms.txt ranking” metric. Instead, monitor whether relevant user agents request the file, whether they follow its links, whether updates happen on time, and whether controlled agent tests reach the right answers. Keep foundational SEO, accessible HTML, useful original content, internal linking, structured data where appropriate, and crawler controls ahead of this experiment.
If your organisation has enough content to justify llms.txt, the hard part will not be writing Markdown. It will be agreeing which information is canonical and keeping that decision current. Greg can help audit the opportunity, design the CMS workflow, validate the output, and decide whether the file deserves a place in your operating model. Start with a practical website and content-governance review.
Related on GrN.dk
- Google’s AI Search Toggle Is a Publishing Decision, Not an SEO Setting
- AI disclosure rules belong in the CMS, not a spreadsheet
- AI images need a media-library audit before they reach clients
Need help with this kind of work?
Discuss your llms.txt and CMS setup with Greg Get in touch with Greg.
