Test Whether llms.txt Is Used on Your Website

Before investing heavily in llms.txt, ask a narrow question you can measure: does the file receive relevant requests on your site? That is easier to establish than whether it changes AI visibility, and the two questions should not be confused.

Define the test before creating the file

Record the publication date, file contents, response status and logging coverage. Decide which request types you will inspect and how you will distinguish verified agents, unknown clients and your own checks.

If your CDN serves the file without origin requests, origin logs alone may miss access. Confirm which logging layer captures the event you intend to count.

Collect a clean observation record

Exclude your own browser checks and monitoring probes from the main request count, but retain them in the raw log with a label.

Worked example: access without outcome

Suppose a hypothetical four-week test records three relevant file requests and no detectable change in a fixed AI prompt panel. The result supports limited observed access during that period. It does not prove the file is useless everywhere, nor that the requests improved visibility.

Likewise, zero recorded requests is meaningful only within the completeness of your logs and the chosen window. State those boundaries.

Avoid bundled changes

If you publish llms.txt while rewriting pages, changing internal links and launching a PR campaign, later visibility changes cannot be attributed to the file alone.

Keep a change log. For a stronger outcome investigation, choose a design and comparison suited to your site, acknowledging that external answer systems introduce substantial variability and limited observability.

Decide the appropriate maintenance level

Compare observed use and maintenance cost. A small generated file may require little effort; a manually maintained parallel documentation system may become expensive and stale.

Do not divert effort from accurate pages, crawlable navigation and useful documentation merely because a new file is easy to add. Google’s AI search guidance does not require a special AI text file for eligibility in its features.

The practical result is a documented access finding and a proportionate maintenance decision, not a promise that one file controls AI discovery.

What the file is, and what claims surround it

llms.txt is a proposed convention: a plain-text or Markdown file at a site's root that lists the pages an author considers most useful for language-model systems, sometimes with short descriptions. It is simple to create, which is part of its appeal and part of the problem. Simplicity invites strong claims that outrun the evidence, and the claims cluster into a few types:

• Adoption claims: "Major providers read it." Check each provider's own documentation for a statement about the file. Absence of a statement is not evidence either way, but it is not adoption.
• Outcome claims: "Sites with the file get cited more." These are usually correlations across sites that also differ in size, documentation quality, and everything else.
• Mechanism claims: "It tells the model what your site is about." Retrieval systems fetch pages; whether any consult this file first, and what they do with it, is not publicly specified.

The test in this article is designed around that uncertainty. It measures the one thing you can observe, requests for the file, and keeps that strictly separate from the outcome you might hope for.

Make the log evidence complete

Access evidence is only as good as the logging layer, and a zero can mean "nobody asked" or "we could not see". Before the test window opens, confirm these points and write them down:

Then record a baseline: requests for the path in the weeks before the file existed will be 404s, and their user-agents tell you which clients were probing for it already. That is a genuinely interesting observation on its own, and it is free.

Report the result at three levels

The finished test produces three separate statements, and keeping them separate is the entire discipline:

1. Access: how many requests for the file, from which verified agents, over which dates, with the logging coverage stated. Example: "Three verified requests from two documented agents in 28 days; edge logs complete, origin logs partial."
2. Sequence: whether pages listed in the file were fetched by the same agents after the file was requested. Example: "No listed page was fetched within 72 hours of a file request." A sequence is suggestive; it is not a mechanism.
3. Outcome: the fixed prompt panel's results across the period, with variation, and the change log of everything else that happened to the site. Example: "Panel mentions ranged as before; two page rewrites also shipped in week three, so no attribution is possible."

The decision at the end is about maintenance. If the file is generated automatically from your sitemap and page descriptions, its cost is near zero and keeping it is reasonable regardless of the outcome. If someone is hand-maintaining a second description of the site, that effort competes directly with keeping the actual pages accurate, and the observed-access finding is what should decide whether it continues.

Put this into practice

Copy the worksheet columns below into a spreadsheet and keep one row per item you check. The filled row is an illustrative example, not a reported customer result; replace it with your own verified records.

Use the following prompt only after supplying the records it requests:

Summarize these llms.txt request logs. Separate verified agents, unknown clients and internal checks. State logging coverage and observation window. Do not infer citation impact from requests or assume a later page fetch came from this file.

Research context

The llms.txt study offers an observational starting point; this guide proposes a site-specific access test rather than adopting a universal outcome claim. The related Ahrefs starting points are We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read and 10 SEO Trends I’ve Seen Firsthand in 2026 (With Data). This guide’s checklist, examples and proposed workflow are independently written; they are not results of a SEOVision experiment.

Continue with the next task

• Separate AI Crawlers, Search Bots and User Fetches
• Structured Data for AI Search: What It Can—and Cannot—Do
• AI Search Optimization: A Practical Guide Beyond GEO Hype

Sources

• We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Research starting point; not an endorsement of this original workflow
• 10 SEO Trends I’ve Seen Firsthand in 2026 (With Data) — Research starting point; not an endorsement of this original workflow
• Google: AI features and your website — Primary documentation for the stated platform behavior