Experiment

8 minute read

llms.txt Adoption in the Top 1,000 Sites: November 2026 Data

Our survey of llms.txt adoption in the Tranco top 1,000: 18.7% of reachable domains serve a readable llms.txt, and 21.6% block at least one AI crawler. Method and raw data included.

Abstract illustration for llms.txt Adoption in the Top 1,000 Sites: November 2026 Data

How far has llms.txt adoption spread among the world's most popular domains, and how do their robots.txt files treat AI crawlers? We checked the Tranco top 1,000 on November 15, 2026 with the same code that runs our free llms.txt checker and robots.txt tester. Status: completed. Of the 722 domains that answered, 135 (18.7%) serve an llms.txt file that our checker can read, and 156 (21.6%) block at least one AI crawler from the whole site.

This page sets out the question, the method we wrote down before collecting data, the results and their limits. You can download the raw data, one row per domain, and check every number yourself. It measures llms.txt adoption only. It says nothing about whether llms.txt changes rankings or AI citations.

Question and hypothesis

We asked two questions. First, how many popular domains publish a usable llms.txt file? Second, which AI crawlers do their robots.txt files block from the entire site?

Before collecting any data, we recorded two expectations. We expected fewer than 10% of reachable domains to serve an llms.txt file that our checker grades Good or better. We also expected GPTBot to be the AI crawler blocked most often. Both expectations turned out to be wrong, as the results show.

Setup

The sample is the first 1,000 entries of Tranco list 947V2, created on November 14, 2026. Tranco ranks registrable domains by combining several popularity sources. As a result, the list includes infrastructure domains such as content delivery networks, not only websites.

For each domain, a script requested two files: https://DOMAIN/robots.txt and https://DOMAIN/llms.txt. If the bare domain did not answer, it tried www.DOMAIN once. It followed up to five redirects, waited up to 15 seconds and identified itself as SEOVisionResearch/1.0. It fetched no other pages.

We ran the published collection on November 15, 2026 at about 13:01 UTC from a GitHub-hosted runner in California, United States. An earlier pilot from our editor's own network gave headline numbers within two points of this run. However, some domains failed in different ways there, so we discarded the pilot and publish only the runner data.

Metric and method

Every llms.txt response went through validateLlmsTxt, the same JavaScript that powers our llms.txt checker. A file counts as present when it returns HTTP 200 and the checker does not reject it as fatal. An HTML page, for example, is rejected. Present files keep the checker's grade: Excellent, Good, Needs work or Failing.

Every robots.txt went through the parser behind our robots.txt tester, which follows RFC 9309. A crawler counts as blocked when the file disallows the path "/" for it. The decision comes from a group that names the crawler, or else from the * group. A robots.txt that returns 4xx means no restrictions; no response means unknown.

We report shares of the 722 domains whose robots.txt returned any HTTP response. The 278 domains with no response, mostly hosts without a website at the root, are listed in the data but left out of the percentages.

Timeline

The study ran in one day, in this order.

  • November 14, 2026: Tranco publishes list 947V2.
  • November 15, 2026, morning: protocol and expectations written down, then the list downloaded.
  • November 15, 2026, pilot: test collection from our own network; discarded.
  • November 15, 2026, 13:01 UTC: published collection from the US runner.
  • November 15, 2026: analysis and this page. While analysing the files we found and fixed a bug in our own robots.txt tester, which crashed on rules containing emoji.

Results: llms.txt adoption and crawler blocks

Half of the reachable domains (361, or 50.0%) return 404 for llms.txt. A further 109 (15.1%) return status 200 but serve an HTML page, which is not an llms.txt file. Another 84 (11.6%) answer 403, which usually means bot protection rejected our request.

Adoption is lower at the very top of the list. Among reachable domains ranked 1 to 100, 11 of 72 (15.3%) serve a readable file, compared with 56 of 279 (20.1%) for ranks 101 to 500 and 67 of 371 (18.1%) for ranks 501 to 1,000. Early adopters in the top 100 include cloudflare.com, github.com, adobe.com and wordpress.org, all graded Excellent.

llms.txt resultDomainsShare of 722 reachable
Present and readable13518.7%
Graded Good or Excellent9413.0%
Graded Excellent648.9%
200 but an HTML page, not llms.txt10915.1%
Not found (404 or 410)36150.0%
Blocked (403)8411.6%

Which AI crawlers are blocked

Bytespider and CCBot are blocked most often, ahead of GPTBot and ClaudeBot. In other words, sites most often shut out crawlers associated with large-scale data collection, while search and user-fetch crawlers are blocked less.

The OpenAI rows show the split clearly. GPTBot, which OpenAI uses for training, is blocked by 115 domains, while OAI-SearchBot, which it uses for search, is blocked by 64. Many sites therefore opt out of training while staying reachable for AI search.

CrawlerBlocked from the whole siteShare of 722
Bytespider13018.0%
CCBot12517.3%
GPTBot11515.9%
ClaudeBot11415.8%
Google-Extended9813.6%
PerplexityBot9212.7%
Amazonbot9212.7%
ChatGPT-User7210.0%
OAI-SearchBot648.9%

Blocks that come from the wildcard group

Not every block is deliberate. For 29 domains (4.0%), GPTBot is blocked only because the * group disallows the whole site, not because the file names GPTBot. These sites usually block every crawler they do not name, including AI crawlers they may never have considered.

Conversely, a file that names a crawler replaces the * rules for it. If you name GPTBot to allow it, repeat any rule from the * group that you still want it to follow. Our tester shows which group decides for each crawler.

What did not work or surprised us

Both expectations failed. Readable llms.txt files graded Good or better reach 13.0% of reachable domains, above the 10% we expected. GPTBot is not the most blocked crawler; Bytespider and CCBot are.

The size of the "200 but HTML" group also surprised us. Many sites answer every unknown path with a normal page and status 200, so a simple status check would wrongly count 109 more domains as llms.txt adopters.

Confounders and limits

This is a one-day snapshot from one location. Bot protection that rejected our identified script may let other clients through. Therefore, the 84 blocked responses are unknown, not absent. Some sites may serve different files to different user agents or regions.

The Tranco list mixes websites with infrastructure domains, so the shares describe popular domains, not a random sample of businesses. A present llms.txt also shows only that a file exists and parses; it says nothing about whether any AI system reads it or whether it affects visibility.

What we will do next

We plan to repeat the collection with the same list size every quarter, so the trend in llms.txt adoption becomes visible. The next run is planned for February 2027, using the newest Tranco list on that date.

If you maintain a site in the list and think we misread your files, email contact@seovision.info with the domain. We will check the raw response and correct the data if we got it wrong.

Download the data

The raw data (CSV) has one row per domain: rank, the host we used, robots.txt status, a blocked and a named flag for each of the nine crawlers, and the llms.txt status, content type, grade, score and link count. It contains no response bodies. You may reuse it with a link to this page.

Update log

Dated entries record each change to this page and its data.

  • November 15, 2026: first publication with the November 15 collection.

Frequently asked questions

Quick answers to the questions readers ask most about this topic.

How common is llms.txt adoption among top sites?

On November 15, 2026, 135 of the 722 reachable domains in the Tranco top 1,000 (18.7%) served an llms.txt file our checker could read, and 94 (13.0%) were graded Good or Excellent.

Which AI crawler is blocked most often?

Bytespider (18.0% of reachable domains) and CCBot (17.3%), ahead of GPTBot (15.9%) and ClaudeBot (15.8%). OpenAI's search crawler, OAI-SearchBot, was blocked by only 8.9%.

Does having an llms.txt file improve rankings or AI citations?

This survey cannot say. It measures whether a file exists and parses on one date, not whether any system reads it or whether it changes visibility.

Can I use the data?

Yes. The CSV has one row per domain and contains no response bodies. Please link to this page when you reuse it.

Sources

These pages define the sample, the standard and the file format used above.

  1. Tranco: a research-oriented top sites ranking tranco-list.eu
  2. RFC 9309: Robots Exclusion Protocol rfc-editor.org
  3. llmstxt.org: The /llms.txt file proposal llmstxt.org
  4. OpenAI: Overview of OpenAI crawlers developers.openai.com
  5. Google Search Central: Introduction to robots.txt developers.google.com