See the deciding line
Every verdict names the exact Allow or Disallow line that won, so you can fix the rule instead of guessing.
Free and private tool
Robots.txt tester
Robots.txt tester
and generator.
Paste your robots.txt, pick a crawler, and see exactly which line allows or blocks each URL. Lint common mistakes and build new rule groups—all in your browser, nothing is uploaded.
Evaluated in your browser
Follows RFC 9309
1
Your robots.txt
Paste the file or start from the example
Lint results
No errors+Generate a rule group
Build a User-agent block and add it to the editor
Generate a rule group
Build a User-agent block and add it to the editor
Rules
Preview
Why test first?
A one-character mistake can hide a whole site.
robots.txt is read literally. This tester applies the same precedence rules that Google and other RFC 9309 crawlers use.
Crawler-aware
Choose Googlebot, Bingbot, GPTBot, ClaudeBot and more. Fallback to parent tokens and the * group is handled for you.
Lint before you publish
Typos, rules outside a group, relative sitemap URLs, unsupported directives, and accidental site-wide blocks are flagged.
A practical guide
How crawlers read robots.txt
Three steps decide whether a URL may be fetched. The tester follows them in the same order.
- Pick the group.
A crawler looks for a
User-agentline matching its own product token, case-insensitively. Googlebot-Image first looks forGooglebot-Image, thenGooglebot. Only if nothing matches does it use theUser-agent: *group. Groups are never combined across tokens: a specific group replaces the wildcard group entirely. - Find matching rules.
Every
AllowandDisallowpath in that group is compared against the URL path and query string as a prefix.*matches any characters and a trailing$anchors the end. Matching is case-sensitive. - Apply the longest one.
The rule with the longest path wins. When an Allow and a Disallow are equally long, the least restrictive rule—Allow—is used. If nothing matches, the URL is allowed.
- Remember what robots.txt cannot do.
It controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. Use a
noindexmeta tag or header on pages that must stay out of search, and keep those pages crawlable so the tag can be read.
| Rule | Matches | Does not match |
|---|---|---|
Disallow: / | Every URL on the host | — |
Disallow: /fish | /fish, /fishing, /fish/salmon.html | /Fish, /catfish |
Disallow: /fish/ | /fish/, /fish/salmon | /fish, /fishing |
Disallow: /*.php$ | /index.php, /folder/file.php | /index.php?x=1, /index.php5 |
Disallow: /*? | Any URL containing a query string | /page |
Disallow: (empty) | Nothing; the group allows everything | — |
Mistakes the linter looks for
- Rules before any User-agent line. They belong to no group and are ignored.
- Typos such as
DisaloworUser agent. Crawlers skip unknown fields silently. - Paths without a leading slash.
Disallow: adminnever matches, because every URL path starts with/. - Relative Sitemap URLs. The Sitemap field requires an absolute URL.
Noindexin robots.txt. Google stopped honouring it in 2019.Disallow: /under*. Correct on staging, catastrophic in production.
AI crawlers and robots.txt
OpenAI, Anthropic, Perplexity, Common Crawl and others publish product tokens such as GPTBot, ClaudeBot, PerplexityBot and CCBot. Google uses the Google-Extended token to control training use without affecting Search. The generator's crawler list includes these so you can write a group for each and test it immediately.
Remember that robots.txt is voluntary. Well-behaved crawlers honour it; others may not. For hard limits, use authentication or server-side blocking.
What this tester does and does not do
The tester parses the text you paste, selects the group for the chosen crawler, and reports the deciding rule for each path. It implements RFC 9309 group selection, prefix matching with * and $, longest-match precedence with ties going to Allow, and percent-encoding comparison for non-ASCII paths. Google-specific behaviours that are documented—such as AdsBot ignoring the * group, the 500 KiB size limit, and unsupported Noindex lines—are included.
It does not fetch robots.txt from a live site, follow redirects, or model HTTP status handling (for example, a 5xx on robots.txt causing Google to pause crawling). Test the file you intend to deploy, then confirm with the report in Google Search Console after it goes live. Continue with our technical SEO audit checklist.
Before you deploy
Robots.txt tester FAQ
Is this robots.txt tester free?
Yes. There is no account, quota, or upload. Everything runs in your browser and the file never leaves your device.
Why is my URL allowed even though I wrote a Disallow for it?
Usually one of three reasons: the crawler you chose has its own group that overrides *; a longer Allow rule matches the same URL; or the path is case-sensitive and does not match. The result row names the deciding line so you can tell which one applies.
Does blocking a URL remove it from Google?
No. robots.txt stops crawling, not indexing. A blocked URL can still be indexed from links alone, without a snippet. Use a noindex meta tag or the X-Robots-Tag header on a crawlable page to keep it out of results.
Where does robots.txt have to live?
At the root of each host and protocol, for example https://www.example.com/robots.txt. A file on example.com does not apply to blog.example.com, and a subdirectory path is ignored.
Does Crawl-delay work?
Google ignores it. Bing and Yandex read it. Crawl rate for Google is managed through server responses and Search Console. The linter marks Crawl-delay as informational for that reason.
Can I test a live site's robots.txt here?
Open https://your-site/robots.txt in a new tab, copy the contents, and paste them into the editor. The tool intentionally does not fetch remote files.