Prompt
7 minute readrobots.txt Audit Prompt: Find Crawl Rule Mistakes
A reusable robots.txt audit prompt that explains what each group allows, checks the file against your crawl goals and flags common mistakes, with a real test run.
This robots.txt audit prompt reads your robots.txt file and explains, in plain language, what each group of rules allows and blocks. It flags common mistakes, such as blocking CSS files, using unsupported directives or locking out a crawler by accident. Then you confirm each finding with a tester before you change the live file.
The page explains each variable, gives the full prompt and shows one real run on an invented file. It also lists the checks that catch wrong conclusions. A model is not a robots.txt parser, so the final word always belongs to a tester that follows the standard.
What this robots.txt audit prompt does
The prompt summarises each user-agent group, lists the paths it blocks and compares the file with the goals you describe. For example, if you want AI search crawlers to reach your guides but keep training crawlers out, it checks whether the rules say so.
It relies on the rules in RFC 9309 and Google's robots.txt documentation: the most specific user-agent group wins, the longest matching path decides, and robots.txt controls crawling, not indexing. It labels anything it cannot settle as "test this".
When to use it
Use it before you edit robots.txt, after a site migration, or when a crawler seems to miss pages. It also helps you explain an inherited file to a colleague.
- A long file built up over years by different people.
- A plan to change how AI crawlers can access the site.
- Search Console reports pages "blocked by robots.txt" that should be crawlable.
However, do not deploy a change based only on the model's reading. Test every rule you rely on with a parser that follows the standard, such as our tester.
Inputs and variables
Replace every bracketed variable before you run the prompt. The goals matter most, because the same rule can be right for one site and wrong for another.
- [ROBOTS_TXT]: the full current file, copied from https://yoursite.com/robots.txt.
- [GOALS]: what should and should not be crawled, for example "allow all search crawlers; block GPTBot and CCBot; keep /cart/ and /account/ out".
- [KEY_URLS]: five to ten important URLs, including CSS or JavaScript files your pages need.
robots.txt files are public, so pasting one into an AI tool shares nothing new. Leave out any private notes you keep in comments.
Reusable prompt — version 1.0
Copy the prompt below as it is. It asks the model to separate certain findings from ones you must test.
You are helping a technical SEO specialist audit a robots.txt file.
<task>
Explain what <robots> allows and blocks, compare it with <goals>, and list problems.
</task>
<rules>
1. Apply RFC 9309: a crawler obeys only the most specific user-agent group that names it, otherwise the * group; within a group, the longest matching path wins, and Allow wins a tie.
2. Treat robots.txt as crawl control only. Never say it removes pages from search results.
3. Flag unsupported directives (such as noindex or crawl-delay for Google), syntax errors, blocked CSS or JavaScript, and rules that contradict <goals>.
4. For each URL in <urls>, say whether Googlebot and each crawler named in <goals> may fetch it, and which line decides.
5. If a case is ambiguous, label it TEST THIS instead of guessing.
6. Do not invent crawler names or user-agent tokens.
</rules>
<output_format>
1. Summary of each group in one line.
2. Table: URL | crawler | allowed? | deciding line
3. Problems: issue | line | why it matters | suggested fix
4. TEST THIS list
</output_format>
<goals>[GOALS]</goals>
<urls>[KEY_URLS]</urls>
<robots>
[ROBOTS_TXT]
</robots>Rule 1 spells out the matching logic, because models often apply the * group and a named group together, which is wrong.
How to use it
Any capable chat model can run this robots.txt audit prompt. Then confirm the important answers in a tester.
- Copy your live robots.txt and write down your crawl goals.
- List the key URLs, including one CSS and one JavaScript file.
- Fill in the variables, paste the prompt into a new chat and run it.
- Check every "allowed?" answer you rely on in our robots.txt tester.
- Edit the file, test it again and only then publish it.
Example input and output
We ran version 1.0 once on Claude Opus 5.5 on November 15, 2026. The file is invented for this example; the output is trimmed, and trims are marked with "…".
GOALS: allow search crawlers everywhere except /cart/; block GPTBot
KEY_URLS: /guides/fix-a-flat-tire, /assets/site.css, /cart/
ROBOTS_TXT:
User-agent: *
Disallow: /cart/
Disallow: /assets/
Noindex: /old/
User-agent: GPTBot
Disallow: /cart/1. Summary: * blocks /cart/ and /assets/. GPTBot has its own group,
so it ignores the * rules and is blocked only from /cart/.
2. | /guides/fix-a-flat-tire | GPTBot | yes | no matching rule in the GPTBot group |
| /assets/site.css | Googlebot | no | Disallow: /assets/ (* group) |
…
3. Problems:
- GPTBot is allowed almost everywhere, which contradicts the goal.
Fix: change its group to "Disallow: /".
- /assets/ is blocked, so Googlebot cannot fetch the CSS it needs
to render pages. Fix: remove that line or allow /assets/*.css.
- "Noindex" is not supported by Google in robots.txt; remove it and
use a meta robots tag on the pages instead.
4. TEST THIS: none.In this run, the model caught the classic mistake: the named GPTBot group overrides the * group. That single run is an illustration, not a benchmark.
How to check the output
Verify the findings before you change the live file.
- Matching: each "allowed?" answer matches what a standards-based tester returns.
- Tokens: every crawler name is a real, documented user-agent token.
- Scope: no finding claims that robots.txt removes pages from search.
- Resources: CSS, JavaScript and images that pages need stay crawlable.
- Goals: the suggested file does what your goals say, and nothing more.
Troubleshooting
These fixes address the bad outputs we see most often.
- The model merges groups: repeat rule 1 and ask it to name the group it used for each answer.
- It invents user agents: list the exact tokens you care about in [GOALS].
- It suggests noindex in robots.txt: point it to rule 2 and ask for meta robots instead.
- The file is huge: audit one group at a time.
Limitations
A language model can misapply wildcards, end-of-path anchors and percent-encoding. Therefore, treat its table as a draft and rely on a parser for the final answer. Some crawlers also ignore robots.txt entirely, which no file can fix.
Outputs vary by model, version and input. The audit explains crawl rules; it does not predict rankings, indexing or AI citations.
Version history
Each version carries a date, so you can tell which prompt produced an older audit.
- 1.0: November 15, 2026. First version, tested once on Claude Opus 5.5.
Frequently asked questions
Quick answers to the questions readers ask most about this topic.
Which AI models does this robots.txt audit prompt work with?
Any capable chat model that follows structured instructions, such as current models from OpenAI, Anthropic or Google. We tested it once on Claude Opus 5.5. Confirm the verdicts you rely on in a standards-based tester.
Can robots.txt remove a page from Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it; use a noindex meta tag on a crawlable page to keep it out.
Why does my named crawler ignore the rules for all user agents?
Under RFC 9309, a crawler follows only the most specific group that names it. Once a group names GPTBot, for example, GPTBot ignores the rules in the * group, so repeat any rule you need in its own group.
Is it safe to paste robots.txt into an AI tool?
Yes in general, because robots.txt is a public file. Remove any private notes you keep in comments before you paste it.
Sources
These pages support the robots.txt rules and the prompting approach above.
Keep reading