Prompt
7 minute readLog File Analysis Prompt: See How Crawlers Visit Your Site
A reusable log file analysis prompt that summarises crawler requests by bot, section and status code, quotes the lines behind each issue and flags unverified Googlebot claims.
This log file analysis prompt summarises a sample of server access logs to show how search and AI crawlers actually visit your site. It counts requests by crawler, status code and site section, and it points out wasted crawling, errors and pages crawlers never reach. Every figure comes with the rows behind it, so you can check it.
The page explains each variable, gives the full prompt and shows one real run on an invented log sample. It also lists the checks that catch fake crawlers and arithmetic mistakes. A user-agent string can be faked, so the prompt treats crawler names as claims until you verify them.
What this log file analysis prompt does
The prompt groups log lines by the crawler they claim to be, such as Googlebot, Bingbot, GPTBot or ClaudeBot. Then it reports which sections each one requests, how many requests return errors or redirects, and which important URLs receive no crawler visits at all.
It also marks requests that claim to be Googlebot but come from addresses it cannot verify. Google explains how to confirm Googlebot with a reverse DNS lookup, and the prompt reminds you to do that before trusting the numbers.
When to use it
Use it when you need a quick read of crawler behaviour and have no log analysis tool. It suits samples of a few thousand lines.
- Search Console shows crawl problems you want to see in raw requests.
- You want to know how often AI crawlers fetch your pages.
- A migration or new section may have created crawl traps or errors.
However, for months of logs or millions of lines, use a dedicated log analyser or a script. Give the prompt that tool's summary, not the raw files.
Inputs and variables
Replace every bracketed variable before you run the prompt. A clean sample gives far better results than a random slice.
- [SITE_SECTIONS]: the main URL patterns, for example "/guides/, /products/, /blog/".
- [KEY_URLS]: ten to twenty important pages you expect crawlers to visit.
- [LOG_SAMPLE]: access log lines in combined log format, filtered to known crawler user agents.
- [DATE_RANGE]: the exact period the sample covers.
Access logs contain IP addresses, which can be personal data. Remove or mask visitor IPs before you paste logs into a public AI tool, and keep only crawler lines.
Reusable prompt — version 1.0
Copy the prompt below as it is. It separates what the log shows from what you must verify.
You are helping a technical SEO specialist analyse server access logs.
<task>
Summarise crawler activity in <logs> and list the issues worth fixing.
</task>
<context>
Date range: [DATE_RANGE]
Sections: [SITE_SECTIONS]
Key URLs: [KEY_URLS]
</context>
<rules>
1. Group requests by the crawler named in the user agent; call these "claimed" crawlers.
2. Count requests per crawler by section and by status code (2xx, 3xx, 4xx, 5xx).
3. Quote example log lines for every issue you report.
4. List key URLs with no crawler requests in the sample.
5. Never state that a request came from Google or another company; say it must be verified (for example by reverse DNS).
6. Use only the lines given. Never estimate traffic, rankings or crawl budget figures.
</rules>
<output_format>
1. Table: claimed crawler | requests | 2xx | 3xx | 4xx | 5xx | top section
2. Issues: issue | example lines | suggested check or fix
3. Key URLs with no crawler visits
4. What to verify before acting
</output_format>
<logs>
[LOG_SAMPLE]
</logs>Rule 5 keeps the report honest. Anyone can send a request that says "Googlebot", so the counts describe claims until you verify the addresses.
How to use it
Any capable chat model can run this log file analysis prompt. Filter and clean the sample first so it fits in one conversation.
- Export access logs for a recent period from your host or CDN.
- Keep only lines from known crawler user agents and mask visitor IPs.
- Fill in the variables, paste the prompt into a new chat and run it.
- Recount the main figures with a spreadsheet or command-line tool.
- Verify suspicious Googlebot requests with a reverse DNS lookup before acting.
Example input and output
We ran version 1.0 once on Claude Opus 5.5 on November 15, 2026. The 12-line log sample is invented for this example; the output is trimmed, and trims are marked with "…".
DATE_RANGE: November 1 to November 7, 2026
SECTIONS: /guides/, /products/, /search
KEY_URLS: /guides/fix-a-flat-tire, /products/steel-bottle
LOG_SAMPLE:
66.249.66.1 - - [03/Nov/2026:10:01:12] "GET /guides/fix-a-flat-tire HTTP/1.1" 200 "Googlebot/2.1"
66.249.66.1 - - [03/Nov/2026:10:02:40] "GET /search?q=lube&page=7 HTTP/1.1" 200 "Googlebot/2.1"
203.0.113.9 - - [04/Nov/2026:02:11:05] "GET /products/old-bottle HTTP/1.1" 404 "Googlebot/2.1"
20.15.240.1 - - [05/Nov/2026:08:30:00] "GET /guides/fix-a-flat-tire HTTP/1.1" 200 "GPTBot/1.1"
…1. | claimed crawler | requests | 2xx | 3xx | 4xx | 5xx | top section |
| Googlebot | 8 | 6 | 0 | 2 | 0 | /search |
| GPTBot | 3 | 3 | 0 | 0 | 0 | /guides/ |
…
2. Issues:
- Crawling of internal search pages: "GET /search?q=lube&page=7".
Check whether /search should be blocked in robots.txt.
- 404 for a retired product: "GET /products/old-bottle". Redirect it
to the current product if one exists.
3. No crawler visits: /products/steel-bottle.
4. Verify: 203.0.113.9 claims to be Googlebot; confirm with reverse DNS.In this run, the model flagged the unverified "Googlebot" address and quoted the lines behind each issue. That single run is an illustration, not a benchmark; always recount the figures.
How to check the output
Recheck the summary before you change anything on the site.
- Counts: the totals match a recount in a spreadsheet or with a command-line tool.
- Verification: suspicious Googlebot or Bingbot addresses pass a reverse DNS check.
- Quotes: every issue cites real lines from your sample.
- Privacy: no visitor IPs or personal data went into the prompt.
- Scope: the report makes no claims about rankings or traffic.
Troubleshooting
These fixes address the bad outputs we see most often.
- Counts do not add up: paste fewer lines and ask it to show the total per status.
- It trusts every Googlebot line: repeat rule 5 at the end of the prompt.
- The sample is too large: filter to one crawler or one section per run.
- It guesses causes: ask only for observations and checks, as rule 6 says.
Limitations
A sample shows only the period and lines you include, so rare problems can hide outside it. Therefore, treat the summary as a starting point and confirm patterns over a longer period. The model also cannot run DNS lookups or see your server configuration.
Outputs vary by model, version and input. Log analysis shows how crawlers behave; it does not predict indexing, rankings or AI citations.
Version history
Each version carries a date, so you can tell which prompt produced an older report.
- 1.0: November 15, 2026. First version, tested once on Claude Opus 5.5.
Frequently asked questions
Quick answers to the questions readers ask most about this topic.
Which AI models does this log file analysis prompt work with?
Any capable chat model that follows structured instructions and handles long input, such as current models from OpenAI, Anthropic or Google. We tested it once on Claude Opus 5.5.
Can I trust a log line that says Googlebot?
Not on its own. User-agent strings can be faked. Google documents how to verify Googlebot with a reverse DNS lookup on the requesting IP address.
Is it safe to paste server logs into an AI tool?
Only after cleaning them. Access logs contain IP addresses, which can be personal data. Keep only crawler lines and mask visitor IPs first.
How many log lines can I analyse at once?
A few thousand cleaned lines is a practical limit. For longer periods, summarise the logs with a script or log tool and give the prompt the summary.
Sources
These pages support the crawler guidance and the prompting approach above.
Keep reading