Separate AI Crawlers, Search Bots and User Fetches

“Allow AI” is too broad to be a useful crawler policy. A provider may use different mechanisms for search discovery, model development and a user-requested page fetch. Decide what you want to permit, then verify the current controls for each documented agent.

Inventory observed access

Review server or CDN logs for relevant requests, recording timestamp, path, status, user-agent and available verification data. A user-agent string can be imitated, so do not treat its text alone as proof of identity.

Consult each provider's current official crawler documentation before editing access rules. Names, purposes and verification methods can change; a copied list from an old article is not a durable policy.

Write a purpose-based decision

Robots instructions are not authentication or a substitute for protecting private information. Keep sensitive content behind appropriate access controls regardless of crawler preferences.

Worked example: a broad block

Imagine a hypothetical CDN rule that blocks requests containing a provider's name. It could affect more than the specific activity the team intended to restrict. Before changing it, identify which documented agents and request types the rule actually matches.

Test on a limited scope where feasible and inspect response codes. A robots file can express one preference while a firewall independently blocks access, leaving the effective behavior different from the written policy.

Keep evidence for each rule

Store the official documentation URL, checked date, purpose, selected action and owner. Record exceptions for paths that should remain publicly available or restricted.

Do not infer that allowing a crawler guarantees citations, rankings or training inclusion. Access is one condition in a larger system whose selection behavior may not be visible.

Review after infrastructure changes

Recheck behavior after CDN migrations, security rule updates and major site launches. Compare expected access with logs and user-facing availability. Investigate unusual load separately from the general policy decision.

The useful deliverable is a small, current policy tied to observable requests and documented controls. It is more maintainable than a large unexplained blocklist copied from a social post.

Three mechanisms that are often confused

Access policy conversations go wrong when three separate controls are treated as one. Each answers a different question, and a decision made with one cannot be assumed to hold for the others.

A team that wants public pages discoverable through a provider's search product but not used for another documented purpose expresses that in robots.txt against the specific agents the provider documents for each purpose. A team that is seeing abusive load from something imitating a known agent handles that at the firewall, after verification shows the requests are not genuine. Mixing the two produces a firewall rule that silently overrides a robots preference, and a written policy that no longer describes what the site does.

A review routine for the access log

The log is the only ground truth. A repeatable review, run monthly or after any infrastructure change, keeps the policy connected to observed behavior:

1. Export requests for a fixed window, grouped by user-agent family and response status.
2. For each agent family that matters to the policy, verify a sample of requests using the provider's documented method. Note how many claimed requests failed verification.
3. Compare what each verified agent fetched with what the policy intends. A disallowed path being fetched by a verified agent is a finding; the same path fetched by an unverified imitator is a different finding.
4. Check that intended-public paths return success to verified agents, and that any challenge or block pages are not being served to them by mistake.
5. Record the counts, the checked documentation URLs with dates, and any rule changed as a result.

Handle the ambiguous cases explicitly

Some traffic does not fit the neat categories, and the policy should say what happens to it rather than leaving it to whoever is on call:

• A user-triggered fetch where a person asked an assistant to read a specific public page. Many sites treat this like a browser visit, since a person requested it. Decide, document, and make sure the rule you write does not also block the provider's search crawler.
• An agent the provider does not document. Treat it as unverified. Rate-limit if load is a problem; do not add it to a permanent blocklist on the strength of its name.
• A documented agent with a purpose you have not decided on. Default to the more conservative documented preference until the decision is made, and put the decision on someone's list with a date.
• Your own tools and monitors. Label them in the log so they never inflate a count.

The policy document that results is short: a table of documented agents, the purpose each serves, the decision, the control that implements it, the evidence URL with its checked date, and an owner. That is enough to answer, months later, why a rule exists and whether it still should.

Put this into practice

Copy the worksheet columns below into a spreadsheet and keep one row per item you check. The filled row is an illustrative example, not a reported customer result; replace it with your own verified records.

Use the following prompt only after supplying the records it requests:

Review this crawler policy against supplied current official documentation and observed logs. Return purpose, verified identity evidence, effective access, mismatch and proposed rule change. Do not infer identity from user-agent text alone.

Research context

Crawler reporting and product updates make it important to verify current agent purposes before changing access rules. The related Ahrefs starting points are Bot Analytics, free API access, and more (February & March 2026) and New in June ’26: Firehose, spoofed bot detection, and more. This guide’s checklist, examples and proposed workflow are independently written; they are not results of a SEOVision experiment.

Continue with the next task

• Training, Retrieval and Tools: Diagnose an AI Answer’s Source
• Structured Data for AI Search: What It Can—and Cannot—Do
• AI Search Optimization: A Practical Guide Beyond GEO Hype

Sources

• Bot Analytics, free API access, and more (February & March 2026) — Research starting point; not an endorsement of this original workflow
• New in June ’26: Firehose, spoofed bot detection, and more — Research starting point; not an endorsement of this original workflow
• Google: AI features and your website — Primary documentation for the stated platform behavior