AI crawlers: who is fetching your pages, what each one is for, and what to let in

Open the access logs of almost any website and a large share of the requests now come from bots run by AI companies. Some of them are building training sets for the next model. Some are building a search index that ChatGPT, Claude or Perplexity will cite from. Some arrive seconds after a person typed a question into an assistant, and leave again with one page. They identify themselves with names like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot, and each operator documents its own set.

For SEO, this matters in two directions. Let the wrong bot in and you give away content for little return. Shut the wrong bot out and you vanish from an assistant that your customers use to choose a supplier. This guide explains what each kind of crawler does, who runs them, what the data says about how much they take and give back, and the controls you have. Each operator has its own post with the full user agent strings, IP lists and robots.txt lines, linked from the directory below.

How assistants turn crawled pages into answers is covered in the AI search guide, and the wider mechanics of crawling, rendering and indexing in the technical SEO guide. This page stays with the crawlers themselves.

The single most useful thing to know about AI crawlers is that the big operators split them by purpose and let you control each one separately. OpenAI, for example, says each of its settings is independent of the others, so a site can allow OAI-SearchBot while disallowing GPTBot.¹ Anthropic splits its bots the same way into ClaudeBot, Claude-SearchBot and Claude-User.² Once you know which job a bot does, the decision about it usually becomes obvious.

The line between the last three has blurred, and the operators describe them in slightly different terms. The test I use is simple: does a person stand behind the request, and does the product link back to you? A search crawler and a user-triggered fetcher both feed answers that can cite and link your page. A training crawler feeds a model that may know your content later but will not send anyone to it.

Every major operator now publishes a page listing its user agents, what each one is for and, in most cases, the IP ranges it crawls from. The table summarises them as documented in October 2026. Each operator has a dedicated post with the full user agent strings, verification steps and recommended robots.txt lines.

Two gaps in that table catch people out. ByteDance's Bytespider is common in logs but has no public documentation I could find, so there is nothing official to say about its purpose or whether it follows robots.txt. And Microsoft and Apple control AI use mostly through meta tags, which robots.txt cannot do for them: a page can stay in Bing and in Apple search while being kept out of Copilot or out of Siri's generated answers.

AI crawling is no longer a rounding error. In its review of 2025, Cloudflare reported that AI crawlers made up 20% of the verified bot traffic on its network, against 40% for search engine crawlers. GPTBot alone accounted for about 7.5% of verified bot traffic and Googlebot for over 28%. Across HTML requests, AI bots averaged 4.2% of all traffic during the year, slightly less than Googlebot's 4.5% on its own.¹⁴

The more useful figure for a business is what each operator sends back. Cloudflare measures this as a crawl-to-refer ratio: the HTML pages a platform's bots requested, divided by the visits that arrived with a referrer from that platform. Over 2025 it reported Anthropic's ratio reaching up to 500,000 to one early in the year and then settling between about 25,000 and 100,000 to one. OpenAI's peaked at 3,700 to one in March. Perplexity's stayed below 200 to one from September. Microsoft's peaks ran between about 50 and 70 to one, and Google's moved between about 3 and 30 to one.¹⁴

Purpose explains most of the gap. Across a fixed set of customers in the 12 months to July 2025, Cloudflare classed 80% of AI crawling as training, 18% as search and 2% as user actions.¹⁵ Its 2025 review found training crawling running at seven to eight times search crawling at its peak, while user-action crawling grew more than fifteen-fold over the year from a low base.¹⁴ That last trend is the one I would watch. User-triggered fetches happen because a real person asked about something, and they are growing faster than either of the other two.

Cloudflare's figures cover traffic on its own network, which skews towards the sites that choose Cloudflare, and the ratio counts only visits that carry a referrer. Some assistant traffic arrives without one, so the real referral numbers are somewhat higher. The direction is still clear: training crawlers take a great deal and return very little, and search crawlers and fetchers are the ones with a route back to your site.

More sites are saying no to training. A large audit by the Data Provenance Initiative, led by Shayne Longpre and covering about 14,000 web domains behind common AI training sets, found that within a single year from 2023 to 2024, restrictions in robots.txt had fully closed off 5% or more of all tokens in the C4 dataset and 28% or more of its most actively maintained, critical sources.¹⁶ Cloudflare's 2025 review of the top 10,000 domains found that the user agents with the most whole-site disallows were AI crawlers such as GPTBot, ClaudeBot and CCBot, while Googlebot and Bingbot were mostly given partial disallows.¹⁴ Across the web as a whole, naming AI bots is still a minority practice: HTTP Archive's 2025 Web Almanac found GPTBot mentioned in 4.5% of desktop robots.txt files and ClaudeBot in 3.6%.¹⁷

None of that tells you what your own site should do. The decision depends on how the business makes money from its content. I work through it with three questions.

For most SaaS, ecommerce, B2B and local businesses my default is the first configuration in how to set up robots.txt for AI crawlers: be cited, but not trained on. That post has the worked files for all three goals and the parsing rules that catch people out, chief among them that a bot named in its own group stops reading your User-agent: * rules.

robots.txt is the main control, but it is one of several, and each works at a different layer.

Two pieces of standards work are worth following because they will change these controls. The IETF's AI Preferences working group is drafting a shared vocabulary for how content may be used by automated systems, with three categories so far: AI training, AI use (content used as input to a model, as in grounding) and search. A companion draft covers attaching those preferences to content through robots.txt and HTTP headers. The vocabulary draft was on its eighth revision in September 2026 and states that it does not yet reflect working group consensus.²² If it is adopted, a site could say "search yes, training no" once, in a form every operator reads, in place of listing each company's tokens.

The second is identity, covered in the next chapter.

A user agent string is just text, and scrapers routinely borrow the names of well-known crawlers. Before you draw conclusions from your logs, or write a firewall rule that lets a bot through, check that the request really came from the operator. There are three ways to do it, and the operator decides which ones are available.

• Published IP ranges. OpenAI, Anthropic, Perplexity, Microsoft, Apple and Common Crawl publish JSON files listing the IP ranges their bots use,^121123624 and Google publishes separate files for its common crawlers, special crawlers and user-triggered fetchers.²⁵ Match the request IP against the list for that user agent. The lists change, so fetch them on a schedule and do not hard-code them.
• Reverse DNS. Look up the hostname for the IP, check it belongs to the operator's domain, then look up that hostname and confirm it resolves to the same IP. Google documents this for its crawlers,²⁵ Microsoft for Bingbot, whose hostnames end in search.msn.com,²³ and Apple for Applebot, whose hostnames end in applebot.apple.com.⁶ Anthropic, Perplexity and Meta document no reverse DNS check, so for them the IP list is the test.
• Signed requests. Newer agents sign each request cryptographically under a protocol called Web Bot Auth, built on the HTTP Message Signatures standard, RFC 9421. The request carries a Signature-Agent header pointing at the operator's public keys, so a server or CDN can verify it without trusting the IP. The IETF moved the protocol to a working group draft in September 2026, with authors from Cloudflare and Google,²⁶ and Google already signs some requests from Google-Agent this way.³

In practice most site owners never do this by hand. CDNs such as Cloudflare keep a list of verified bots, checked by signatures, IP lists or reverse DNS,²⁷ and log tools such as Screaming Frog's Log File Analyser verify the main search engine bots automatically.²⁸ What matters is knowing which of your numbers are verified and which only count requests that claim a name.

Being allowed in is the first gate. The second is whether the crawler can read anything once it arrives. The providers do not document their rendering, so the best public evidence is a December 2024 study by Vercel and MERJ, based on monitoring nextjs.org and the Vercel network over several months. It found that none of the major AI crawlers it measured executed JavaScript, including OpenAI's crawlers, Anthropic's ClaudeBot, Meta's, ByteDance's and PerplexityBot. ChatGPT's and Claude's crawlers fetched JavaScript files without running them. The exceptions were Google's Gemini, which uses Googlebot's infrastructure, and Applebot, which renders with a browser-based crawler.²⁹

The same study found that about 34% of fetches by the ChatGPT and Claude crawlers hit 404 pages, against about 8% for Googlebot.²⁹ AI crawlers are less practised at choosing URLs, so redirects from old URLs and a clean internal link structure matter more for them, not less.

The practical rule follows. Anything you want an assistant to quote, such as prices, plan limits, specifications, opening hours and policies, should be in the HTML your server sends. Checking what crawlers see on a JavaScript site shows how to test it. The study is nearly two years old, so check your own logs before assuming it still holds for a given crawler.

Server logs are the only place you can see what every crawler requested and what your server sent back. Analytics tools that run in the browser miss crawlers entirely, because crawlers do not run the tracking script. The method is the same as for Googlebot, covered in the log analysis chapter of the technical SEO guide: filter by user agent, verify the IPs, then look at which URLs each bot requests and which status codes it gets.

# Requests per AI user agent in a combined-format access log
grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Meta-ExternalAgent|Meta-ExternalFetcher|Applebot|Amazonbot|CCBot|Bytespider" access.log | sort | uniq -c | sort -rn

# Status codes served to anything calling itself OAI-SearchBot (verify the IPs separately)
grep "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

Three patterns are worth looking for. A search crawler that only ever receives 403s, 429s or challenge pages is being blocked at the edge. A training crawler you have disallowed that keeps requesting pages other than robots.txt more than a day after the change is either not compliant or not genuine. And user-triggered fetches from ChatGPT-User, Claude-User or Perplexity-User are as close as logs come to a record of your pages being used in answers, because each one follows a person's question.¹²¹¹ They are not a complete record, since assistants also answer from pages fetched earlier.

The other half of the picture is the visits that come back. How to measure visibility in AI search covers referral traffic and what the assistants report.