Anthropic's crawlers: ClaudeBot, Claude-SearchBot and Claude-User explained

Anthropic runs three documented bots, and they do three different jobs. One gathers content that may train future Claude models, one builds the index behind Claude's search answers, and one fetches a page when a person using Claude asks for it.¹ Each has its own robots.txt token, so you can keep your content out of training and still be found and cited by Claude. Most sites that block Anthropic today block all three, or block old tokens that no longer do anything on their own.

This post covers Anthropic specifically. How AI crawlers fit together across operators is in the AI crawlers guide, and the parsing rules that apply to every bot are in how to set up robots.txt for AI crawlers.

In business terms, ClaudeBot is the one with no route back to your site. Claude-SearchBot and Claude-User are how your pages get into Claude's answers, and Claude's answers are where the links are. Blocking ClaudeBot and leaving the other two alone is the configuration I would start from for most businesses.

Anthropic does not publish full user agent strings for any of the three in its help article, only the tokens. Match on the token in your logs and your firewall rules. Under RFC 9309 a crawler matches its product token against User-agent lines without regard to case, and the token should appear within the crawler's full identification string,² so ClaudeBot in your robots.txt is all you need.

Two further limits apply to ClaudeBot. Anthropic's privacy policy for people who are not Claude users says it collects content from pages that are freely available to anyone browsing the internet and does not access password-protected pages.³ The crawler help article adds that Anthropic's bots respect anti-circumvention technology and will not try to get past CAPTCHAs.¹ Content behind a login is outside ClaudeBot's reach by Anthropic's own rules, though a login remains the only control that actually enforces that.

Many robots.txt files still carry User-agent: anthropic-ai and User-agent: Claude-Web, often copied from older blocklists. In July 2024, 404 Media reported that many sites were blocking those two names while leaving ClaudeBot, the crawler actually in use, unblocked. Anthropic then told 404 Media that the anthropic-ai and Claude-Web user agents were no longer in use, and that it had configured ClaudeBot to respect any robots.txt directives previously set for them.⁴

That statement came from a spokesperson in a news report, and Anthropic's current help article lists only the three tokens above.¹ I would not rely on the old names carrying over indefinitely. Leaving the old lines in place does no harm, but add an explicit ClaudeBot line so the file says what you mean. And treat any request that identifies itself as anthropic-ai or Claude-Web today with suspicion, since Anthropic says it no longer sends them.

Anthropic supports the Crawl-delay directive, and its own example sets Crawl-delay: 1 in a ClaudeBot group.¹ Crawl-delay is not part of the robots.txt standard. RFC 9309 lets crawlers interpret records it does not define,² and Google says it does not support crawl-delay,⁵ so adding it to a ClaudeBot group slows Anthropic without changing anything for Googlebot. If ClaudeBot's crawl rate is a server load problem, this is the gentler option before a full block.

Rules apply per subdomain. Anthropic asks you to add the block on every subdomain you want to opt out,¹ because each host serves its own robots.txt. A shop on shop.example.com and a help centre on help.example.com each need their own file with the Anthropic lines in it.

Anthropic advises against blocking the IP addresses its bots use. It says doing so may not work correctly or reliably as an opt-out, because it prevents the bots from reading your robots.txt.¹ A bot that cannot read the file does not know you want it to stop, so it may keep trying. I would use robots.txt for all three Anthropic bots and keep IP rules for operators whose user-triggered fetchers do not follow the file.

These follow the same logic as the configurations in how to set up robots.txt for AI crawlers. Remember the rule that catches people out: once a bot is named in a group, it follows only that group and ignores your User-agent: * rules.⁵ Any shared Disallow lines have to be repeated in a named group.

# Goal 1: stay in Claude's search, stay out of training (my default)
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /checkout/
Disallow: /account/
# Claude-SearchBot and Claude-User are not named, so they follow the * group.


# Goal 2: keep Anthropic out entirely
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /


# Goal 3: allow training, but slow ClaudeBot down
User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /checkout/
Disallow: /account/
# ClaudeBot is named, so the shared disallows are repeated here.

Goal 2 is more effective with Anthropic than with some other operators. OpenAI and Perplexity both say their user-triggered fetchers may not follow robots.txt, which you can compare in the OpenAI crawlers post and the Perplexity crawlers post. Anthropic says its bots honour robots.txt as a group,¹ and Claude's web fetch tool for developers lists robots.txt among the reasons it will refuse a URL.⁶ So a Claude-User block should hold when a person pastes your URL into Claude.

Bot management is the usual reason a site that welcomes Claude-SearchBot never sees it. That problem, and how to fix it without opening the door to everything, is covered in when your CDN blocks AI crawlers.

When a question benefits from current information, Claude calls a search tool and grounds its answer in live web content, with citations and source links in the response.⁸ The same help page says image search is powered by Bing. It does not name a provider for general web search.

Anthropic's subprocessor list does. It names two subprocessors for web search: Brave Search, covering all products, and TurboPuffer, covering all products except Claude for Government.⁹ Anthropic does not explain how the two divide the work, or how Claude-SearchBot's own index fits alongside them. What it does mean in practice is that being findable in Brave Search is worth checking if Claude visibility matters to you, alongside allowing Claude-SearchBot.

The developer documentation adds two useful details. Citations are always switched on for Claude's web search tool, and each one carries the source URL and title.¹⁰ The web fetch tool does not support pages rendered with JavaScript,⁶ so content that only appears after scripts run may be invisible when Claude fetches a page directly. Checking what crawlers see on a JavaScript site shows how to test for that.

Anthropic does not document a tracking parameter on the links Claude shows, so there is no equivalent of the utm_source value some other assistants add. In analytics I would look for referrals from claude.ai and claude.com, and treat the total as a floor, because some visits arrive with no referrer at all. How to measure visibility in AI search covers what to track beyond referrals.

Cloudflare measures how many HTML pages each AI platform's user agents request for every visit that platform refers back, using the Referer header across its network. In July 2025, Anthropic's ratio was 38,065.7 crawls per referral, down from 286,930.1 in January, against 1,091.4 for OpenAI and 194.8 for Perplexity in the same month.¹² Cloudflare's 2025 Year in Review, covering 1 January to 2 December 2025, found Anthropic had the highest ratio of the leading AI and search platforms. It peaked at around 500,000:1 early in the year, when Cloudflare thinks sparse referral traffic made the figure erratic, and then held between about 25,000:1 and 100,000:1.¹³

The ratio counts all of a platform's crawling, so a training crawler as active as ClaudeBot pushes it up. In July 2025 ClaudeBot made up 23.3% of the AI-only bot traffic Cloudflare saw.¹² The ratio does not tell you what Claude-SearchBot alone returns. It does show that most of Anthropic's crawling comes from the bot that sends nobody back, which is why blocking ClaudeBot costs a typical business very little.

For most SaaS, ecommerce, B2B and local businesses, I would disallow ClaudeBot, leave Claude-SearchBot and Claude-User unnamed so they follow your normal rules, and repeat the block on every subdomain. Check that your CDN lets the two search-side bots through, and verify a sample of requests against bots.json once a quarter. That keeps you citable in Claude without contributing to training.

If you are happy to be trained on, leave Anthropic out of robots.txt entirely, or add a Crawl-delay if ClaudeBot is heavy on your servers. If you need to keep Anthropic out altogether, name all three tokens; with Anthropic, robots.txt is enough to do it. Deleting the anthropic-ai and Claude-Web lines is not worth a change request of its own, but tidy them up the next time the file is edited.