The other AI crawlers in your logs: CCBot, Amazonbot, Bytespider and more

Once you start filtering server logs for AI crawlers, the familiar names from OpenAI, Anthropic and Google are only part of what turns up. CCBot, Amazonbot and Bytespider often appear alongside them, and smaller assistants such as DuckDuckGo, Mistral and You.com run their own fetchers. Each one does a different job, and blocking the wrong one costs you visibility for nothing.

This post covers the operators whose documentation I could read in October 2026, plus two where documentation is missing or says there is no crawler. How the crawler types fit together, and where each feeds, is in the AI crawlers guide. For the large operators, see OpenAI's crawlers, Anthropic's crawlers and Google's AI crawlers.

Two operators people often ask about are missing from the table. Cohere states that it does not currently use bots to crawl the web for training,⁷ and I could not find any crawler documentation published by xAI for Grok. Both are covered further down.

CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/). Common Crawl says it checks robots.txt before fetching, obeys Crawl-delay, and will stop crawling a site that disallows it.² It runs on dedicated IPv4 and IPv6 ranges published as JSON, and IPv4 addresses reverse-resolve to hostnames under crawl.commoncrawl.org, so a forward-confirmed reverse DNS check works. IPv6 addresses do not yet have reverse DNS. Common Crawl also warns that other crawlers falsely use the CCBot name.¹

The crawler itself is light on most sites. In Cloudflare's comparison of AI crawler traffic across a fixed set of customers, CCBot held 0.1% of AI-only bot traffic in both July 2024 and July 2025.⁸ What makes it important is where the data goes. Common Crawl is a nonprofit that publishes its archive for anyone to use, and its own pages describe the purpose as research and analysis.²

The Mozilla Foundation's February 2024 report, written by Stefan Baack, describes an archive of more than 9.5 petabytes going back to 2008 and calls it one of the most important sources of training data for generative AI since GPT-3. It reviewed 47 text-generation LLMs published between 2019 and October 2023 and found that at least 64% of them (30 models) used at least one filtered version of Common Crawl for pre-training. It also notes that Common Crawl made up more than 80% of the tokens in GPT-3.⁹ AI builders rarely use the raw archive. They use filtered versions of it, so a page in Common Crawl can reach many models through datasets you will never see.

Site owners have noticed. Among 3,816 robots.txt files Cloudflare found on the top 10,000 domains, GPTBot was the most disallowed bot in 2025, followed by CCBot and Google-Extended.¹⁰ Common Crawl also keeps a public list of the legal opt-out requests it receives, renamed from the Opt-Out Registry to the Opt-Out Ledger, and says it honours legitimate requests.¹¹

Two limits are worth knowing. A robots.txt block stops future crawls, and none of the Common Crawl pages I read say that it removes your pages from crawls already published. And because anyone can download the archive, blocking CCBot is the only control you have over that route into training data. If staying out of training is a goal, I would block it.

Amazon documents three user agents, and each setting is independent. All three use the same string pattern with a different token, for example Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36.³

• Amazonbot improves Amazon's products and services, and Amazon says it may be used to train Amazon AI models. Amazon also says that allowing it may make you eligible for benefits with Amazon Content Partners.³
• Amzn-SearchBot makes your content eligible for search experiences in Amazon products, with Alexa given as the example. It does not crawl for generative AI training. If your robots.txt does not name it but allows other search bots, it follows the rules given to those bots.³
• Amzn-User fetches live information when a user's action needs it, such as an Alexa question that needs current facts. Because a person starts the request, Amazon says it may not follow all robots.txt directives. It is not used for training.³

All three follow RFC 9309, check robots.txt separately on each host, and may use a cached copy up to 30 days old. Amazon says changes can take about 24 hours to show in its systems, and none of the three support Crawl-delay. They respect rel=nofollow on links and the robots meta tags noindex, none and noarchive. Amazon defines noarchive as an instruction not to use the page for model training, which gives you a page-level training opt-out on top of robots.txt.³

Amazon publishes a separate IP list for each user agent. It does not document reverse DNS verification, so match requests against those lists. Amazonbot's share of AI and search crawler traffic in Cloudflare's fixed customer set fell from 10.2% in July 2024 to 5.9% in July 2025.⁸

Amazon's page names Alexa as the place your content can appear and does not name other products. If you want to stay out of Amazon's training but remain eligible for Alexa answers, block Amazonbot and leave Amzn-SearchBot unnamed.

Bytespider is the hardest crawler in this post to write about accurately. I searched for a ByteDance page documenting it, with its user agent string, purpose, robots.txt behaviour or IP ranges, and found none. Every description I found came from third-party bot directories, which disagree with each other on its purpose and on whether it obeys robots.txt. I have left those claims out.

Cloudflare's network data is the most reliable public evidence. In July 2024 Cloudflare reported that Bytespider led AI crawlers by number of requests, had accessed 40.40% of the websites Cloudflare protects, and was the crawler its customers blocked most often. It also found that its customers' robots.txt files often named GPTBot and CCBot but did not specifically disallow Bytespider.¹² A year later it had fallen away. Comparing May 2024 with May 2025, Cloudflare found Bytespider's request volume down 85%,¹⁰ and its share of AI and search crawler traffic fell from 14.1% in July 2024 to 2.4% in July 2025.⁸

With no operator documentation there is nothing to tell you what blocking it costs, and no IP list to verify it against. I would add Bytespider to the training group in robots.txt, then check the logs. If requests carrying the token keep arriving after a few days and you want them stopped, a firewall or CDN rule on the user agent is the enforceable option. A user agent string is easy to fake, so treat traffic using the name as unverified either way.

Mistral splits its crawling across three tokens, which makes it one of the easier operators to configure. Each uses the pattern Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots) with its own token.⁴

• MistralAI-User handles user actions in Vibe, Mistral's assistant. When a user asks a question, Vibe may visit a page to answer and link to it. Mistral says it is not used to crawl the web automatically or to crawl content for training.⁴
• MistralAI-Index crawls for indexing only, to power Mistral search inside Vibe. Mistral says the content is not used for generative AI training of any kind.⁴
• MistralAI-Training builds datasets for training Mistral models and is not used for search or live answers. Mistral says webmasters can disallow it in robots.txt.⁴

Mistral publishes IP lists for the User and Index agents but not for Training. The page does not say in so many words whether MistralAI-User and MistralAI-Index follow robots.txt, though it describes the tokens as robots.txt controls. If you want to be found in Vibe without training Mistral's models, block MistralAI-Training only.

DuckAssistBot identifies itself as DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html). It fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources, and DuckDuckGo says the data is not used in any way to train AI models.⁵

A robots.txt disallow for DuckAssistBot takes effect 72 hours after you update the file. DuckDuckGo says opting out does not affect your organic search rankings or whether your site appears in its search results. The IP addresses are published on the help page and as JSON.⁵ Because the bot only serves cited answers, I would leave it allowed for almost every business.

YouBot is the crawler behind the You.com search engine. Its user agent contains compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod within a Chrome-style string. You.com says it respects robots.txt, including rules for its own user agent and Crawl-delay, and caches robots.txt for 30 minutes. It slows down when it receives a 429 response and pauses on a 503.⁶

You.com offers three ways to verify it: Cloudflare's Web Bot Auth signatures, with public keys at you.com/.well-known/http-message-signatures-directory, reverse DNS to hostnames under search.you.com, and the IP range 68.67.112.0/24.⁶ The page does not say whether crawled content is used for AI training, so I cannot tell you either way.

Cohere's documentation says it does not use bots or user agents to crawl or scrape web content to train generative AI models at this time, and that it would list any future bots on the same page. It requires crawlers to be designed to respect robots.txt. The Coherebot token in its robots.txt example is an illustration of the format, with no crawler of that name listed. Cohere also warns that blocking by IP address may not give a lasting opt-out, because it stops a bot reading your robots.txt.⁷

I could not find any page from xAI documenting crawlers for Grok. Third-party directories list several possible tokens, but none link to an xAI source, so I am not repeating them here. If a token claiming to be xAI's turns up in your logs, treat it as unknown and use the checks below.

For most businesses the goal is the same as in how to set up robots.txt for AI crawlers: stay out of training, stay findable in search and answers. This file adds the training crawlers from this post to one group and leaves everything else on the default rules.

# Training and dataset crawlers from this post
User-agent: CCBot
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
Disallow: /

# Everyone else, including Amzn-SearchBot, MistralAI-Index,
# MistralAI-User, DuckAssistBot and YouBot
User-agent: *
Disallow: /checkout/
Disallow: /account/

Keep these lines in the same group as the training tokens for OpenAI, Anthropic, Google, Apple and Meta if you already have one. Leave the search and answer crawlers unnamed. A crawler that finds a group naming it follows that group, and obeys the * group only when no group matches,¹³ so naming a bot in an Allow: / group would drop your shared disallows for it.

Blocking Amazonbot covers more than training, since Amazon also uses it to improve its products, and it may affect eligibility for Amazon Content Partners.³ A business that sells through Amazon or works with it as a content partner should check before adding that line.

New tokens keep appearing in logs. Before deciding anything, find out whether the operator documents the bot, and whether the request really came from them.

For a typical business that wants to be cited in AI answers but not used as training data, I would block CCBot, MistralAI-Training and Bytespider, and block Amazonbot unless the business has a commercial reason to keep Amazon happy. I would leave Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-User, DuckAssistBot and YouBot allowed. Each of those serves search or answers, and none is documented as a training crawler.

Then I would check that the file is working. Filter the logs for each token, verify the IPs where a list exists, and confirm after a few days that the blocked crawlers request only robots.txt. Amazon can take up to 30 days to pick up a change because of its cache,³ so give it time before deciding a block has failed.