How to set up robots.txt for AI crawlers

Most robots.txt files written for AI crawlers are copied from a list somewhere, and many of them do something the owner did not intend. A SaaS company blocks every bot with “AI” in its description and disappears from ChatGPT search. A publisher blocks Google-Extended and expects to leave AI Overviews, which that token does not control. A retailer adds an allow rule for one search bot and accidentally opens its checkout and account pages to it. The fix is to start from the business goal, use the tokens each provider documents, and check the result in your server logs.

Why training and search crawlers are separate, and where each one feeds, is covered in the AI search guide. Everything below assumes that split and gets into the file itself.

These are the tokens I would consider in October 2026, checked against each operator's documentation. Group them by effect, because that is how you will write the file.

Naming these tokens has become more common, though it is still a minority practice. HTTP Archive's 2025 Web Almanac found GPTBot named in 4.5% of the desktop robots.txt files it analysed, up from 2.9% in 2024, and ClaudeBot in 3.6%, up from 1.9%.¹ Those figures count mentions, so they include files that allow a bot as well as files that block it.

CCBot needs a judgement call. Common Crawl describes itself as an open repository for research, and its archive is open to anyone,⁸ so you cannot know every use made of it. If your goal is to stay out of training data generally, I would block it along with the named training crawlers.

Most broken AI configurations come from how robots.txt is parsed. The standard is RFC 9309, and Google publishes its own reading of it, which helps where the RFC leaves room.

This is the setup I would recommend for most SaaS, ecommerce, B2B and local businesses. You want assistants to find, quote and link to your pages, and you would prefer future models not to train on them. The simplest version adds one group for the training tokens and leaves the search crawlers to follow your normal rules.

Cloudflare's traffic data shows why the split matters. Across a fixed set of its customers, it classed 80% of AI crawling in the 12 months to July 2025 as training, 18% as search and 2% as user actions, assigning purpose from operator disclosures. In July 2025 Anthropic's crawlers visited 38,000 pages for every page visit Anthropic referred, against 1,091.4 for OpenAI and 194.8 for Perplexity.¹² Training is most of the crawling and is the part with no route back to your site, so blocking it while leaving the search crawlers alone keeps the traffic that can send visitors.

Leaving the search crawlers unnamed is deliberate. Naming them in an Allow: / group looks tidy but removes your default rules for them. If you do want them named, copy every shared Disallow line into their group as well.

Know the cost of one line. Blocking Google-Extended also opts you out of grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.⁵ A business that wants to be cited in the Gemini app may decide to leave that token out, accepting Gemini training as the price.

A publisher with licensing deals, or a membership site, may want to keep content out of AI products altogether. Add the search and user-triggered tokens to the training group: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Expect three gaps.

• User-triggered fetchers may ignore the file. OpenAI says robots.txt rules may not apply to ChatGPT-User,² Perplexity says Perplexity-User generally ignores them,⁴ and Meta says Meta-ExternalFetcher may bypass them.⁷ Enforcing a block on those needs firewall rules matched to their published IP ranges.
• Google's AI Overviews and AI Mode are not controlled by any AI token. Blocking Googlebot would remove you from Google Search. The documented route is the Search generative AI control in Search Console, which excludes your links and content from AI Overviews, AI Mode and generative AI features in Discover, and does not affect other parts of Search.¹³
• robots.txt is not access control. RFC 9309 says its rules are not a form of access authorisation, and listing paths in it makes them discoverable.¹⁰ Anything truly private belongs behind a login.

Some businesses want every model and assistant to know them: a developer tools company whose documentation should be in every coding assistant, or a destination marketing body that wants visitors from anywhere. Here the file needs no AI-specific lines at all. Keep your normal User-agent: * group and do not name the AI crawlers.

The work in this configuration happens at the edge. OpenAI says that to be eligible for ChatGPT search, the host or CDN must allow traffic from its published search bot IP addresses.¹⁴ Bot management that challenges unfamiliar user agents can block crawlers your robots.txt welcomes, so check the firewall rules as well as the file.

A robots.txt that parses correctly is only half the job. These checks show whether crawlers are behaving as you intended.

I would put a short version of this check on the list for every CDN change, platform move and security review. A new bot management rule can undo a careful robots.txt overnight, and nobody notices until AI referrals fall.