OpenAI's crawlers: GPTBot, OAI-SearchBot and ChatGPT-User explained

OpenAI sends several different visitors to websites, and each one does a different job. One collects training data, one builds the index behind ChatGPT search, one fetches a page when a person asks ChatGPT about it, one checks advertisers' landing pages, and a cloud browser carries out tasks for ChatGPT Work users. A robots.txt line aimed at one of them has no effect on the others, so mixing them up can take a site out of ChatGPT search when the owner only meant to opt out of training.

This post is part of the AI crawlers guide, which explains the split between training, search and user-triggered fetching across all the major operators. For the parsing rules behind the file itself, see how to set up robots.txt for AI crawlers. Everything below was checked against OpenAI's own pages on 12 October 2026.

OpenAI's crawler overview lists four user agents: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User.¹ The page used to live at platform.openai.com/docs/bots, which now redirects to developers.openai.com/api/docs/bots, so update any internal documentation that still links to the old address.

Three sentences at the top of OpenAI's page explain most of how these interact. Each setting is independent of the others, so a site can allow OAI-SearchBot to appear in search while disallowing GPTBot. If a site allows both, OpenAI may use the results of one crawl for both purposes to avoid crawling twice. And after a robots.txt change, it can take about 24 hours for search results to adjust.¹

The full user agent strings

OpenAI publishes the GPTBot and OAI-SearchBot strings as examples and notes that the version number may change, so match on the token in your log filters and firewall rules, never on the whole string. ChatGPT-User and OAI-AdsBot are given as full strings.¹

# GPTBot (example; version may change)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

# OAI-SearchBot (example; version may change)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

# OAI-SearchBot when fetching robots.txt
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

# ChatGPT-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

# OAI-AdsBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

The robots.txt; marker is a small, useful detail. GPTBot and OAI-SearchBot may add it when they request your robots.txt file, which lets you tell those requests apart from page crawls even when your logs do not record the path.¹ A crawler you have blocked will keep requesting robots.txt, and the marker makes it easy to confirm that this is all it is doing.

• GPTBot. Blocking it is the training opt-out. It tells OpenAI your content should not be used to train its foundation models, and it has no effect on ChatGPT search, because search is controlled by a different token.¹
• OAI-SearchBot. Blocking it means your pages will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing it, and allowing requests from its published IP ranges, if you want to appear.¹ For most businesses this is the block that costs money.
• ChatGPT-User. This is the visit that happens when someone pastes your URL into ChatGPT or asks a question that needs your page. OpenAI says robots.txt rules may not apply to it and that it is not used to decide search eligibility.¹ If you block it at the firewall, ChatGPT cannot read your page for that user, and the answer they get is built from other sources.
• OAI-AdsBot. It matters only if you advertise in ChatGPT. It visits submitted landing pages only, and OpenAI says advertisers must allow it for ads review.² If you do not advertise, you can ignore it.

The relationship between training and search is the point people most often get wrong. Blocking GPTBot does not remove you from ChatGPT search, and allowing OAI-SearchBot does not opt you in to training. A site that wants out of both has to block both. One gap is worth knowing: OpenAI's page says OAI-AdsBot data is not used for training, but it makes no equivalent statement about OAI-SearchBot or ChatGPT-User.¹ The training signal it describes is the GPTBot disallow, so that is the line to rely on.

Cloudflare's traffic data gives a sense of scale. In an August 2025 analysis of Cloudflare Radar data, using a fixed set of customers to remove the effect of its own growth, GPTBot's share of AI-only crawler traffic rose from 11.9% in July 2024 to 28.1% in July 2025. OAI-SearchBot and ChatGPT-User reached 2.2% and 2.4% of that traffic. OpenAI's crawl-to-refer ratio, the number of crawls for each visit it referred, fell from 1,217 in January 2025 to 1,091 in July.³

Cloudflare's 2025 Radar Year in Review, published in December 2025 and covering 1 January to early December, adds detail. GPTBot was the most active verified bot after Google's own crawlers, originating about 7.5% of verified bot traffic. ChatGPT-User request volumes peaked at as much as 16 times their level at the start of the year, with a weekly pattern that Cloudflare reads as use at school and work. OAI-SearchBot's late October peak was about five times its January volume, and OpenAI's crawl-to-refer ratio was spiky, reaching as much as 3,700 to 1 in March.⁴

Site owners have noticed. HTTP Archive's 2025 Web Almanac, which analyses robots.txt files across the sites in its crawl, found gptbot named in 4.5% of desktop sites and 4.2% of mobile sites, up from 2.9% and 2.7% in 2024. chatgpt-user appeared in 2.8% of desktop sites.⁵ Those figures count mentions, so they include files that allow a bot as well as files that block it. Among Cloudflare's top 10,000 domains, GPTBot was one of the user agents with the most whole-site disallow rules.⁴

My reading of this is simple. Training crawls make up most of OpenAI's volume and have no direct route back to your site. Search and user-triggered visits are smaller but growing quickly, and they are the ones attached to people reading answers about you. That is the case for blocking GPTBot if training bothers you, and for leaving the other two alone.

One parsing rule decides how these files behave. A crawler follows the group that names it, and it reads the User-agent: * group only when no group matches it.⁶ So the moment you name OAI-SearchBot, it stops reading your shared disallows. The configurations below leave the bots you want to keep unnamed for that reason, which matches the approach in the robots.txt guide.

# Goal 1: stay in ChatGPT search, stay out of training (my default)
User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /checkout/
Disallow: /account/
# OAI-SearchBot, ChatGPT-User and OAI-AdsBot are not named,
# so they follow the * group.


# Goal 2: keep OpenAI out as far as robots.txt allows
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /
# ChatGPT-User may not follow this. Enforce it at the firewall
# using https://openai.com/chatgpt-user.json.


# Goal 3: allow everything
# No OpenAI lines at all. Keep your normal * group.

Be careful with OpenAI's advertiser example. Its help page shows User-agent: OAI-SearchBot and User-agent: OAI-AdsBot groups with Allow: /.² Pasted into a file that already disallows checkout and account pages for everyone, those two groups open every one of those paths to both bots. If you want them named, copy each shared Disallow line into their groups as well.

To make a site eligible for ChatGPT search, OpenAI asks for two things: allow OAI-SearchBot to crawl, and confirm that the host or CDN allows traffic from its published search bot IP addresses. Even then, placement is not guaranteed, because results are ranked on several factors.⁷

OpenAI's advertiser guidance describes the layers that most often get in the way, and the same layers can block OAI-SearchBot. Bot mitigation from providers such as Cloudflare or Akamai can block crawlers with 403 errors, application checks such as CAPTCHAs and JavaScript challenges can stop them after they pass the CDN, and sudden spikes can trigger rate limits that return 429 responses. The same page says Cloudflare has verified and allowlisted OAI-AdsBot.² Cloudflare also lists OpenAI's crawlers among its verified bots.³ What a verified bot is allowed to do still depends on your own settings, which the post on CDNs that block AI crawlers goes through.

Anyone can put GPTBot in a user agent string, so the string alone proves nothing. These checks separate OpenAI's traffic from imitations.

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot" access.log \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head -50

Then compare the top addresses with the JSON files. On most sites I would do this once to establish a baseline, and again after any CDN or platform change.

OpenAI's agent products have changed quickly. Its help page for ChatGPT agent now says the feature is no longer available and points users to ChatGPT Work for longer, multi-step tasks.⁸ ChatGPT Work's cloud browser runs on a separate computer in the cloud and can read pages, click buttons, fill in forms and carry out steps on public and signed-in websites. By default it asks the user before visiting a new site, and OpenAI notes that each website decides whether to allow cloud browser traffic.⁹

The cloud browser signs its outbound requests using Web Bot Auth, so site operators can check that a request really comes from ChatGPT.¹⁰ The signatures follow HTTP Message Signatures, the IETF standard published as RFC 9421 in February 2024.¹¹ Each request carries three headers: Signature, Signature-Input and Signature-Agent, the last set to "https://chatgpt.com". The public keys are published at https://chatgpt.com/.well-known/http-message-signatures-directory.¹⁰

Two things are missing from OpenAI's page. It gives no user agent string for the cloud browser, and it says nothing about robots.txt.¹⁰ So robots.txt is not a documented control for this traffic. If you want it, allowlist it by signature at the edge and make sure any proxy in front of your origin keeps the three signature headers, which is the first thing OpenAI's troubleshooting steps ask you to check. If you do not want it, a rule on the same signature or detection ID is the way to block it.

Retailers, booking sites and anyone selling a service online should think about this one. A customer who asks ChatGPT to check availability or request a quote is a customer, and a bot rule that challenges the cloud browser turns them away. I would allow the signed traffic unless there is a specific reason not to.

Answers that use web search may include citations that open the source, a Sources view listing cited and related links, and images that link back to where they came from.⁷ A site that blocks OAI-SearchBot can still appear as a navigational link,¹ which is why a robots.txt block alone does not make a brand disappear from ChatGPT.

For measurement, clicks from ChatGPT can be identified by their referrer. Cloudflare's crawl-to-refer ratio counts HTML requests whose Referer header points to the AI platform,⁴ so segmenting analytics on chatgpt.com as a referrer or source is a sound starting point. I could not find a current OpenAI page that documents a UTM parameter on ChatGPT links, so I would not build reporting that depends on one. Check your landing page reports for any tagged URLs, and see how to measure visibility in AI search for the wider method.

The publisher controls OpenAI currently documents are the robots.txt tokens and the IP allowlisting described above. Its crawler page does not describe a noindex meta tag or any other page-level control for ChatGPT search. If a page must never appear even as a bare link, robots.txt cannot promise that, and a login is the dependable answer.

• OAI-AdsBot now sits on OpenAI's crawler page with its own IP file, tied to ads on ChatGPT.¹ It is the one OpenAI bot that advertisers are told they must allow.²
• ChatGPT agent is no longer available, and the help page, updated in September 2026, sends users to ChatGPT Work.⁸ The signed cloud browser now carries that browsing.⁹
• ChatGPT Atlas, OpenAI's browser, has been retired and no longer receives updates.¹²
• OpenAI's Publishers and Developers FAQ at help.openai.com/en/articles/12627856, redirected to the Atlas retirement article when I checked on 12 October 2026. An archived copy from 4 October 2026 shows it was written for ChatGPT Atlas, and advice that cites it may no longer match what OpenAI publishes.
• User agent versions. The example strings for GPTBot and OAI-SearchBot now show version 1.4.¹ Rules written against an older full string will miss them.

For most SaaS, ecommerce, B2B and local businesses, I would use the first configuration. Leave OAI-SearchBot and ChatGPT-User unnamed so they follow your normal rules, add a GPTBot block if training on your content bothers you, and skip it if it does not. Then spend the time on the edge: test that your CDN lets the addresses in searchbot.json through, and that the cloud browser's signed requests are not challenged. That check is worth more than any line in robots.txt.

A publisher with licensing deals, or a membership site, can use the second configuration, with a firewall rule for the chatgpt-user.json ranges and a block on the cloud browser's signature, accepting that its brand may still appear as a navigational link. I would not name OAI-AdsBot unless you advertise, and I would not spend time hunting down every request that pretends to be GPTBot. If it is not on OpenAI's list, it is not OpenAI, and your normal bot rules should deal with it.

The other operators handle the same split in different ways. Anthropic's crawlers and Perplexity's crawlers have their own posts, and Google's AI crawlers work differently again, because AI Overviews use Googlebot.