When your CDN blocks AI crawlers, whatever robots.txt says

A site can have a carefully written robots.txt and still be invisible to AI search, because the crawler's request is stopped before it reaches the server. Bot management at a CDN or web application firewall sits in front of the origin and decides, request by request, whether a crawler gets the page, a challenge or a 403. Since 2025 that layer has increasingly been set up to stop AI crawlers by default, and a rule meant for training crawlers can also catch search crawlers, user-triggered fetchers and, now and then, Googlebot. robots.txt records what you would like. The edge decides what happens.

Choosing the tokens for the file itself is covered in how to set up robots.txt for AI crawlers, and the wider mechanics of crawling and retrieval in the AI search guide. What follows assumes the file is right and checks whether the edge agrees with it.

Two layers decide what a crawler receives, and they work differently. robots.txt is read voluntarily. Cloudflare's own documentation says compliance is voluntary and that the file does not prevent crawlers from accessing content at a technical level.¹ The edge enforces whatever rules it holds, whether or not the crawler read robots.txt first, and whether or not it would have obeyed it.

The last step explains why so many teams miss the problem. They check their server logs, see no errors for OAI-SearchBot, and conclude that all is well, when the requests were being stopped one hop earlier and never logged at all.

Cloudflare is the clearest case because it documents its defaults in detail. In July 2025 it said it would block AI crawlers that access content without permission or compensation by default, and that every new domain would be asked whether to allow AI crawlers.² In July 2026 it gave every plan, including Free, three AI behaviours to control separately: Search, for crawlers that index content to answer questions later; Agent, for automated activity on a person's behalf such as chat fetch bots and browser-use agents; and Training, for crawlers that take content to train or fine-tune models. Each can be blocked on all pages, blocked only on pages that display ads, or allowed.³

The same announcement set new defaults from 15 September 2026. New domains will have Training and Agent bots blocked on pages that display ads, with Search allowed, and multi-purpose crawlers that combine Search and Training will be affected by the block on Training.³ By Cloudflare's definition, user-triggered fetchers belong under Agent. So a publisher site that onboards after that date can stop assistants fetching an article when a reader asks about it, without anyone choosing to.

The defaults follow what Cloudflare measures on its own network. Across a fixed set of customers, it classed 80% of AI crawling in the 12 months to July 2025 as training, 18% as search and 2% as user actions, and in July 2025 Anthropic's crawlers visited 38,000 pages for every page visit Anthropic referred.⁴ They come from a company that sells AI crawler controls, so read them as one well-placed vendor's view. They do explain why training crawlers are the first thing an edge blocks, and why the search and user-triggered traffic that does send visitors can get caught alongside it.

Other providers make similar choices with different defaults. In AWS WAF's Bot Control rule group, the category rules block only unverified bots, with one exception: the CategoryAI rule blocks AI bots whether they are verified or not.⁷ AWS also notes that Bot Control verifies bots using the IP address of the request origin, so verified bots arriving through a proxy or load balancer need a rule placed before the group.⁷ Perplexity's crawler documentation gives example rules for both Cloudflare and AWS WAF that match the user agent and the published IP ranges together.⁸ Whatever your provider, find the equivalent of each row in the table above and write down how it is set.

The operators split their crawlers by purpose, and the cost of a block at the edge follows the purpose. The robots.txt post lists every token. The table below covers what an edge block does to each kind.

The last row has a practical consequence. An edge rule cannot target Google-Extended, so any rule written to stop "Google AI" by matching Google user agents or IP ranges is blocking Googlebot. That costs more than AI features, which themselves need crawling to be allowed in robots.txt and by any CDN or hosting infrastructure.¹² The status code the edge returns matters too. A 403 means the server understood the request and refuses to authorise it,¹³ and Google does not index URLs that return a 4xx status code and removes indexed URLs that start returning one.¹⁴ A 429 signals too many requests in a given amount of time,¹⁵ and Googlebot treats it like a server error that slows crawling. Google's documentation also advises against using 401 or 403 to limit crawl rate.¹⁴

When Googlebot is caught at the edge, the cause is rarely the AI controls. I would look first at rate limits tuned for human traffic, country blocks, custom rules that match "bot" in the user agent, and verification that fails because the CDN sees a proxy's address instead of the crawler's.

Anyone can send a request with GPTBot or Googlebot in the user agent. Cloudflare treats a bot as verified only when it identifies itself in a way that can be checked: a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS.¹⁶ You can run the same checks on any IP address in your logs.

A request that claims to be OAI-SearchBot from an address outside OpenAI's list is not OAI-SearchBot. Blocking it is correct, and it should not count as evidence of what OpenAI's crawler receives.

The quickest test is to request a page while pretending to be each crawler, and compare the results with a normal browser request. In bash:

• for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User Googlebot; do printf '%-18s' $ua; curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; $ua)" https://www.example.com/; done
• The same in PowerShell: foreach ($ua in 'GPTBot','OAI-SearchBot','ClaudeBot','PerplexityBot','Googlebot') { try { $r = Invoke-WebRequest 'https://www.example.com/' -UserAgent "Mozilla/5.0 (compatible; $ua)" -UseBasicParsing; "$ua $($r.StatusCode)" } catch { "$ua $($_.Exception.Response.StatusCode.value__)" } }
• Fetch the live file as the crawlers see it: curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot)" https://www.example.com/robots.txt. If a managed block appears above your own rules, the CDN is adding it.
• Run each test against a product or article page as well as the home page, because rules for ads or particular paths only show up there.

Read the results with care, because this test only exercises rules that look at the user agent. On Cloudflare's Free plan, AI Crawl Control identifies crawlers by user agent string,⁵ so a spoofed request will usually meet the same block as the real crawler. Rules that verify identity behave the other way: your laptop is not on OpenAI's list, so a well-configured edge should treat your fake OAI-SearchBot as unverified.

Your server's access log shows what got through. To count status codes by crawler in a log in the common combined format, where the status is the ninth field: for b in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot PerplexityBot Googlebot; do echo "$b"; grep "$b" access.log | awk '{print $9}' | sort | uniq -c; done. A crawler you allow that is missing entirely, or that only ever reaches /robots.txt, is a sign of a block upstream. Verify a sample of its IP addresses before you trust the counts.

Blocked requests are only in the CDN's records. On Cloudflare, requests challenged by Bot Fight Mode appear under Security, Analytics, in the Events tab, labelled Bot Fight Mode in the Service field.⁶ AI Crawl Control lists each AI crawler with its requests and robots.txt violations, and notes that unsuccessful requests can come from any rule or error, not only its own blocks.⁵ Read both views together: the origin log for what arrived, the edge log for what was turned away and by which rule.

The same check belongs on the list for every CDN change, security review and replatform. The website migrations guide makes the same point for Googlebot when a site changes host. The Cite skill in my open-source pack includes AI crawler access in its checks, and how to measure visibility in AI search covers spotting the referral drop that is often the first symptom.