Perplexity's crawlers: PerplexityBot and Perplexity-User explained

Perplexity is the simplest of the large AI operators to configure for, and the one with the most argued-over record. It documents two user agents, says neither is used to train foundation models, and publishes an IP list for each.¹ It was also the subject of a public dispute with Cloudflare in August 2025 over whether it crawled sites that had blocked it. This post covers both: what the documented agents do and how to handle them, then what each side said.

It is one of a set of operator posts that sit under the AI crawlers guide, which explains the split between training, search and user-triggered crawlers across the industry. The file-level rules that apply to every operator are in how to set up robots.txt for AI crawlers.

Perplexity's crawler documentation lists PerplexityBot and Perplexity-User, with a full user agent string and a JSON file of IP ranges for each.¹ Both strings are written in the form below, and the token you match in logs and robots.txt is the part after compatible;.

PerplexityBot:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Perplexity-User:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Two details matter for configuration. Each robots.txt setting works independently, so a rule for PerplexityBot does nothing to Perplexity-User and the reverse. And changes can take up to 24 hours to take effect.¹ The two URLs inside the strings both redirect to the same crawler documentation page, which is a quick way to confirm you are reading the current version.

Perplexity's position is that a fetch triggered by a person asking a question is a different thing from crawling, so the crawling rules do not bind it. The same idea exists elsewhere: Google says its user-triggered fetchers generally ignore robots.txt rules for the same reason, that a user requested the fetch.² RFC 9309, the robots.txt standard, describes rules for crawlers and also says plainly that they are not a form of access authorisation.³ Whatever you think of the argument, the practical result is the same: robots.txt will not reliably keep Perplexity-User off a page.

If you need Perplexity-User kept out, the enforceable route is a firewall or bot management rule that matches the user agent and the published IP ranges together. If you only want to keep pages out of Perplexity's index, PerplexityBot is the agent to address, and robots.txt is enough.

• Blocking PerplexityBot takes you out of the index Perplexity uses to surface and link sites.¹ Expect fewer citations and fewer referrals from Perplexity answers. It does not change anything about AI training, because Perplexity says PerplexityBot does not crawl for foundation models.
• Blocking Perplexity-User with robots.txt is a statement of preference that the agent generally does not follow. Blocking it at the firewall stops Perplexity reading your page live when someone asks about it, so answers about you have to rely on other sources.
• Blocking neither is the default. PerplexityBot follows your User-agent: * group like any other well-behaved crawler, as long as you have not named it in a group of its own.

The last point catches people. Under RFC 9309, a crawler obeys the group that matches its own token and falls back to the * group only when none does.³ Google's reading of the standard is the same: the most specific matching group wins and the others are ignored.⁴ So a group that says User-agent: PerplexityBot followed by Allow: / gives PerplexityBot your checkout and account pages, because it no longer reads the disallows in the * group.

Because Perplexity separates its agents by purpose and neither is a training crawler, there are only two goals worth writing a file for. The training opt-outs for other operators belong in their own groups, as set out in how to set up robots.txt for AI crawlers.

# No Perplexity lines needed. PerplexityBot is not named,
# so it follows the shared rules below.
User-agent: *
Disallow: /checkout/
Disallow: /account/

# PerplexityBot follows this. Perplexity-User generally
# does not, so add a firewall rule if you need it kept out.
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /

User-agent: *
Disallow: /checkout/
Disallow: /account/

Listing Perplexity-User in the blocking group costs nothing and records your preference, even though you should not rely on it. If you want PerplexityBot to stay out of one section only, give it its own group and copy every shared Disallow line into that group as well as the new one.

Anyone can send a request that claims to be PerplexityBot. Perplexity's documentation treats its JSON files as the source of truth for its IP addresses, recommends combining user agent and IP checks, and advises refreshing allowlists automatically because the ranges change.¹ It does not document a reverse DNS check, so the IP lists are the only verification method Perplexity supports.

# Published ranges (needs curl and jq)
curl -sL https://www.perplexity.com/perplexitybot.json | jq -r '.prefixes[].ipv4Prefix'
curl -sL https://www.perplexity.com/perplexity-user.json | jq -r '.prefixes[].ipv4Prefix'

# Status codes per agent in a combined-format access log
for b in PerplexityBot Perplexity-User; do
  echo "$b"; grep "$b" access.log | awk '{print $9}' | sort | uniq -c
done

# Client IPs per agent, to check against the ranges above
grep "PerplexityBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn

On 4 August 2025 Cloudflare published a post by Reid Tatoris alleging that Perplexity used undeclared crawlers to get around sites' no-crawl preferences.⁵ Perplexity replied with a blog post dated the same day.⁶ Both posts are still online. What follows is what each one says, and I have not tried to settle the question between them.

Cloudflare's account

Cloudflare says it bought several new domains that were not indexed or publicly discoverable, served a robots.txt that disallowed all automated access, and added firewall rules blocking PerplexityBot and Perplexity-User. When it then asked Perplexity about those domains, Perplexity still returned detailed information about their content.⁵ Cloudflare says that when the declared agents were blocked, it saw requests with a generic user agent impersonating Chrome on macOS, from IP addresses outside Perplexity's published ranges and from several ASNs, rotating addresses after blocks, and that robots.txt was sometimes ignored or not fetched at all.⁵

Cloudflare put the declared traffic at 20 to 25 million requests a day and the undeclared traffic at 3 to 6 million a day across tens of thousands of domains, identified with machine learning and network signals. It removed Perplexity from its verified bots and added the undeclared crawler's signatures to the managed rule that blocks AI crawling, which it says is available to all customers including the free plan.⁵ The same post reports that in an equivalent test ChatGPT-User fetched robots.txt and stopped when disallowed.⁵

Perplexity's account

Perplexity's reply makes two arguments.⁶ The first is about principle: a user-driven assistant fetches a page because a person asked a specific question, uses the content immediately, and does not store it or train with it, so treating it as a bot is a mistake. The second is about attribution. Perplexity says the 3 to 6 million daily requests Cloudflare attributed to it came from BrowserBase, a third-party cloud browser service, and that Perplexity uses BrowserBase only occasionally, for fewer than 45,000 requests a day. It also says Cloudflare would not explain its method or answer its questions, and that a diagram in Cloudflare's post does not reflect how Perplexity works.⁶

Two things are worth noting. Perplexity's reply does not address Cloudflare's test domains directly, and Cloudflare's post, when I read it in October 2026, carried no editor's note responding to the BrowserBase explanation. Neither side has published data the other has accepted.

PerplexityBot exists to surface and link sites in Perplexity results, and Perplexity-User may link to a page it visits in the answer it gives.¹ Perplexity does not document a tracking parameter on outbound links in its crawler documentation, so look for referrals from perplexity.ai in your analytics and do not expect a tagged campaign.

Referral volume is modest next to the crawling. Cloudflare's August 2025 analysis of its customer traffic put Perplexity's crawl-to-refer ratio at 54.6 crawls for every referred visit in January 2025, rising to 194.8 in July 2025.⁷ Those figures come from Cloudflare's own network, and its post does not give customer counts, so I read them as a direction of travel. Perplexity still sends visits, which is more than a training crawler ever will.

Perplexity runs commercial terms for publishers alongside the technical controls. It launched the Perplexity Publishers' Program on 30 July 2024 with partners including TIME, Der Spiegel, Fortune, Entrepreneur, The Texas Tribune and WordPress.com, offering a share of advertising revenue when a partner's content is referenced, free API access and Enterprise Pro for staff.⁸ On 25 August 2025 it announced Comet Plus, a $5 subscription whose revenue it says goes to participating publishers minus a small portion for compute, allocated across human visits, search citations and agent actions.⁹ Both posts give publishers@perplexity.ai as the contact.⁸⁹ These programmes matter to large publishers. For most businesses they are not worth chasing.

For a typical SaaS, ecommerce, B2B or local business, I would leave both Perplexity agents alone in robots.txt and make sure the CDN lets PerplexityBot through from its published ranges. Neither agent trains models by Perplexity's account, so blocking them gains nothing on training and costs you citations. Check your logs once after any CDN change, and verify a sample of IPs before trusting what the user agent column says.

A publisher that licenses content, or a membership site, may reasonably decide otherwise. In that case block both tokens in robots.txt, add a firewall rule for Perplexity-User, and accept that the evidence from 2025 is disputed and that no file setting is a guarantee. To see whether any of this is changing how often you are cited, start with how to measure visibility in AI search.