Applebot and Applebot-Extended: Apple's crawlers explained

Applebot gets less attention than GPTBot or ClaudeBot, but it does more jobs than either. One crawler feeds Apple's search features on iPhone, iPad and Mac, collects training data for Apple's foundation models, and can supply context for AI-generated answers in Siri. Apple splits the controls for those jobs across a robots.txt token, a meta tag and a structured data property, and each one switches off something different.

This post covers Apple's crawlers in detail. For how Apple fits alongside the other operators, and the different jobs AI crawlers do, start with the AI crawlers guide. For the robots.txt file as a whole, see how to set up robots.txt for AI crawlers.

Applebot's data powers the search technology built into Spotlight, Siri and Safari, and allowing it in robots.txt lets your pages appear in those results for Apple users worldwide. The same data may also be used to train the Apple foundation models behind generative AI features across Apple products, including Apple Intelligence, Services and Developer Tools.¹

There is a third use, which matters more each year. Apple says Applebot data may provide additional context and up-to-date content when AI models generate output in Apple products, for example answers to broad world knowledge questions in Siri and Search, which may include links to the sources used.¹ That is closer to how ChatGPT search or Perplexity use the web than to training, and it has its own opt-out.

For ranking, Apple lists aggregated user engagement with results, how well search terms match the page, the number and quality of links from other pages, approximate location signals and page design characteristics, with no fixed weighting between them.¹

For search crawling and rendering, Applebot sends a Safari-style user agent that contains Applebot and a version number. Apple publishes the general format and two examples, reproduced here as published.¹

# General format
Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko)Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot)

# Desktop example
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

# Mobile example
Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)

Apple says Applebot occasionally updates the browser version it advertises while keeping this format,¹ so match on the Applebot token in log filters and firewall rules, never on the full string.

Two other names belong to Apple. iTMS traffic can come from applebot.apple.com hosts, crawls only URLs associated with content registered on Apple Podcasts, and does not follow robots.txt because it is not a general search crawler.¹ If you host a podcast feed, those requests are expected. Applebot-Extended is a robots.txt token with no crawler behind it. Apple says it does not crawl webpages and is only used to decide how data crawled by Applebot may be used,¹ so it will never appear in your logs.

• It falls back to Googlebot. If robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions.¹ HTTP Archive's 2025 Web Almanac calls this out as a case where the * group is not the fallback, because Applebot applies the Googlebot rules first.²
• Naming it may change which rules it reads. Under RFC 9309 a crawler obeys the group that matches its token and reads the * group only when no group matches.³ Apple's own example is looser: it shows an Applebot group disallowing /private/, a * group disallowing /not-allowed/, and says Applebot will crawl neither.¹ I would not rely on that. If you give Applebot its own group, repeat every shared Disallow line in it.
• It ignores `Crawl-delay`. Apple says the crawl rate adjusts automatically when a site slows down or returns errors, and that crawled content is cached to avoid recrawling.¹ A Crawl-delay line aimed at Applebot does nothing.
• It renders. Applebot may render pages in a browser, and if JavaScript, CSS or XHR requests the page needs are blocked in robots.txt, it may not render the content properly. Apple asks for everything a user needs to render the page to be available, or for the page to degrade gracefully without it.¹

Independent data backs up the rendering point. A December 2024 study by Vercel and MERJ, based on server monitoring of nextjs.org and the Vercel network, found Applebot rendering JavaScript through a browser-based crawler, as Googlebot does, while most AI crawlers it measured did not execute JavaScript. It counted 314 million Applebot fetches across the Vercel network in the month measured.⁴ Apple's 2025 foundation model update also describes headless rendering with JavaScript execution in the pipeline that collects web pages for training.⁵ A JavaScript-heavy site is therefore in better shape with Apple than with most AI operators, though checking what crawlers see on a JavaScript site is still worth doing.

When Apple announced Apple Intelligence in June 2024, its machine learning research team said the foundation models were trained on licensed data and on publicly available data collected by Applebot, and that publishers could opt out of training with a data usage control.⁶ The technical report that followed in July 2024 described respecting publishers' right to opt out of being crawled by Applebot through standard robots.txt directives.⁷

The June 2025 update is more specific. It lists data licensed from publishers, curated public and open-source datasets, and information crawled by Applebot, and says Apple follows robots.txt protocols so publishers can opt out of training while still appearing in Siri and Spotlight. Applebot remained the source of a significant portion of pre-training data, spanning hundreds of billions of pages, and Apple says it does not use users' private personal data or interactions in training.⁵

Apple's legal summary of its training datasets, dated 9 September 2026, adds detail that matters to site owners. Apple has collected text for training since 2018 and images since 2020, and collection is ongoing. Applebot does not crawl sites that require a login or sit behind a paywall. Robots.txt can tell Applebot not to crawl a site, or tell Apple not to use the content for training. Crawled content is filtered to remove some categories of personal information, such as social security and credit card numbers.⁸

Two points follow. Applebot-Extended controls how Apple uses what it crawls, so it is the token to use if you want to stay in Apple search. And Apple does not say whether disallowing it affects models already trained on your pages. Text collection began in 2018, years before the token existed, so I would treat it as a control over future training.

Large publishers use it. BuzzStream's study of the top 50 news sites in each of the US and UK by Similarweb traffic share, last updated in April 2026, found 61% blocking Applebot-Extended.⁹ That is a sample of big news brands with content to license, so it says little about a typical business site.

Most businesses want to stay visible in Apple search and AI answers, which link to sources, and may prefer not to be used for training. That needs one group. Applebot stays unnamed, so it keeps following your Googlebot or default rules.

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /checkout/
Disallow: /account/

To stay in search but keep a page out of AI-generated answers as well, add nosnippet. Apple accepts it in a robots meta tag, in a meta tag named for Applebot, or for PDFs, images and server-wide rules in an X-Robots-Tag header.¹ The cost is that suggestions to visit the page show only its title, with no description.

<!-- Applebot only -->
<meta name="applebot" content="nosnippet">

<!-- Same rule as a response header -->
X-Robots-Tag: applebot: nosnippet

Subscription publishers have a second route. Pages marked isAccessibleForFree: false in page-level JSON-LD stay eligible for search results, but Apple will not use them as context for AI-generated output. It works per page only; section-level markup with hasPart is not supported, and it does not cover training, which still needs Applebot-Extended.¹

To leave Apple altogether, name both tokens in one group. This removes the site from Spotlight, Siri and Safari results, which is a real loss for most businesses with customers on Apple devices.

User-agent: Applebot
User-agent: Applebot-Extended
Disallow: /

To allow everything, add nothing. Applebot follows your existing rules, and Apple says allowing Applebot-Extended helps improve its generative models over time.¹

# Requests from anything calling itself Applebot, by status code
grep "Applebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# Reverse lookup of the IP, then forward lookup of the hostname (Apple's own example)
host 17.58.101.179
host 17-58-101-179.applebot.apple.com

# Apple's published IP ranges
curl -s https://search.developer.apple.com/applebot.json

Expect modest volumes. In Cloudflare's comparison of more than 30 AI and search crawlers across its network, Applebot's share of crawl requests fell from 1.9% in May 2024 to 1.2% in May 2025, and its raw request volume dropped 26%.¹¹ A small site seeing few Applebot requests is normal, and is no sign of a block on its own.

The visits that come back are harder to see. I could not find any Apple documentation of a referrer or tracking parameter for clicks from Siri, Spotlight or Safari suggestions, so expect some of that traffic to land as direct or as Safari referrals. How to measure visibility in AI search covers what you can measure.

• June 2024. Apple announced Apple Intelligence and said publishers could opt out of training with a data usage control.⁶ Its July 2024 technical report linked to the Applebot page at its old support address, HT204683; the page now lives at support.apple.com/en-us/119829.⁷
• June and July 2025. Apple described its updated models, Applebot as a significant source of pre-training data across hundreds of billions of pages, and headless rendering in its collection pipeline.⁵
• September 2026. The Applebot page carries a published date of 4 September 2026 and now documents nosnippet as the opt-out from AI-generated answers, X-Robots-Tag support and isAccessibleForFree.¹ The training data summary is dated 9 September 2026.⁸ Apple does not date when each control was added.

For a typical business, allow Applebot. Apple's search features sit on every iPhone, iPad and Mac, and the cost of being crawled is low. Disallow Applebot-Extended if you would prefer not to train Apple's models; it does nothing to your search visibility. Leave nosnippet off unless a page holds content you sell, because Apple's AI answers can link to the sources they use.

A publisher with content to license would add Applebot-Extended to its training group, mark paid pages with isAccessibleForFree: false, and use nosnippet on anything that should not be summarised. Everyone should check whether robots.txt has a Googlebot group, because that is what Applebot will follow. Two things are not worth the effort: a Crawl-delay for Applebot, which it ignores, and IP blocks against a crawler that reads robots.txt.