Meta's crawlers: Meta-ExternalAgent, Meta-ExternalFetcher and link previews

Meta sends more than one crawler to your site, and the one most people have heard of is the one they should never block. facebookexternalhit has fetched pages for link previews on Facebook for years. It now shares Meta's crawler documentation with crawlers for AI training, Meta AI search, advertising and user-requested fetches, each with its own robots.txt token. A blocklist that treats every Meta user agent as an AI crawler can break the previews your social team depends on.

This post covers what each Meta crawler does, what blocking it costs, and how to check the traffic is real. How Meta fits alongside OpenAI, Anthropic, Google and the rest is in the AI crawlers guide, and the general rules for writing the file are in how to set up robots.txt for AI crawlers.

Meta lists its crawlers on one page on Meta for Developers, checked for this post in October 2026.¹ The tokens below are the ones to use in robots.txt. Meta writes them in lower case, and RFC 9309 requires crawlers to match the user agent token case-insensitively,² so Meta-ExternalAgent and meta-externalagent in your file mean the same thing.

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1

meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

# Each of the four meta-* crawlers may also send the short form,
# for example meta-externalagent/1.1

Meta's page says to allow up to 24 hours for robots.txt changes to take effect, because its crawlers may cache the file for that long.¹ It also says Meta relies on industry-standard practices like robots.txt and not on non-standard formats such as NoAI tags. A noai meta tag is not a control Meta has said it reads, so put your preferences in robots.txt.

When someone pastes your URL into a Facebook post, an Instagram message or a Messenger chat, facebookexternalhit fetches the page and reads its Open Graph tags to build the preview card. It caches what it finds.¹ Open Graph is a set of <meta> tags, and its four required properties are og:title, og:type, og:image and og:url, with og:description optional but recommended.⁴

Meta sets four technical requirements for this crawler. Your server must support gzip and deflate encoding. Open Graph tags must appear within the first 1 MB of the page or they are cut off. The page must respond within a few seconds, or Facebook cannot display the content. And the server must either honour the byte range in the crawler's Range header or ignore that header altogether.¹

Blocking facebookexternalhit in robots.txt, or challenging it at a CDN, takes away the crawler's access to the information it needs, and Meta says that when the crawler cannot fetch the content, Facebook cannot display it.¹ Shared links then appear without the title, description and image that make people click. This is a common side effect of bot management rules that challenge every user agent containing a Meta or Facebook name, and it is covered in more detail in when your CDN blocks AI crawlers.

facebookexternalhit is a link preview fetcher, and industry data treats it that way. Cloudflare left it out of its 2025 analysis of AI crawler traffic, describing it as the bot that fetches page content for link previews.⁵ If a client asks to block AI crawlers, this is the Meta user agent to leave alone.

If a preview is wrong or missing, Meta's documented fix is to pass the URL through the Sharing Debugger or the Sharing API, which forces a fresh crawl. You can also simulate the crawler's request yourself.¹

curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" \
  -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" \
  "https://www.example.com/page/"

Check that the response is a 200, that it arrives quickly, and that the og: tags are in the HTML you get back. If your Open Graph tags are added by JavaScript after the page loads, test whether they appear in this raw response at all. Checking what crawlers see on a JavaScript site walks through that test.

Three of the five crawlers relate to Meta AI, and they do different jobs.

• Meta-WebIndexer is the search crawler. Meta says it navigates the web to improve Meta AI search results, and that allowing it helps Meta cite and link to your content in Meta AI responses.¹ It is the closest thing Meta has to OAI-SearchBot or Claude-SearchBot.
• Meta-ExternalAgent is described as crawling for use cases such as training foundation AI models or improving products by indexing content directly.¹ The second half of that description is broad. Meta does not say which products it means, so blocking this token may have effects beyond training that Meta has not spelled out.
• Meta-ExternalFetcher fetches individual links when a user asks, and supports agentic features that help AI navigate websites to complete tasks.¹ It is the equivalent of ChatGPT-User or Claude-User.

Meta does not document a tracking parameter, a referrer value or a referral report for Meta AI, so I would not expect Meta AI clicks to be labelled cleanly in analytics. How to measure visibility in AI search covers the prompt-sampling approach that works when referral data is thin.

Meta-ExternalAgent is one of the busiest AI crawlers on the web. Cloudflare's Radar data put it at 19% of AI-only crawler traffic in May 2025, third behind GPTBot at 30% and ClaudeBot at 21%, identifying crawlers by matching user agent strings against the open-source ai.robots.txt list.⁵ A second Cloudflare analysis, published in August 2025, put Meta-ExternalAgent at 17.7% of AI-only bot traffic in July 2025, up from 2.4% in July 2024. Counting search crawlers such as Googlebot as well, its share was 7.5%, measured across a fixed set of Cloudflare customers.⁶ The same analysis published crawl-to-referral ratios for OpenAI, Anthropic, Perplexity and others but gave none for Meta, so there is no public figure for how much traffic Meta sends back for what it crawls.

These match the configurations in the robots.txt post. A crawler follows only the group that names it, and falls back to the User-agent: * group when none does.² Leaving facebookexternalhit and Meta-WebIndexer unnamed keeps them on your shared rules, which is what most sites want.

# Training crawler. Add GPTBot, ClaudeBot, CCBot and the
# other training tokens to this same group if you block them.
User-agent: meta-externalagent
Disallow: /

# facebookexternalhit and meta-webindexer are not named,
# so they follow the shared rules below.
User-agent: *
Disallow: /checkout/
Disallow: /account/

# meta-externalfetcher may bypass this because a user asked
# for the page. Use a firewall rule if it must be kept out.
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
Disallow: /

# Link previews still work: facebookexternalhit is not named.
User-agent: *
Disallow: /checkout/
Disallow: /account/

I would leave meta-externalads alone unless you have a specific reason. Meta describes it only as crawling to improve advertising and business products, and does not say what changes for an advertiser or a site that blocks it. If you run Meta ads to your own landing pages, blocking an ads crawler is a change I would test before making. And robots.txt is not access control: RFC 9309 says its rules are not a form of access authorisation,² so anything private belongs behind a login.

Anyone can send a meta-externalagent user agent string, so a log line proves nothing on its own. Meta's current crawler page recommends allowlisting facebookexternalhit by IP address as the more secure option, but no longer says how to find those addresses.¹ Archived copies of the same page from 2024 and 2025 had a Crawler IPs section. It said a crawler whose source IP appears in the routes announced by AS32934 is coming from Meta, gave a whois command to list them, and warned that the addresses change often.⁷ The current page has no such section. AS32934 is Meta's autonomous system in PeeringDB,⁸ so the check still works, though Meta no longer documents it as the method.

# Routes announced by AS32934 (the command Meta used to document)
whois -h whois.radb.net -- '-i origin AS32934' | grep ^route

# Requests per Meta crawler. Use -i: Meta's agents are lower case.
grep -ioE "facebookexternalhit|meta-webindexer|meta-externalads|meta-externalagent|meta-externalfetcher" access.log \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

# Client IPs claiming to be Meta-ExternalAgent, to check against the routes
grep -i "meta-externalagent" access.log | awk '{print $1}' | sort | uniq -c | sort -rn

Two things to watch in the results. A Meta crawler that only ever receives 403s or challenge pages is blocked at your edge, whatever robots.txt says. And requests that carry a Meta user agent from addresses outside AS32934 are not from Meta, so they can be treated like any other unverified bot. Meta does not document reverse DNS hostnames for its crawlers, so I would not rely on a reverse DNS check.

• By September 2024 the page listed facebookexternalhit, Meta-ExternalAgent and Meta-ExternalFetcher, with the Crawler IPs section and its AS32934 command.³
• By October 2025 Meta-WebIndexer and Meta-ExternalAds had been added. Meta-WebIndexer is the first Meta crawler tied explicitly to citing and linking in Meta AI.⁷
• Since then the Crawler IPs section has been removed, and the description of Meta-ExternalFetcher has grown. The October 2025 copy said it performs user-initiated fetches of individual links to support specific product functions.⁷ The current one adds evaluating and improving agentic AI, including helping AI navigate websites to complete tasks for users.¹

Meta does not date the changes on the page itself, so the archive is the only record. I would check the page again at each robots.txt review.

For most businesses I would block meta-externalagent, leave meta-webindexer, meta-externalads and facebookexternalhit unnamed, and stop there. That keeps your link previews working, keeps you eligible for citations in Meta AI, and opts out of the Meta crawler that shows up in Cloudflare's AI crawler rankings. Adding meta-externalfetcher to the block is worth it only if you have a firm reason to keep out of Meta AI entirely, and then it needs a firewall rule to mean anything.

Then check two things. Run the facebookexternalhit simulation against a few key pages and confirm the previews build. And look at your CDN's bot rules for anything that challenges Meta user agents as a group, because that is where previews usually break. A quick test in the Sharing Debugger after any bot management change takes a minute and saves a social campaign going out with blank cards.