Bingbot and Copilot: how Microsoft's crawler feeds AI answers

Most guides to AI crawlers give you a list of user agents to block. Microsoft does not fit that pattern. There is no Microsoft equivalent of GPTBot or Google-Extended in Bing's crawler documentation, and nothing called a Copilot bot. The same Bingbot that crawls for Bing search feeds Copilot, and the levers for AI use are page-level meta tags. If you copy a robots.txt template looking for a Microsoft AI line, you will either find nothing to add or end up blocking Bing altogether.

This post covers Microsoft's crawlers, what the controls do, how to verify Bingbot and how to see where Copilot cites you. For how the main operators compare, start with the AI crawlers guide. The sibling posts cover OpenAI, Google and Perplexity.

Bing lists five main crawlers. Bingbot is the standard crawler and handles most of the crawling. AdIdxBot checks ads and the pages they link to. BingPreview and MicrosoftPreview create page snapshots, and BingVideoPreview creates video previews for Bing.¹ As of October 2026 that list has no AI training crawler and no Copilot fetcher.

Desktop:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Mobile:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36  (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

W.X.Y.Z is replaced by the version of Microsoft Edge that Bing's rendering engine currently uses, so match on bingbot/2.0 and do not hard-code the full string.¹

Bing's Webmaster Guidelines now describe one system. They cover how Bing discovers, crawls, indexes and surfaces content across Bing search, Copilot and grounding API results, and they say Bing and Copilot search experiences rely on the same core crawling, indexing and ranking foundation as traditional search.² In practice, a page Bingbot cannot crawl and index is not available for Copilot to cite.

Bingbot is still one of the busiest crawlers on the web. Cloudflare compared a cohort of more than 30 AI and search crawlers across its network in May 2024 and May 2025: Bingbot ranked second by share of that cohort's crawling, though its share slipped from 10% to 8.7% while its raw requests grew 2%.³

Disallowing Bingbot removes you from much more than Bing's blue links. The guidelines list blocking Bingbot in robots.txt among the things to avoid, and say content that cannot be reliably rendered may not be indexed or selected for grounding results.² Bing's AI Performance report counts citations across Microsoft Copilot, AI-generated summaries in Bing and select partner AI integrations, and notes that, as with Bing search, it reflects only content eligible for indexing.⁴ So a Bingbot block reaches every one of those surfaces, including partner products Microsoft does not name.

The guidelines also say robots.txt controls crawl access, not indexing, and that noindex is the directive for a URL that should not appear in Bing search, Copilot experiences or grounding API results.² A robots.txt block is therefore a blunt way to leave Bing, and not a reliable way to keep a known URL out of it.

Microsoft's AI controls sit in the robots meta tags Bing supports. Each one can also be sent as an X-Robots-Tag response header, and name="robots" can be replaced with name="bingbot" so the rule applies only to Bing.⁵

The table follows Bing's meta tag reference,⁵ which matches what Bing announced in September 2023, when Fabrice Canel introduced nocache and noarchive as the controls for Bing Chat and said content carrying either tag would still appear in search results.⁶ The training rows refer to Microsoft's generative AI foundation models, and apply to content in the Bing index.

There is a cost to nocache. The current guidelines say it limits Copilot to the URL, title and snippet, reducing citation depth and answer quality, and that noarchive prevents content from being used in Copilot responses and grounding results. They also warn that nosnippet and data-nosnippet may limit Copilot citation quality.²

<!-- Stay in Bing search, stay out of Copilot answers and Microsoft model training -->
<meta name="bingbot" content="noarchive">

<!-- Let Copilot show only the link, title and snippet -->
<meta name="bingbot" content="nocache">

# The same rule as an HTTP response header, useful for PDFs
X-Robots-Tag: noarchive

One consequence is easy to miss. There is no way to keep Copilot citations and opt out of Microsoft training on full content at the same time. noarchive removes both. nocache keeps a thin citation and a thin training use. The full Copilot citation comes with the training use, which is a different trade from OpenAI or Anthropic, where separate robots.txt tokens handle search and training.

Because there is one crawler, robots.txt offers two choices for Microsoft: let Bingbot crawl or block it. Remember that a crawler named in its own group stops reading the User-agent: * group,⁷ so if you give Bingbot a group for any reason, copy your shared disallows into it. The full set of parsing rules is in how to set up robots.txt for AI crawlers.

# Goal: stay in Bing, Copilot and Microsoft grounding.
# No Bing-specific lines needed. Bingbot follows your default group.
User-agent: *
Disallow: /checkout/
Disallow: /account/

# Goal: leave Bing entirely, which also removes you from Copilot.
User-agent: bingbot
Disallow: /

Adding GPTBot, ClaudeBot or Google-Extended to a training group does nothing for Microsoft. If staying out of Microsoft training matters to you, the route is noarchive, accepting that Copilot will no longer use or link those pages.

Bing points out that user agent strings are easy to spoof, so a request claiming to be Bingbot proves nothing on its own.¹ It documents two checks.

Older advice says Bing publishes no IP list. That was true in 2012, when the Bing Webmaster blog said so and recommended DNS checks instead.¹⁰ The current help page links the JSON list, so treat both methods as valid today and prefer DNS when the two disagree.

If Bingbot is being blocked by a bot management rule, when your CDN blocks AI crawlers covers where to look.

IndexNow lets a site notify search engines directly when a URL is added, updated or removed. You host a key file to prove ownership, then send single URLs or a JSON batch of up to 10,000 URLs per request, and participating engines share submissions with each other.¹¹ The participants listed on indexnow.org are Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot.¹²

Bing ties it to Copilot directly. The guidelines recommend IndexNow for adding, updating and removing URLs, and say timely notifications reduce outdated or incorrect URL references in Copilot responses and grounding results.² A 200 response means only that the engine received the URL,¹¹ so IndexNow speeds discovery and promises nothing about indexing. I would switch it on wherever your platform supports it.

Bing launched the AI Performance report in Bing Webmaster Tools as a public preview on 10 February 2026.¹³

• Surfaces: Microsoft Copilot, AI-generated summaries in Bing and select partner AI integrations. The partners are not named.⁴
• Metrics: total citations, average cited pages per day, page-level citation counts, and grounding queries, which are grouped phrases the AI used when retrieving the content it cited. They are not the questions users typed.⁴
• Limits: citations do not represent clicks or traffic, the data is a sample of overall citation activity, and very low activity may not appear. Data refreshes daily and exports to CSV or Excel.⁴
• Additions: in June 2026 Bing began rolling out four preview features globally: intent categories for grounding queries, topic clusters, citation share for a grounding query, and comparison with an earlier period.¹⁴

I would read it for patterns: which pages are cited, for which themes, and how that moves after you change content. Do not report citations as traffic. For the visits themselves, use your analytics, and see how to measure visibility in AI search for a method that combines the two.

Bing's index has reached other assistants. At Build in May 2023 Microsoft announced that Bing was coming to ChatGPT as the default search experience.¹⁵ OpenAI has since documented its own search crawler, OAI-SearchBot, which is covered in the OpenAI post; I would not assume any particular split between the two today.

The commercial route has changed too. Microsoft retired the Bing Search APIs on 11 August 2025 and pointed customers to Grounding with Bing Search, a tool for Azure AI Agents that brings real-time public web data into LLM responses.¹⁶ In February 2026 Jordi Ribas, Microsoft's corporate vice president for search and AI, wrote that Microsoft grounding powers nearly every major AI assistant in the market.¹⁷ He named no products, so treat that as Microsoft's claim. It is consistent with the unnamed partner integrations in the AI Performance report, and it is one more reason a Bingbot block costs more than Bing traffic.

For a typical SaaS, ecommerce, B2B or local business, I would leave Bingbot allowed in robots.txt and in the firewall, and check the logs to confirm it gets 200s on important pages. I would verify the site in Bing Webmaster Tools, switch on IndexNow where the platform supports it, and look at AI Performance once a month alongside analytics.

I would use noarchive only on content you are willing to lose from Copilot, such as paid or licensed material, and I would not use nocache as a compromise on pages you want cited, since Bing says it reduces citation depth. Blocking Bingbot entirely is worth it only if you have decided Microsoft search and AI traffic has no value to you, which for most businesses would be a mistake.