Google's crawlers and AI: Googlebot, Google-Extended and the rest

Google runs more crawlers than any other operator a site owner is likely to meet, and only a few of them have anything to do with AI. The one that matters most for AI search is Googlebot, which is also the one that keeps you in Google Search. AI Overviews and AI Mode show pages from the Search index, and Google treats robots.txt rules for Googlebot as the access control for both.¹ The tokens with AI in their description, such as Google-Extended and Google-CloudVertexBot, control other products.

This post covers Google's part of the AI crawlers guide: each crawler and fetcher that matters for AI, what blocking it does in business terms, and how to check what is really visiting. For the general rules of writing the file, see how to set up robots.txt for AI crawlers. Bing and Copilot are covered in Microsoft's AI crawlers.

Google sorts everything that fetches pages into three groups, and the group tells you most of what you need to know about robots.txt and verification.²

Googlebot has a smartphone and a desktop version, and both use the robots.txt token Googlebot. Rules addressed to it affect Google Search, including Discover and all Search features, as well as Google Images, Google Video and Google News.³ The published strings are:

# Smartphone
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

# Desktop
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Chrome/W.X.Y.Z is a placeholder for the current Chromium version, so filter logs with a wildcard there and never an exact version.³

For AI search, the point to hold on to is that AI Overviews and AI Mode have no crawler of their own. To be eligible as a supporting link in either, a page must be indexed and eligible to show in Google Search with a snippet, and there are no extra technical requirements, special files or schema.¹ Google states that robots.txt directives for Googlebot are the control for how a site is crawled for Search, AI features included.¹ Blocking Googlebot to keep out of AI Overviews would take the site out of Google Search altogether, which for most businesses would be the most expensive robots.txt line they could write.

Googlebot's share of crawling has grown alongside these features. Across a fixed set of Cloudflare customers, Googlebot requests rose 96% between May 2024 and May 2025, and its share of requests from the AI and search crawlers Cloudflare tracks rose from 30% to 50%.⁴ Cloudflare noted that the growth coincided with the launch and expansion of AI Overviews, without claiming that one caused the other.

Google-Extended is a product token, not a crawler. It has no user agent string: Google crawls with its existing user agents and reads the token from robots.txt to decide what the crawled content may be used for.³ RFC 9309 says a product token should normally be a substring of the user agent a crawler sends,⁵ and Google-Extended is a deliberate exception, which is why you will never see it in a log file.

Disallowing Google-Extended covers two uses. It stops content Google crawls from your site being used to train future generations of Gemini models, which power Gemini Apps and the Vertex AI API for Gemini. It also stops the content being used for grounding, meaning Search index content given to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal.³

So blocking Google-Extended is a trade, with a cost on each side. You keep your content out of Gemini training, and you also make it ineligible as a grounding source when someone asks the Gemini app a question your site could answer. It does nothing to AI Overviews or AI Mode, which Google handles under Search. Google rewrote the token's description on 25 April 2025, after publisher feedback, to add specificity and clarity.⁶

Plenty of sites have made the call. HTTP Archive's 2025 Web Almanac found google-extended named in 3.4% of desktop files and 3.0% of mobile files.⁷ Those are mentions, so they include files that allow the token as well as those that block it.

GoogleOther is Google's generic crawler for product teams. Rules addressed to it do not affect any specific product, and Google gives one-off crawls for internal research and development as an example use. It has image and video variants, GoogleOther-Image and GoogleOther-Video, which also follow a GoogleOther group.³ Its strings end in (compatible; GoogleOther) within the same Chrome-based patterns Googlebot uses. Cloudflare's 2025 analysis found Googlebot and GoogleOther together made up most of Google's crawling.⁴

Google does not say GoogleOther feeds AI products, and I would not claim that it does. Equally, blocking it has no documented cost to Search, so a site that wants to keep its exposure to Google's internal research down can block it without losing anything it can measure.

Google-CloudVertexBot crawls sites when the site owner asks it to, as part of building Vertex AI Agents, and has no effect on Google Search or other products. Its user agent contains the substring Google-CloudVertexBot. It answers to two tokens, Google-CloudVertexBot and Googlebot,³ so if it has no group of its own it follows your Googlebot group. You only need to think about it if your own company is building a Vertex AI agent over your site, and then the fix is to allow it.

Two of Google's user-triggered fetchers are AI products. Because a person asked for the fetch, Google says these fetchers generally ignore robots.txt rules.⁸

• Google-Agent is used by agents hosted on Google infrastructure to navigate the web and take actions when a user asks. Its strings contain compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent inside a Chrome-based user agent, and it uses the IP ranges in user-triggered-agents.json.⁸ Google added it to its documentation on 20 March 2026.⁶
• Gemini Notebook fetches individual URLs that users add as sources to their projects. Its strings contain compatible; Google-GeminiNotebook. It replaced the Google-NotebookLM user agent, which Google supported until August 2026.⁸ The rename followed NotebookLM becoming Gemini Notebook, announced in the changelog on 16 July 2026.⁶
• Google Read Aloud fetches a page and reads it out with text to speech when a user asks. It is not a generative AI product, but it is the kind of request that looks automated in logs.⁸

Google-Agent is also where Google is testing a stronger form of identification. Some of its requests are signed under the Web Bot Auth draft and carry a Signature-Agent header with the value g="https://agent.bot.goog". The public keys are published at https://agent.bot.goog/.well-known/http-message-signatures-directory. Google signs only a subset of requests during the experiment, so IP and DNS checks are still needed.⁹

In business terms, a visit from either fetcher means a real person used a Google AI product on your page. I would not block them. If a business has a reason to, robots.txt will not do it reliably, and the enforceable route is a firewall rule matched to the published IP ranges.

Since robots.txt cannot separate Search from AI Overviews, Google offers two other kinds of control.

The first is the Search generative AI control in Search Console, under Settings and then Search generative AI. Excluding a property stops its links and content appearing in AI Overviews, AI Mode and generative AI features in Google Discover, and its content can no longer be used to generate a response or preview in them. Google says the control is not used as a ranking or inclusion signal for the rest of Search, that it does not affect AI training (Google-Extended handles that), and that content is typically excluded within one to two days of the change going live. The control was rolled out to all websites worldwide on 31 August 2026.¹⁰ Child properties inherit the setting from their closest configured parent, so check which property you are changing.¹⁰

The control arrived alongside regulatory pressure. On 3 June 2026 the UK's Competition and Markets Authority imposed a publisher conduct requirement on Google, which requires tools for publishers to keep their content out of AI features in search, such as AI Overviews, and proper attribution with clear links in AI results. Google was given nine months to implement all the changes.¹¹

The second kind is the snippet controls, which work page by page. Google's robots meta tag documentation says nosnippet applies to AI Overviews and AI Mode and also prevents the content being used as a direct input for them, and that max-snippet limits how much of the content may be used as a direct input. The data-nosnippet attribute on a span, div or section keeps that part of the page out of snippets.¹² Google lists all three, with noindex, as the ways to limit what Search shows from your pages, AI features included.¹

<!-- No snippet, and no direct input to AI Overviews or AI Mode -->
<meta name="robots" content="nosnippet">

<!-- Limit snippets, and AI input, to 160 characters -->
<meta name="robots" content="max-snippet:160">

<!-- Keep one section out of snippets -->
<div data-nosnippet>Trade pricing for account holders</div>

There is a catch with max-snippet. Google says the limit does not apply where a publisher has separately granted permission, such as content supplied as in-page structured data or under a licence agreement with Google.¹² And nosnippet also removes the ordinary text snippet from blue links, which can cost clicks, so I would use it only on pages where that cost is acceptable.

These lines sit alongside your normal User-agent: * group. Remember that a crawler named in its own group stops reading the * group,⁵ so a group with only Disallow: / is safe, but a group that allows some paths must repeat your shared disallows.

# Goal 1: stay in Search and AI Overviews, keep content out of
# Gemini training and Gemini Apps grounding
User-agent: Google-Extended
Disallow: /

# Optional: also opt out of Google's generic research crawler.
# No documented effect on any Google product.
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
Disallow: /

# Do NOT block Googlebot to leave AI Overviews or AI Mode.
# That removes you from Google Search. Use the Search generative
# AI control in Search Console, or nosnippet, instead.

# Everyone else, including Googlebot, follows your normal rules
User-agent: *
Disallow: /checkout/
Disallow: /account/

Be careful if you ever give Googlebot a group of its own. Several Google crawlers answer to Googlebot as a second token, including Googlebot-Image, Googlebot-News, Google-InspectionTool and Google-CloudVertexBot, and a rule needs to match only one of a crawler's tokens to apply.³ A restrictive Googlebot group therefore reaches all of them. Apple's crawler also falls back to Googlebot rules, as explained in Apple's AI crawlers.

Cloudflare customers should also check the dashboard. Cloudflare classes Googlebot as a mixed-use crawler, serving both search and training, and from 15 September 2026 its Block setting for training crawlers applies to Googlebot too, stopping it entirely, search included. Its Disallow AI Training setting instead publishes the no-training preference in robots.txt, which for Google is the Google-Extended opt-out, and leaves Googlebot crawling for search.¹³ More on this in when your CDN blocks AI crawlers.

User agent strings can be spoofed, so check before you trust one. Google documents two methods: a DNS check for one-off lookups, and matching IP addresses against its published ranges for anything at scale.²

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Google still returns far more traffic per page crawled than the AI-only operators. Cloudflare's crawl-to-refer ratio, the number of pages a platform crawls for each visit it refers, put Google at 5.4 in July 2025, against 1,091 for OpenAI and over 38,000 for Anthropic. Google's ratio rose from 3.8 in January to a peak of 22.5 in April before falling back.¹⁴ In the same study, Google referrals to Cloudflare's news customers fell from February 2025, with April 15% below January, which Cloudflare linked in time to the expansion of AI Overviews and the arrival of AI Mode.¹⁴

That is the trade-off in numbers. Google crawls a lot, but it sends visitors back at a rate the AI-only operators do not, and its AI features are built on the same crawl.

• Leave Googlebot alone. Every AI feature in Google Search depends on it, and so does the rest of your Google traffic.
• Decide on Google-Extended on its own merits. If being cited in the Gemini app matters to you, leave it allowed. If keeping content out of Gemini training matters more, disallow it and accept that Gemini Apps will not use your pages for grounding.
• Block GoogleOther if you like; it costs nothing documented. Ignore Google-CloudVertexBot unless you build on Vertex AI.
• Use the Search generative AI control only if you are prepared to lose AI Overviews and AI Mode traffic, and check the Generative AI performance report first. For most businesses that traffic is worth keeping, and data-nosnippet on the few sections you want withheld is the more precise tool.
• Verify Google traffic by DNS or IP before acting on it, and after any CDN or firewall change, confirm Googlebot still gets 200 responses.