AI search: how assistants find, read and cite websites, and what it changes about SEO

AI search is the name that has stuck for answers written by a language model on top of a search system: Google AI Overviews and AI Mode, ChatGPT search, Perplexity, Microsoft Copilot, Claude with web search, and others. Someone types or speaks a prompt, the system decides whether it needs the web, fetches and reads pages, and returns prose with a handful of links. This guide covers the mechanics, what each provider has documented, and where I think the work sits for different kinds of business.

This part of search attracts the most confident claims and has the least evidence behind them. I tie each claim to the provider's own documentation, to independent studies or to published research where they exist, and mark general practice and my own judgement where it does not. The products change monthly, so read every dated statement as true in October 2026.

Your customers use AI search in three ways. They ask it things, they tell it to do things, and increasingly they hand it whole tasks to finish. Each of those prompts pulls in different pages from your site, and most content plans are only built for the first.

A question can be answered from a good explainer. An instruction such as “compare these three CRMs in a table” cannot: the system needs each vendor's pricing, plan limits and integration list, and it will take them from wherever they are stated most clearly, which may be a review site and not the vendor. A task needs the stock level, the returns window and the checkout to be machine-readable. In Google's own words, AI Mode suits queries needing “further exploration, reasoning, or complex comparisons”, and people can ask nuanced questions that might previously have taken several searches.¹ OpenAI documents ChatGPT finding restaurant availability and, depending on the reservation provider, completing the booking flow inside the chat.²

Being “in the answer” can mean being cited as a source, being named as an option, or being the page whose data an agent uses to act. They overlap, and all of them start with retrieval.

A model either already knows something because it was in its training data, or it looks it up while answering. The research literature calls these parametric and non-parametric memory. The paper that named retrieval-augmented generation described combining a model's built-in knowledge with a separate searchable index of documents, and found the combination produced more specific and factual answers than the model alone.³

Parametric knowledge is what the model absorbed when it was trained. It is frozen at a training cutoff, it cannot be updated for your business on request, and the influence of any single page on it is unknowable. Live retrieval happens at answer time: the system runs searches, fetches pages, and writes from what it read. For anything current, priced, local, or specific to one product, retrieval is where the answer comes from.

The two routes are fed by different crawlers. That clears up most of the confusion about robots.txt and AI. Blocking a training crawler changes what future models may learn from your site, and has no documented effect on whether today's assistants can find and cite you. Blocking a search crawler does the opposite.

Use the scale to set priorities. The inputs you control most are also the ones retrieval reads. The ones you control least, training data and third-party opinion, get most of the commentary.

The system searches first and writes second. For Google this is documented directly: retrieval-augmented generation, also called grounding, relies on its core Search ranking systems to retrieve relevant, up-to-date pages from the Search index, and the model then reviews specific information from those pages to generate a response with links to them.⁷ Microsoft's transparency note for Copilot describes the same idea in general terms: for conversations where people seek information, Copilot is grounded in web search results, centres its response on high-ranking content and links to it.⁸ Other providers describe pieces of the same pipeline. The steps below mix documented behaviour with general industry practice, and the text says which is which.

Understanding and rewriting the prompt

OpenAI says that when ChatGPT search uses partner search providers it typically rewrites the prompt into one or more targeted queries, and may send further, more specific queries after reviewing the first results.² It may add general location inferred from the IP address, and if memory is on, saved details such as dietary preferences can shape the rewritten query.² The term “query fan-out” came from Google at the launch of AI Mode, for issuing multiple related searches concurrently across subtopics and multiple data sources, then bringing the results together.⁹ In a later Google example, a prompt about a lawn full of weeds fans out into searches about herbicides, chemical-free removal and prevention.⁷

So the searches that decide whether you are retrieved are not the prompt the person typed. They are narrower, more literal sub-queries. You can see real examples of them: Bing Webmaster Tools now reports the “grounding queries” its AI used when retrieving content it later cited.¹⁰

Retrieving candidates from an index

Each provider documents this differently. Google retrieves from its own Search index using its core ranking systems.⁷ OpenAI says ChatGPT search sometimes partners with other search providers, and its help article points readers to Microsoft's and Shopify's privacy policies for how those providers process queries.² Perplexity says its public answer engine runs on its own index covering hundreds of billions of web pages.¹¹ Anthropic's help page for Claude's web search says Claude processes multiple sources and that every response includes citations, but does not name the index behind text results.¹² The detail is in the table further down this page.

Ranking, reranking and reading passages

A first-stage retriever is built to be fast and returns many candidates. Retrieval systems commonly add a second, more expensive ranking pass over a short list, and then pass only the best sections to the model, because a model's working context is limited and every extra page costs time and money. That two-stage pattern is general industry practice; no AI search provider has published its exact reranking. Perplexity has said the most about the passage part: its indexing divides documents into “fine-grained units” that are individually scored against the query, so relevant snippets come back already ranked.¹¹ Google's systems likewise review “specific information” from retrieved pages.⁷

Writing the answer and attaching citations

Providers describe this step only in outline. The model writes from the retrieved material and links to pages that support what it said. Google's models also identify further supporting pages while the response is being generated, which is why AI features can show a wider set of links than a classic results page.¹ OpenAI notes that search results and citations can be incomplete, outdated or wrong, and tells users to open the source and check it.² Citation is not proof that a page was the main input, and not being cited is not proof it was unused. It is the visible end of the pipeline.

A retrieval system usually works with sections of a page, so each section needs to make sense on its own. Perplexity's description of sub-document units is the clearest primary statement of this.¹¹ Research systems work the same way: the dense passage retrieval paper that much later work builds on split each Wikipedia article into disjoint 100-word passages, prefixed each one with the article title, and retrieved those passages as its basic units.¹³ The title prefix matters. A passage that has lost its context is harder to match, which is one reason to name the product, plan or place inside each section.

These writing rules are my reasoning from how passage retrieval works. No provider documents them as ranking rules.

• State the claim near its heading. A section titled “Pricing” that opens with the price is easy to match and quote. One that opens with brand story is not.
• Make each section self-contained. Name the product, plan or service in the section itself, so a passage does not depend on “this” or “as mentioned above”.
• Prefer specifics a sub-query can match: numbers, units, model names, certifications, place names, dates, limits.
• Use tables for comparable facts. A row such as “Business plan, £18 per user per month, SSO included” survives being lifted out of the page. Microsoft's advice to publishers on AI answers names clear headings, tables and FAQ sections for the same reason.¹⁰
• Cut filler. A sentence with no fact in it gives a system nothing to retrieve and nothing to cite.

The closest thing to a controlled test is academic. The GEO paper (Aggarwal and others, KDD 2024) rewrote source pages in different ways and measured how much of a generated answer drew on each. Adding citations to other sources, quotations or statistics gave a relative improvement of 30 to 40% on its main visibility measure, and keyword stuffing gave little to no improvement.¹⁴ The tests ran on GPT-3.5 fed with search results and on Perplexity, using a benchmark the authors built, so read it as evidence for the direction. It does not show that any current assistant behaves the same way.

Lexical and semantic matching

Older search matched words. Newer retrieval also matches meaning. This is general information-retrieval background; no provider documents its own mix. Lexical methods such as TF-IDF and BM25 score a passage by the query words it contains and how rare those words are. They are fast and precise when the words match, and blind to synonyms. Semantic, or dense, retrieval turns the query and each passage into embeddings, lists of numbers that place similar meanings close together, so “cancel my plan” can match a passage about “ending a subscription”. In 2020 the dense passage retrieval paper reported that a learned dense retriever beat a strong BM25 baseline by 9 to 19 percentage points on top-20 passage retrieval accuracy across open-domain question answering benchmarks.¹³ Production systems commonly combine both.

Google's AI systems understand synonyms and general meaning, so for Google you do not need to cover every long-tail variant of a phrase.⁷ But exact terms still matter where the exact term is the fact: a model number, a certification such as ISO 27001, an integration name, a fare class. Write the way your customers and their prompts do, and spell out the specific names.

Providers vary in how much they say about their index. The table records only what each has documented, and says so where something is undocumented.

Whichever assistant you care about, a page the system cannot fetch, or that its index does not hold, cannot be used. The answer is also assembled from several sources at once, so your site is one input among many. I have written up how an assistant decides which businesses to name as a working model of how that plays out.

Google's position is more direct than most commentary about it: the best practices for SEO remain relevant, and there are no additional requirements to appear in AI Overviews or AI Mode and no other special optimisations necessary.¹ To be eligible as a supporting link, a page must be indexed and eligible to be shown in Google Search with a snippet.¹ A newer guide to generative AI features, last updated in July 2026, adds one condition: the site must also be included in Search generative AI features in Search Console, which is the default.⁷

The recommended practices are ordinary ones:¹

• Allow crawling in robots.txt, and by any CDN or hosting infrastructure.
• Make content findable through internal links.
• Provide a good page experience.
• Make sure important content is available as text.
• Support text with high-quality images and video where it helps.
• Make sure structured data matches the visible text on the page.
• Keep Merchant Center and Business Profile information up to date.

The guide goes further than the AI features page on two points. Creating content for every variation of how people search, including fan-out queries, primarily to manipulate rankings or AI responses, violates Google's scaled content abuse policy.⁷ And it addresses the AEO and GEO labels head on: from Google Search's perspective, optimising for generative AI search is optimising for search, and so still SEO.⁷

I think Google is right here. If a site is not appearing in AI Overviews, the first places to look are whether it is indexed, whether its snippets are restricted, whether it has been excluded in Search Console, and whether it ranks for the narrower searches a fan-out is likely to run. A separate AI strategy is rarely the missing piece.

What independent data says about clicks

Google's AI features page makes one claim about traffic: “when people click from search results pages with AI Overviews, these clicks are higher quality (meaning, users are more likely to spend more time on the site)”.¹ It gives no figure for how many clicks there are. Independent studies have measured that, and they point the same way.

• Pew Research Center tracked the Google searches of 900 US adults in March 2025. On pages with an AI summary, people clicked a traditional result in 8% of visits, against 15% on pages without one, and clicked a link inside the summary in 1% of visits.¹⁵
• Ahrefs compared 300,000 keywords using aggregated Search Console data, desktop only. With December 2025 data, the presence of an AI Overview correlated with a 58% lower average click-through rate for the top-ranking page. Its April 2025 version of the study had found 34.5%.¹⁶
• Seer Interactive, an agency, measured 3,119 informational queries across 42 client organisations from June 2024 to September 2025. Organic click-through rate on queries with an AI Overview fell from 1.76% to 0.61%, and on queries without one from 2.74% to 1.62%. Brands cited in the AI Overview had a 35% higher organic click-through rate than brands not cited.¹⁷

The methods differ, so the percentages cannot be compared with each other, and Ahrefs sells SEO software while Seer sells SEO services. The cited-versus-uncited figure is a correlation; cited brands may differ in other ways. Google's quality claim and the volume data can both be true. For reporting, the practical point is that AI Overview impressions are a poor proxy for traffic, and being cited is the part worth working towards.

Most providers now split their crawlers by purpose, so you can make separate decisions about search visibility and model training. The table below follows each operator's own documentation.

OpenAI says each of its settings is independent, so a site can allow OAI-SearchBot and disallow GPTBot.⁴ Anthropic says its bots honour robots.txt, asks for rules to be set per subdomain, and advises against blocking by IP address, because that can stop it reading your robots.txt at all.⁵

My default for a business that wants to be recommended is to allow the search crawlers and the user-triggered fetchers. Whether to block training crawlers is a separate decision about how you feel about your content being used to train models. A publisher with a licensing position will reach a different answer from a SaaS company that wants its documentation known, and that is a commercial call more than an SEO one. Cloudflare's network data puts numbers on the trade. In July 2025 Anthropic crawled 38,065.7 pages for every visitor it referred, OpenAI 1,091.4 and Perplexity 194.8, against 5.4 for Google, and over the preceding 12 months 80% of AI crawling was for training, 18% for search and 2% for user actions.¹⁹ Those ratios moved a lot within 2025, so check Cloudflare Radar for current figures. I have set out worked robots.txt configurations for each goal.

JavaScript rendering: what AI crawlers can see

If your content only appears after JavaScript runs in a browser, most AI crawlers will not see it. The providers do not document their rendering. The best public evidence is a December 2024 log analysis by Vercel and MERJ, which found that none of the major AI crawlers it measured rendered JavaScript, including OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot and PerplexityBot. Google's Gemini and Applebot were the exceptions.²⁰ ChatGPT's and Claude's crawlers did fetch JavaScript files, around 11.5% and 23.8% of their requests respectively, but did not execute them.²⁰

Google itself can process JavaScript as long as it is not blocked, though JavaScript-heavy sites are generally more complex to work on.⁷ The safe position for every other assistant is server-rendered or statically generated HTML for anything you want quoted: prices, specifications, plan limits, policies, availability. A single-page app that shows a pricing table only after a client-side API call is, as far as the 2024 evidence goes, a blank page to most AI crawlers. The study is nearly two years old, so test your own logs before assuming it still holds for a given crawler.

There are now two documented sets of controls for AI Overviews and AI Mode. Neither is Google-Extended, the control people most often misunderstand. Google-Extended governs Gemini training and grounding outside Search, and does not affect Search inclusion.¹⁸

Page-level: snippet controls

Site-level: the Search generative AI control

Since 31 August 2026 every Search Console property has a Search generative AI control under Settings.²² Including the site is the default. Excluding it removes your links and content from AI Overviews, AI Mode and generative AI features in Discover, and stops content crawled from your site being used as input for those responses. A change generally takes a few days to apply, and some content takes longer because of caching.²² The control is not used as a ranking or inclusion signal for other parts of Search, it does not affect AI training (Google-Extended covers that), and your content may still be used to help Search understand language generally.²²

These are blunt instruments for most sites. nosnippet costs the ordinary snippet as well, and a site-wide exclusion gives up a surface where other sites' content will still appear, possibly similar to yours.²² I would use them only for a specific reason: paywalled or licensed content, or a block of text you do not want quoted out of context, where data-nosnippet on that block is the proportionate choice.

Assistants are starting to act as well as answer, and for shops that means product data is read by machines that may complete the purchase. Everything in this section is dated, because it is changing quickly.

ChatGPT

OpenAI's shopping help article says that when a prompt suggests shopping intent, ChatGPT shows products with images, details and links to merchants, and that product results are selected independently and are not ads.²³ It lists what ChatGPT considers: structured metadata such as price and description from first-party and third-party providers, other third-party content, and the responses the model generated before it looked at new search results.²³ When several merchants sell the same item, they are ranked on factors such as availability, price, quality and whether the merchant is the maker or primary seller.²³ Merchants on Shopify are already integrated through Shopify Catalog, and other merchants can apply to send OpenAI a direct product feed.²³

In September 2025 OpenAI launched Instant Checkout, built on the Agentic Commerce Protocol it co-developed with Stripe and released as an open standard. The merchant stays the merchant of record and handles payment, fulfilment and returns in its own systems.²⁴ OpenAI said Instant Checkout items are not preferred in product results, but that whether Instant Checkout is enabled is one factor when ranking merchants selling the same product.²⁴ In March 2026 OpenAI said the first version of Instant Checkout lacked the flexibility it wanted, and that merchants can use their own checkout while it concentrates on product discovery. I have set out what that changes for product data.

Google AI Mode and Gemini

Shopping in AI Mode is powered by Google's Shopping Graph, which Google puts at more than 50 billion product listings with 2 billion updated every hour, and AI Mode can show comparison tables with insights drawn from reviews.²⁵ In November 2025 it began rolling out agentic checkout in the US: a shopper tracks a price, and when it falls within budget Google can buy the item on an eligible merchant's site with Google Pay, after the shopper confirms.²⁵ In January 2026 Google announced the Universal Commerce Protocol, an open standard for agentic checkout in AI Mode and the Gemini app with the retailer as seller of record, along with new Merchant Center attributes for things like answers to common product questions, compatible accessories and substitutes.²⁶ Browser agents may read a site through screenshots, the DOM and the accessibility tree.⁷

My view is that for a shop the product feed is now a search asset in its own right, and it has to agree with the product page. Price, availability, variants, shipping and returns need to be accurate in the feed, in the structured data and in the visible text, because each surface may read a different one. An agent that finds a price mismatch or an out-of-stock variant will move on. The ecommerce SEO pillar covers feeds and product data in more depth. For services and SaaS, the equivalent is a clean, crawlable signup or booking path and accessible forms, because a browser agent works from the same page a person sees.

Measurement is still partial, but 2026 brought the first proper first-party reports. Each source answers a different question.

Google Search Console

Search Console now has a Generative AI performance report, rolled out to all sites on 31 August 2026. It shows impressions in AI Overviews and AI Mode, grouped by page, country, date and device, with a filter for text and image-based searches.²⁷ It reports impressions; Google's report documentation does not list clicks or queries as metrics. AI feature traffic also stays inside the main Performance report under the Web search type.¹ The counting rules matter when you read it: an AI Overview occupies one position and every link in it gets that position, a click on an external link in an AI Overview or AI Mode counts as a click, and a follow-up prompt in AI Mode is counted as a new query.²⁸

Bing Webmaster Tools

Microsoft added an AI Performance report in February 2026, in public preview. It shows how often your pages are cited in AI answers across Copilot, Bing's AI summaries and select partner integrations, which pages are cited, and the grounding queries behind those citations.¹⁰ The grounding queries are the most useful part for anyone working on content, because they show the narrower searches an AI system ran. Check it even if Bing is a small share of your traffic.

Analytics referrals

Clicks from ChatGPT, Perplexity, Copilot, Claude and Gemini usually arrive as referrals from the assistant's domain, so a channel grouping for those referrers is the first thing to set up. It misses a lot. Clicks from AI Overviews and AI Mode arrive as ordinary Google organic traffic, some apps strip the referrer, and a recommendation that leads to a branded search or a phone call days later leaves no referral at all. Treat the referral figure as a floor.

Server logs

Logs are the only direct evidence of whether these crawlers reach your pages and what status codes they get. Filter by the user agents in the crawler table. User agent strings are easy to fake, so verify anything important against the IP lists the operators publish: Anthropic links to its list from its crawler page.⁵ A search crawler that only ever receives 403s or redirects is a problem you can fix this week.

Sampling prompts

Answers vary between runs, models, users and places. AI Mode and AI Overviews may use different models and techniques, so the responses and links they show will vary,¹ and OpenAI says location and saved memories can change the queries ChatGPT runs.² That variation also differs by country and language, which matters for anyone running international SEO. A single check tells you very little. My own method, with no documented standard behind it, is to write a fixed set of prompts that mirror your buyers' questions, instructions and tasks, run each several times per assistant on a regular schedule, and record how often you are named or cited. Report the share across runs and watch the trend over months. I have written up how to build and sample the prompt set in more detail. Be wary of any tool that reports a single visibility score without publishing its prompts and sampling method; no third-party tool has access to Google's internal ranking or AI systems.⁷

These priorities are my judgement, based on the mechanics above. No provider documents them.

The SaaS SEO pillar covers pricing, docs and comparison pages in detail. For local businesses the work is existing local SEO done more strictly, and I have set out what AI search changes for a local business separately.

Across all of them, consistency is the thread. An answer assembled from several sources is safer for the system when those sources agree. My working assumption is that a product or business described the same way on its own site, its feeds and profiles, and on the review and comparison sites in its sector, is a lower-risk thing to name than one with conflicting details. That is my reasoning about how a careful system would behave, with no documentation behind it, and it is the same consistency that brand and local search have always rewarded.

This is the order I would follow on most sites. The early steps are cheap and catch most of the problems.