How to check what crawlers see on a JavaScript site

A SaaS marketing site built with React can look perfect in a browser and arrive almost empty at a crawler. The browser runs the JavaScript and builds the page. Googlebot does the same, but later and in a separate step. Many AI crawlers do not run it at all, so they read only the HTML the server sent. The test that matters is what sits in the HTML before any script runs, and what changes after. This post shows how to run that test with curl and PowerShell, how to read Google's own view of the rendered page, and which rendering choices fix what you find.

The wider technical picture for software companies, from app subdomains to staging environments, is in the SaaS SEO guide. This post takes its section on JavaScript rendering and turns it into a method you can run in an afternoon.

JavaScript pages go through three phases at Google: crawling, rendering and indexing.¹ Googlebot fetches the URL, checks robots.txt first, and parses the HTML it gets back. The page then joins a render queue. It may stay there for a few seconds, or for longer.¹ Rendering happens in an evergreen version of Chromium, which executes the JavaScript, and the rendered HTML is what gets indexed.¹

You will still hear this called the two-wave model. At I/O in 2018, Google told developers that indexing and rendering do not happen at the same time, and that it may defer rendering to a later point.² SEOs summarised that as two waves: index the raw HTML first, then render and index again. Google's current documentation describes phases and a queue instead of waves. The practical lesson is the same either way. Anything that exists only after JavaScript runs depends on an extra step that happens later, and anything in the raw HTML does not.

What the Web Rendering Service does not do

The rendering component is the Web Rendering Service, or WRS. It does not keep state between page loads: local storage, session storage and cookies are cleared each time.³ So content that only appears after a visitor accepts a cookie banner, picks a region or logs in is unlikely to be in what Google renders. Three more rules from the same JavaScript guidance catch SaaS sites often:

• A noindex in the initial HTML may stop Google rendering the page at all, so JavaScript that removes it may never run.¹
• Set the canonical in the HTML. If JavaScript has to touch it, it should set the same value the HTML already has.¹
• Google only discovers links that are <a> elements with an href attribute.¹ A button with a click handler that changes the route is not a link to a crawler.

The canonical rule is one most sites already follow. HTTP Archive's 2025 Web Almanac found that in only around 2% of cases was a canonical missing from the raw HTML but present in the rendered DOM.⁴ If your site is in that group, the fix is usually a small template change.

Seeing the page as Google rendered it

The URL Inspection tool in Search Console and the Rich Results Test both show the loaded resources, JavaScript console output and the rendered DOM.³ In URL Inspection, run Test live URL and open View tested page. A screenshot of the rendered page is only available in a live test, and the live result can differ from the indexed version.⁵ Use the live test to check a fix, and the indexed result to see what Google stored.

The most detailed public evidence comes from a log analysis Vercel and MERJ published in December 2024. It found that none of the major AI crawlers it measured rendered JavaScript, naming OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot.⁶ ChatGPT's and Claude's crawlers did fetch JavaScript files, in 11.50% and 23.84% of their requests, but did not execute them.⁶ The exceptions were Gemini, which uses Googlebot's infrastructure, and Applebot, which renders pages in a browser-based crawler.⁶

Read that with its limits in mind. It is one analysis of traffic on one network, from late 2024, and crawler behaviour can change without notice. I still treat it as the working assumption for any page you want an assistant to read: if the copy is not in the server's HTML, ChatGPT search, Claude and Perplexity probably cannot see it. Vercel's own recommendation was to server-render important content for this reason.⁶ MDN's web docs list the same benefit for server-side rendering: search engines, social media crawlers and other bots can read the content without executing JavaScript, and while major search engines can run JavaScript, social media crawlers usually cannot.⁷

Rendering is the second gate. The first is whether you let these crawlers in at all, which is covered in how to set up robots.txt for AI crawlers. A crawler you allow but that receives an empty shell is no better off than one you block. For how that missing copy affects which products an assistant names, see how an assistant decides which businesses to name.

Illustrative. Rotawise is a fictional staff scheduling product. Its marketing site is a single-page React app: the server returns the same small HTML file for every route, and the browser fetches plan data from an API and draws the page. In a browser the pricing page shows three plans, a comparison table, an FAQ and links to integrations. This is what each version contains.

Google can read JSON-LD that a script adds to the DOM, because it processes the structured data present when it renders the page.⁸ A crawler that never renders gets none of it. If Rotawise moved its marketing routes to static generation, the raw HTML would carry the same content as the rendered DOM, and the only thing left to JavaScript would be the monthly and annual price toggle.

Start with a handful of URLs that matter: the homepage, pricing, one feature page, one integration page, one comparison page and one blog post. Fetch each one exactly as a non-rendering crawler would, save it, and look for the things that should be there. Pick a sentence from the visible copy for each page, such as a price line, so you can search for it.

Bash, on macOS, Linux or WSL

PowerShell, on Windows

These work in Windows PowerShell 5.1 and PowerShell 7. Fetch once and keep the response in a variable.

Run the fetch a second time with a crawler's user agent string in place of Mozilla/5.0, for example GPTBot. If the two responses differ, something on the server or CDN is treating bots differently, which is worth knowing before you trust either result. A spoofed user agent does not reproduce rules based on IP address, so a bot management layer can still treat the real crawler differently; server logs are the place to confirm that.

The raw HTML tells you what a non-rendering crawler gets. The rendered DOM tells you what a browser, and eventually Googlebot, builds from it. The gap between the two is the part of the page that depends on JavaScript.

• Headless Chrome can save the rendered DOM. In bash: google-chrome --headless --dump-dom https://www.example.com/pricing > rendered.html. In PowerShell: & "C:\Program Files\Google\Chrome\Application\chrome.exe" --headless --dump-dom "https://www.example.com/pricing" | Out-File rendered.html -Encoding utf8. Then run the same counts on rendered.html.
• To list links that exist only after rendering: diff <(grep -o 'href="[^"]*"' raw.html | sort -u) <(grep -o 'href="[^"]*"' rendered.html | sort -u)
• For a quick visual check, open DevTools, press Ctrl+Shift+P (Cmd+Shift+P on a Mac), and run Disable JavaScript. It stays off in that tab while DevTools is open.⁹ Reload and look at what is left.
• For Google's own view, use URL Inspection's live test, and the Rich Results Test for structured data. Prefer the URL input to pasting code, because the code input has JavaScript limitations.⁸

The fix for an empty raw HTML is to produce the HTML before it reaches the crawler. The vocabulary comes from web.dev: server-side rendering sends HTML from the server for each request, static rendering builds HTML for each URL at build time, client-side rendering builds the page in the browser with JavaScript, and hydration attaches client-side scripts to server-rendered HTML to make it interactive.¹⁰

In the Next.js App Router, layouts and pages are Server Components by default, and Client Components are also prerendered to HTML on the server for the first page load.¹¹ Marking a component 'use client' does not on its own hide it from crawlers. What does is data fetched in the browser after load, such as plans pulled from an API inside an effect, or content that waits for a browser-only API. That data is not in the prerendered HTML, so move the fetch into a Server Component.

Incremental Static Regeneration sits between static and server rendering. It updates static pages without rebuilding the whole site, using a revalidate time or on-demand revalidation.¹² Next.js documents an x-nextjs-cache response header with the values HIT, STALE, MISS and REVALIDATED,¹² so curl -sI https://www.example.com/pricing | grep -i x-nextjs-cache shows whether a page came from the cache, where your hosting passes the header through.

Avoid dynamic rendering for a new build. Google describes it as a workaround and not a long-term solution, and recommends server-side rendering, static rendering or hydration instead.¹³ If the site is a plain client-side React app, the usual fix is a framework or build step that prerenders the marketing routes. If that change also alters URLs, treat it as a website migration and plan the redirects before launch.

Rendering regressions arrive quietly. A developer moves a pricing table into a client component, a new head library sets the canonical at runtime, and nothing looks different in the browser. A short check on each release catches it before a crawler does.

The Reach skill in my open-source pack runs this check: it fetches real pages, reads the HTML served before any JavaScript runs, and moves content into the server response where it is missing. Whatever runs it, the habit matters more than the tool. Compare what the server sends with what the browser builds, and keep the gap small.