Staging sites, test pages and other things that get indexed by accident

Search indexes hold plenty of pages nobody meant to publish: staging copies of whole sites, preview deployments, test pages, internal search results, attachment pages and PDFs written for one client. Each is a URL that a crawler could reach and nobody locked. The same open URLs are there for AI crawlers and assistants to read as well.

This post covers where they come from, how they get found and how to get them out. The general rules for robots.txt, meta robots and canonicals are in the technical SEO guide; here I stick to what matters once something has leaked.

Most of it falls into a few groups.

• Staging and development hosts: staging., dev. and test. subdomains, or a copy of the site on a host's temporary domain. These are full duplicates of the live site, often with unfinished content and test prices.
• Preview deployments. Platforms such as Vercel and Cloudflare Pages give each branch or pull request its own URL,¹ and Cloudflare's preview URLs are public by default.²
• Internal search results, where any query a visitor or a spammer types can become a crawlable page.³
• Test and placeholder pages: the page someone published to check a template, or a service page still full of lorem ipsum. On WordPress, anything published in a public post type is also listed in the core sitemap at /wp-sitemap.xml.⁴
• CMS by-products, of which WordPress attachment pages are the best known.
• PDFs and other uploads. Google indexes and ranks PDFs like ordinary pages.⁵
• Parameter URLs: tracking tags, sort orders, filters and session IDs that create extra copies of real pages, covered in which faceted navigation pages to index.

Preview deployments: what the platforms do by default

Vercel, Netlify and Cloudflare Pages all add an X-Robots-Tag: noindex header to preview deployments by default, and each leaves gaps.

Netlify is frank about the limits: it says the header is no guarantee that previews stay out of search results.⁸ On Vercel, new projects have had protection switched on by default since November 2023, so check older ones.⁷ The Vercel gap I would check first is a staging subdomain on a preview branch, which is one of the staging setups in Vercel's own documentation¹ and gets no noindex.⁶

Internal search pages

Internal search is the leak other people can cause for you. On Google's Search Off the Record podcast in July 2026, John Mueller described spammers finding sites with open search pages and linking to search URLs filled with pharmaceutical or adult terms and a phone number, so the spam shows up in search under someone else's domain. Google may flag that as hacked in Search Console, but he said it is better fixed at the source, with a broad robots.txt disallow or a noindex. He also warned against serving a 500 to keep bots away, because Google may then reduce crawling across the whole site.³

WordPress attachment pages

Sites that started on WordPress before version 6.4 give uploaded files their own attachment pages. New installations since 6.4 redirect those URLs to the file, but upgraded sites keep them, and there is no admin setting: the wp_attachment_pages_enabled option has to be changed directly, for example with WP-CLI.⁹ Johannes Siipola found that the first 6.4 releases only redirected logged-in users, so search engines still saw the pages, and reports that this was fixed in 6.4.3 in January 2024.¹⁰ Check it logged out.

PDFs and uploads

In an October 2026 episode of the same podcast, Mueller and Martin Splitt explained that Google converts indexable PDFs to HTML, ranks them like other documents and can show them in AI Overviews and AI Mode. Google also does not recrawl PDFs often, because its systems assume they will not change for a long time.⁵ A proposal uploaded for one client can sit in the index long after anyone remembers it. For a PDF that has to stay online, the noindex goes in an X-Robots-Tag HTTP header.¹¹

Google's guide to how Search works describes finding new pages through links from pages it already knows and through sitemaps.¹² A preview link pasted into a public forum or issue tracker is a link. Vercel preview links typically appear in pull request comments,¹ so on a public repository the link is published with the build. A sitemap generated while the site address still pointed at staging lists staging URLs.

robots.txt can give a path away as well. RFC 9309 warns that listing paths in the file exposes them publicly.¹³ Guessable hostnames do the rest. Barry Adams has written about coming across a competitor's staging site in Google's results, and warned that unprotected staging sites can be found with a simple site: search.¹⁴

Subdomains also leak through TLS certificates. Certificate Transparency, specified in RFC 6962, records certificates in public, append-only logs that anyone can audit and monitor.¹⁵ Certificates carry the hostnames they cover, so the logs make those names public.¹⁶ Let's Encrypt warns new certificate holders to expect visits from bots that watch the logs.¹⁷ In a 2018 study led by Quirin Scheitle at the Technical University of Munich, 11 random subdomains that existed only in certificate logs drew their first DNS lookups between 73 seconds and about three minutes after logging, and nine of the 11 were requested over HTTP or HTTPS within about two hours.¹⁶

Assistants add another route. If someone pastes a staging link into an assistant and asks about it, the fetch is made on that person's behalf. OpenAI says robots.txt rules may not apply to ChatGPT-User for that reason,¹⁸ and Perplexity and Google both say their user-triggered fetchers generally ignore robots.txt.¹⁹²⁰ A fetch like that does not put the page in a search index, but the assistant has still read it.

robots.txt asks crawlers not to fetch a URL. RFC 9309, the standard that defines it, says its rules are not a form of access authorisation and that content needing protection should use a real security measure, giving HTTP authentication as the example.¹³ It does not keep a URL out of the index either: if other pages link to a blocked URL, Google can still list it in results, without a description.²¹

noindex has the opposite weakness. It only keeps a page out of results if the crawler can fetch the page and read the rule, so a noindex on a page blocked by robots.txt is never seen.¹¹ Bing says the same: Bingbot has to be able to crawl a URL to pick up a noindex or a removal.²² For less important pages, Google warns it can take months before Googlebot revisits and the page drops out.¹¹ A noindex also leaves the page open to anyone who requests it.

Both are also settings inside the site. In a Webmaster Hangout reported in September 2019, John Mueller called server-side authentication generally the best approach for staging, partly because you then do not have to change robots.txt or noindex on the site itself.²³ I think that second point is the strongest argument for it.

Access control works against every crawler. With HTTP authentication, the server answers a request that has no credentials with a 401 status and a WWW-Authenticate challenge, and only serves the page once valid credentials are sent. Basic authentication encodes credentials without encrypting them, so it should only ever run over HTTPS.²⁴ An IP allowlist does the same job for people on known office or VPN addresses, and nginx can combine the two so that a request gets in if it passes either check.²⁵

# Create the password file first (htpasswd comes with apache2-utils or httpd-tools):
#   sudo htpasswd -c /etc/nginx/.htpasswd yourname

server {
    server_name staging.example.com;
    # ...your existing listen, TLS certificate and root settings...

    location / {
        # Office or VPN range gets in directly; everyone else needs the password
        satisfy any;
        allow 203.0.113.0/24;
        deny  all;

        auth_basic           "Staging";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }

    # Second line of defence if the lock is ever removed. It lives in the
    # staging server config, so it does not travel with the site code.
    add_header X-Robots-Tag "noindex" always;
}

On the hosting platforms, use the built-in version. Vercel Authentication needs no paid add-on, while Password Protection costs extra on Pro and is included on Enterprise,²⁶ Netlify offers Password Protection or a private project,⁸ and Cloudflare Pages can put an Access policy on preview deployments.² Anyone who needs to review the work gets a login.

This is also the only control that holds against every AI crawler. The well-behaved ones read robots.txt,¹⁸ user-triggered fetchers generally do not, and a crawler that ignores robots.txt is not bound by it.¹³ A 401 stops all of them. Which AI crawlers to allow on the live site is a separate decision, covered in how to set up robots.txt for AI crawlers, and bot rules at the edge in when your CDN blocks AI crawlers.

Removal is two jobs: hide the URL now, then make a change that keeps it out after the hiding expires.

Google's Removals tool in Search Console hides a single URL, or every URL under a prefix, for about six months. A request covers the http and https and the www and non-www versions, but not other subdomains, and the URL must be in a property you own, which a Domain property takes care of.³⁰ Google is clear that the tool alone will not work: the lasting fix is a 404 or 410, a password, or a noindex.³⁰

Bing's Block URLs tool hides a page or a directory from Bing search, and from Copilot experiences that rely on Bing's index, for 90 days, and a block can be extended by another 90. It does not stop crawling, and the permanent fixes Bing lists are a 404 or 410, a noindex or a redirect.²²

For staging, the lasting fix is the lock from the section above. Google treats every 4xx response other than 429 the same way and removes indexed URLs that return one,³¹ so once staging answers with a 401, its URLs drop out of Google's index as they are recrawled. Bing's Block URLs guidance does not mention 401, so I keep the Bing block in place and renew it until the URLs have gone.

For leftovers on the live site, match the fix to the URL:

• Test and placeholder pages: delete them and let them return a 404 or 410. Google treats the two the same.³¹
• Pages that must stay up but out of search: a noindex, with the page left crawlable so the rule can be read.¹¹
• PDFs: remove the file or send X-Robots-Tag: noindex, and use the Removals tool while you wait, because Google does not recrawl PDFs often.⁵
• Attachment pages: set wp_attachment_pages_enabled to 0 so they redirect to the file.⁹
• Internal search: a broad robots.txt disallow or a noindex, never a 500. Mueller calls noindex almost cleaner, because a URL blocked by robots.txt can in theory still be shown.³ For spam pages that are already indexed, I would start there.

The same settings cause the opposite problem when a site goes live. Ticking "Discourage search engines from indexing this site" in WordPress adds a noindex, nofollow robots meta tag to every page,³² and it switches off the core sitemaps too.⁴ WordPress notes that the setting does not block access and that it is up to search engines to honour it, and like the other settings it is saved to the database.³² A staging database copied to live brings the setting with it, and a staging robots.txt with Disallow: / does the same at the crawl level.

That is why I prefer to lock staging at the server and leave the site's own robots settings as they will be on launch day. There is then nothing inside the site to forget. If a launch has already gone out with a staging setting, the day-one checks are in what to check in the first month after a site migration.

For most sites I would put the hostname check on a quarterly list. A Domain property and a CT search take a few minutes, and they are the quickest way I know to find hosts nobody remembers creating.