Meta-Scan

AI-Friendly AEO & GEO & SEO Analyzer

Blog
WARNING
Support
© 2026 Meta-Scan. All rights reserved.
BlogAboutPrivacy PolicyTerms of Service
SYSTEM WARNING
Home/Blog

How to Check for an Accidental noindex

Yuuuk94·Published 2026-11-12

Summary

noindex is an instruction: "don't include this page in search results". When Google finds it, the page is dropped from the index, even if it was indexed before, on the next crawl.

noindex can live in two places: an HTML meta tag and an HTTP response header. View-source doesn't show headers, so check both or you can miss the cause.

01The two places noindex hides

LocationFormHow to check
HTML meta tag<meta name="robots" content="noindex">View source (Ctrl+U) and search for noindex
HTTP response headerX-Robots-Tag: noindexDevTools Network tab > document request > response headers

The meta tag can target a specific crawler, as in name="googlebot", instead of name="robots". Search the source for the word noindex, not robots, so you catch these too.

Because X-Robots-Tag also works on non-HTML files like PDFs and images, it is often set in bulk in server or CDN configuration, which also makes it easy to apply to an entire site by accident.

02Check both from the terminal

Browser extensions and login state can change what you see, so a clean request like a crawler's is more reliable.

# Is there an X-Robots-Tag header?
curl -sI https://yourdomain.com/page | grep -i x-robots

# Is there a noindex in the HTML?
curl -s https://yourdomain.com/page | grep -io '<meta[^>]*noindex[^>]*>'

If neither prints anything, the server's response contains no noindex. JavaScript can still change meta tags later, so to be certain, run Search Console URL Inspection's "Test live URL" to see the page as Google rendered it.

03Common causes

WordPress's search engine visibility setting

If "Discourage search engines from indexing this site" under Settings > Reading is checked, every page gets noindex. Turning it on during the build and forgetting to turn it off at launch is the classic case.

Staging settings leak into production

If an environment variable switches noindex on for non-production builds, a wrong value in production removes the whole site. Add "search the home page source for noindex" to your release checklist.

Hosting preview deployments

Hosts such as Vercel automatically add X-Robots-Tag: noindex to preview deployment URLs. That is intended, but if your production domain is attached to a preview deployment, production gets noindex too. Make sure the domain is attached to the production deployment.

Inherited framework metadata

In frameworks like Next.js where layout metadata cascades to child pages, robots: { index: false } on a layout applies to everything beneath it. Trying to hide one thin page and hiding a whole section happens here.

04Why not combine it with robots.txt

Some sites block a page in robots.txt and add noindex to be extra sure, but that stops noindex from working. Google can't read a page blocked by robots.txt, so it never sees the noindex. Pages with external links can then linger in results as a bare URL with no description.

Key point

To remove a page from search results, use noindex only and keep it crawlable in robots.txt. Google has to read the page to see the noindex.

05Pages that should have noindex

noindex isn't only a source of accidents. Removing pages with no value as search results raises the average quality of what is indexed across your site.

  • Internal site search results
  • Login, sign-up and password reset screens
  • Cart, checkout and account pages
  • Tool screens that only mean something with input (META_SCAN's scan progress and result screens are an example)
  • Tag or category lists with almost no content

For these, noindex, follow keeps the page out of the index while still letting crawlers follow its links to discover other pages.

06After removing noindex

  1. Run the curl commands above against production and confirm both the tag and header are gone.
  2. In Search Console URL Inspection, run "Test live URL" and look for "Indexing allowed".
  3. Click "Request indexing". Requesting a few key pages is enough; the rest are picked up via the sitemap and links.
  4. Watch "Excluded by 'noindex' tag" in Indexing > Pages shrink over days to weeks.

FAQ

Q. Will the page come back immediately after I remove noindex?
Google has to recrawl it first. With an indexing request it usually happens within days, but if the whole site was removed it can take weeks.
Q. What is the difference between noindex and nofollow?
noindex means don't put this page in results; nofollow means don't follow the links on this page. They are independent, so use only what you need.
Q. Does Bing respect noindex too?
Yes. Bing supports the robots meta tag and X-Robots-Tag. Bing Webmaster Tools' URL Inspection shows how Bing sees the page.

Related reading

  • Site Not Showing Up on Google? 7 Things to Check
  • How to Check robots.txt and 5 Common Mistakes
  • Canonical Tags: How to Handle Duplicate URLs

References

  • Google Search Central: Block search indexing with noindex
  • Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
  • Search Console Help: URL Inspection tool
  • Vercel Docs: Environments

Author

Yuuuk94

Frontend developer who grew a product's MAU by 350% using SEO and Lighthouse as the yardstick, then turned that checklist into META_SCAN.

More about me →

How does your site do?

Enter one URL to run the free SEO/AEO/GEO checklist, including everything covered in this post.

Check for free