Meta-Scan

AI-Friendly AEO & GEO & SEO Analyzer

Blog
ONLINE
Support
© 2026 Meta-Scan. All rights reserved.
BlogAboutPrivacy PolicyTerms of Service
SYSTEM ONLINE
Home/Blog

How to Check robots.txt and 5 Common Mistakes

Yuuuk94·Published 2026-11-12

Summary

robots.txt is a plain text file at the root of your site that tells crawlers which paths they may and may not fetch.

The most common accident is a Disallow: / added during development that survives into production. The most common misunderstanding is that robots.txt blocks crawling, not indexing.

01Basic syntax

robots.txt must live at the domain root, like https://yourdomain.com/robots.txt; in a subfolder it is ignored. Each subdomain (blog.yourdomain.com) and each protocol (http/https) has its own robots.txt.

User-agent: *
Disallow: /admin/
Allow: /admin/public/

Sitemap: https://yourdomain.com/sitemap.xml
DirectiveMeaningExample
User-agentWhich crawler the group applies to; * means allUser-agent: Googlebot
DisallowPath that should not be crawledDisallow: /admin/
AllowException inside a disallowed pathAllow: /admin/public/
SitemapFull URL of your sitemapSitemap: https://.../sitemap.xml

Paths are case-sensitive: Disallow: /Admin/ doesn't block /admin/. When rules overlap, Google follows the most specific (longest) matching rule.

02How to check your robots.txt now

  1. Type yourdomain.com/robots.txt into the address bar. A 404 means the file doesn't exist, in which case Google crawls without restrictions.
  2. In Search Console, open Settings > robots.txt report to see the version Google last fetched, when, and any errors.
  3. For a specific page, use URL Inspection and look at whether crawling is allowed.

Key point

A 404 for robots.txt is fine, but a server error such as 500 can make Google pause crawling the entire site. If the file exists, it must return 200.

03Five common mistakes

1. Disallow: / left over after launch

A Disallow: / meant to keep staging out of search ships to production. If you split robots.txt by environment variable, opening the live file right after each deploy is the most reliable safeguard.

2. Using robots.txt to keep pages out of the index

A blocked page can still appear in results as a bare URL if other sites link to it, because Google can't read its content. To remove a page from results, leave it crawlable and add noindex.

3. Blocking CSS and JavaScript folders

Blocking resource folders such as Disallow: /static/ stops Google rendering the page properly, which can lead to wrong conclusions about mobile-friendliness or content. Don't block resources.

4. A missing trailing slash

Disallow: /blog blocks /blog, /blog/, /blog-old and /blogger. To block just the folder, write Disallow: /blog/.

5. Relying on unsupported directives

Google doesn't support Crawl-delay, and a Noindex: line in robots.txt does nothing. For Google, the meaningful directives are User-agent, Disallow, Allow and Sitemap.

04What about AI crawlers?

AI companies run their own named crawlers, and most say they follow robots.txt. The important detail is that one company may run separate crawlers for training and for search and answers.

NameCompanyPurpose
GPTBotOpenAICollects data for model training
OAI-SearchBotOpenAISurfaces sites in ChatGPT search
ClaudeBotAnthropicCollects data for model training
PerplexityBotPerplexitySurfaces sites in Perplexity search
Google-ExtendedGoogleControls use for Gemini (no effect on Google Search)

If you want AI answers to cite you, at least don't block the search and answer crawlers. If you don't want to be used for training, block only the training crawlers.

# Block training, allow search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Crawler names and policies change, so confirm them in each company's documentation. A fuller list is in AI crawler settings. META_SCAN's AI signals card checks whether these rules block AI search visibility.

FAQ

Q. Is it a problem if I have no robots.txt?
No. Without one (404), Google assumes no restrictions and crawls everything. It is still worth having one to point to your sitemap.
Q. How quickly do robots.txt changes take effect?
Google generally caches robots.txt for up to about a day. If it is urgent, you can request a recrawl from the robots.txt report in Search Console.
Q. I only want to hide one page from search.
Add <meta name="robots" content="noindex"> to that page and don't block it in robots.txt. Google has to read the page to see the noindex.

Related reading

  • Site Not Showing Up on Google? 7 Things to Check
  • What Is llms.txt? How to Make One, What to Expect
  • AI Crawlers (GPTBot and Others): Allow or Block

References

  • Google Search Central: Introduction to robots.txt
  • Google Search Central: Create and submit a robots.txt file
  • Google Search Central: How Google interprets the robots.txt specification
  • RFC 9309: Robots Exclusion Protocol
  • OpenAI: Overview of OpenAI Crawlers

Author

Yuuuk94

Frontend developer who grew a product's MAU by 350% using SEO and Lighthouse as the yardstick, then turned that checklist into META_SCAN.

More about me →

How does your site do?

Enter one URL to run the free SEO/AEO/GEO checklist, including everything covered in this post.

Check for free