How to Check robots.txt and 5 Common Mistakes
Yuuuk94·Published
Summary
robots.txt is a plain text file at the root of your site that tells crawlers which paths they may and may not fetch.
The most common accident is a Disallow: / added during development that survives into production. The most common misunderstanding is that robots.txt blocks crawling, not indexing.
01Basic syntax
robots.txt must live at the domain root, like https://yourdomain.com/robots.txt; in a subfolder it is ignored. Each subdomain (blog.yourdomain.com) and each protocol (http/https) has its own robots.txt.
User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://yourdomain.com/sitemap.xml| Directive | Meaning | Example |
|---|---|---|
| User-agent | Which crawler the group applies to; * means all | User-agent: Googlebot |
| Disallow | Path that should not be crawled | Disallow: /admin/ |
| Allow | Exception inside a disallowed path | Allow: /admin/public/ |
| Sitemap | Full URL of your sitemap | Sitemap: https://.../sitemap.xml |
Paths are case-sensitive: Disallow: /Admin/ doesn't block /admin/. When rules overlap, Google follows the most specific (longest) matching rule.
02How to check your robots.txt now
- Type
yourdomain.com/robots.txtinto the address bar. A 404 means the file doesn't exist, in which case Google crawls without restrictions. - In Search Console, open Settings > robots.txt report to see the version Google last fetched, when, and any errors.
- For a specific page, use URL Inspection and look at whether crawling is allowed.
03Five common mistakes
1. Disallow: / left over after launch
A Disallow: / meant to keep staging out of search ships to production. If you split robots.txt by environment variable, opening the live file right after each deploy is the most reliable safeguard.
2. Using robots.txt to keep pages out of the index
A blocked page can still appear in results as a bare URL if other sites link to it, because Google can't read its content. To remove a page from results, leave it crawlable and add noindex.
3. Blocking CSS and JavaScript folders
Blocking resource folders such as Disallow: /static/ stops Google rendering the page properly, which can lead to wrong conclusions about mobile-friendliness or content. Don't block resources.
4. A missing trailing slash
Disallow: /blog blocks /blog, /blog/, /blog-old and /blogger. To block just the folder, write Disallow: /blog/.
5. Relying on unsupported directives
Google doesn't support Crawl-delay, and a Noindex: line in robots.txt does nothing. For Google, the meaningful directives are User-agent, Disallow, Allow and Sitemap.
04What about AI crawlers?
AI companies run their own named crawlers, and most say they follow robots.txt. The important detail is that one company may run separate crawlers for training and for search and answers.
| Name | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collects data for model training |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search |
| ClaudeBot | Anthropic | Collects data for model training |
| PerplexityBot | Perplexity | Surfaces sites in Perplexity search |
| Google-Extended | Controls use for Gemini (no effect on Google Search) |
If you want AI answers to cite you, at least don't block the search and answer crawlers. If you don't want to be used for training, block only the training crawlers.
# Block training, allow search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Crawler names and policies change, so confirm them in each company's documentation. A fuller list is in AI crawler settings. META_SCAN's AI signals card checks whether these rules block AI search visibility.
FAQ
- Q. Is it a problem if I have no robots.txt?
- No. Without one (404), Google assumes no restrictions and crawls everything. It is still worth having one to point to your sitemap.
- Q. How quickly do robots.txt changes take effect?
- Google generally caches robots.txt for up to about a day. If it is urgent, you can request a recrawl from the robots.txt report in Search Console.
- Q. I only want to hide one page from search.
- Add
<meta name="robots" content="noindex">to that page and don't block it in robots.txt. Google has to read the page to see the noindex.
Related reading
References
How does your site do?
Enter one URL to run the free SEO/AEO/GEO checklist, including everything covered in this post.
Check for free