AI Crawlers (GPTBot and Others): Allow or Block
Yuuuk94·Published
Summary
AI companies run separate crawlers for separate jobs: model training, AI search and answers, and visits on a user's request. Knowing the split lets you say "don't train on me, but show me in search".
You configure this in robots.txt. robots.txt is a voluntary convention, though, so if you need hard blocking you have to do it at the server or CDN.
01The main AI crawlers
| User-agent | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collects data for model training |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search |
| ChatGPT-User | OpenAI | Visits when a ChatGPT user asks it to |
| ClaudeBot | Anthropic | Collects data for model training |
| Claude-SearchBot | Anthropic | Crawls to improve search results |
| Claude-User | Anthropic | Visits on a user's request |
| PerplexityBot | Perplexity | Surfaces sites in Perplexity search |
| Perplexity-User | Perplexity | Visits on a user's request |
| Google-Extended | Control token for Gemini training and grounding | |
| Applebot-Extended | Apple | Control token for Apple's generative model training |
| CCBot | Common Crawl | Public web archive (a source for many AI training sets) |
Google-Extended and Applebot-Extended aren't separate crawlers but robots.txt control tokens. Googlebot and Applebot do the fetching; the token only decides whether that data may be used for generative AI. Blocking Google-Extended therefore has no effect on Google Search visibility or ranking.
02Choosing a policy
| Goal | Training crawlers | Search crawlers |
|---|---|---|
| Maximum AI search visibility | Allow | Allow |
| Visible in search, no training | Block | Allow |
| Keep away from AI entirely | Block | Block |
For free tools and blogs that live on discoverability, like META_SCAN, the first or second option is typical. Paid content and publishers that care about copyright may consider blocking training. Blocking search crawlers too also removes the chance of being cited as a source in AI answers.
03robots.txt examples
No training, search allowed
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlBlock only some folders
User-agent: GPTBot
Disallow: /premium/
Disallow: /members/A crawler that finds a group matching its own name follows only that group and ignores User-agent: *. So shared rules in the * group (such as /admin/) must be repeated in each bot-specific group. This is easy to miss.
04When robots.txt isn't enough
- User-initiated agents: companies note that agents like ChatGPT-User, which fetch a page because a user asked, may treat robots.txt differently from automatic crawlers. Check the current official docs.
- Bots that ignore the rules: scrapers that hide their name or ignore robots.txt can only be stopped with a server firewall or CDN bot management.
- Impersonators: anyone can fake a user-agent string. OpenAI and others publish their crawler IP ranges, so match IPs when you analyze logs.
05Verify after you change it
- Open
/robots.txtin a browser to confirm the deployed content is what you intended. - Check server or CDN logs for each user-agent's requests and status codes. If an allowed bot gets 403s, a firewall is blocking it.
- Check the robots.txt report in Google Search Console for syntax errors.
- Scan your site with META_SCAN: the AI signals card shows whether the main AI search crawlers are blocked by robots.txt.
Crawler names are added and changed as company policies evolve. This list reflects the time of writing, so confirm the latest names in the official docs below before configuring.
FAQ
- Q. If I block GPTBot, is data already collected deleted?
- No. robots.txt only applies to future crawling. Data already collected is handled under each company's own policy.
- Q. Does blocking Google-Extended hurt my Google ranking?
- No. Google states that Google-Extended doesn't affect inclusion or ranking in Google Search. AI Overviews in Google Search use the regular search index and are separate from Google-Extended.
- Q. Can I block all AI bots with one line?
- robots.txt has no directive meaning "all AI bots"; you list them by name. For bulk blocking, a CDN's bot management feature is easier to maintain.
Related reading
References
How does your site do?
Enter one URL to run the free SEO/AEO/GEO checklist, including everything covered in this post.
Check for free