The distinction that matters
AI companies run more than one crawler, and they do different jobs.
Training crawlers collect content used to train models. GPTBot, ClaudeBot, Google-Extended and CCBot are in this group. Blocking them affects whether your content contributes to a future model. It does not decide whether you are cited in an answer today.
Search and browsing bots fetch pages at the moment a question is being answered. OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User are in this group. These are the ones that decide whether you can be cited.
Block the first group and you have made a decision about training data. Block the second and you have removed yourself from answers.
So what does blocking GPTBot actually do?
It stops OpenAI using your content for model training. That is a legitimate choice, and plenty of publishers make it deliberately.
What it does not do is remove you from ChatGPT’s answers. When ChatGPT searches the web to answer a question, it uses its search bots, and those obey their own rules in your robots.txt.
It also has no effect on your Google rankings. Google-Extended is a separate token from Googlebot, and blocking it does not affect traditional search indexing.
Where the confusion causes real damage
Two failure modes, and both are common.
Blocking everything and expecting citations
A blanket Disallow for every AI user-agent removes you from the answers as well as the training data. If the goal was to protect content from training while remaining visible, that is the opposite of what happened.
Unblocking training crawlers and expecting improvement
The more expensive mistake. Someone finds GPTBot blocked, unblocks it, waits, and nothing changes — because the thing preventing citation was never GPTBot. Weeks are lost, and the actual cause is still there.
How to check what you are actually blocking
Open your robots.txt and look for each user-agent separately. A rule for * applies to everything that does not have its own group, so a broad disallow can catch bots you never intended to block.
Check both groups by name rather than assuming. The two lists behave differently, and a single line can move you from visible to invisible.
If you would rather not read robots.txt by hand, our Site Check reports crawler-by-crawler access and separates training crawlers from citation crawlers. It is free and needs no card.
The full list, so you can check by name
Worth having in front of you, because the names are easy to confuse.
| Crawler | Operator | What it decides |
|---|---|---|
GPTBot | OpenAI | Model training |
OAI-SearchBot | OpenAI | Citation in ChatGPT search |
ChatGPT-User | OpenAI | Citation when browsing |
ClaudeBot | Anthropic | Model training |
Claude-SearchBot | Anthropic | Citation in Claude search |
Claude-User | Anthropic | Citation when browsing |
PerplexityBot | Perplexity | Citation |
Perplexity-User | Perplexity | Citation when browsing |
Google-Extended | AI training. Not Googlebot. | |
CCBot | Common Crawl | Dataset collection |
The rows in bold are the ones that decide whether you can appear in an answer. Everything else is a training decision.
What we do not claim
Allowing the citation crawlers does not mean you will be cited. It means you can be. Access is a precondition, not a cause — the page still has to answer the question better than the pages currently being used.
Crawler access is the first thing to check because it is binary and cheap to fix. It is rarely the whole story.
Common questions
Will blocking GPTBot hurt my Google rankings?
No. Google-Extended is a separate token from Googlebot. Blocking it affects Google's AI training, not traditional search indexing.
If I unblock GPTBot, will ChatGPT start citing me?
Not on its own. GPTBot is a training crawler; citation runs through OAI-SearchBot and ChatGPT-User. If those were already allowed, unblocking GPTBot changes nothing about whether you appear in answers.
Should I block AI training crawlers?
That is a business decision rather than a technical one. Blocking them protects your content from training use, and it does not remove you from AI answers - provided you leave the search and browsing bots allowed.
Which crawlers should I definitely allow?
If you want to be citable: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Those fetch pages at answer time.