Blocking AI crawlers does not take you out of AI. It usually takes you out of the one part you were never worried about, and leaves the part you actually care about exactly where it was.
Answer the question one robot at a time. OpenAI, Anthropic and Perplexity each publish separate robots.txt tokens for training, for search indexing, and for live fetches triggered by a user, and each vendor states that the settings are independent of one another. Only the search token decides whether your business can be surfaced inside an answer.
So the SEO plugin that added a GPTBot block in 2024 made a training decision on your behalf. It did not make a visibility decision. And the site owner who allowed everything and then waited to get recommended never made a decision at all.
I pay attention to this because an AI assistant sent me a paying client this year with no ad and no Google click, and the first thing I did was work out which surface it came through. I wrote up the mechanism at the time. Crawler access is now the first check I run on any AI search optimization build, because it is the only layer here that is a settings problem rather than a content problem.
Blocking a training crawler is a copyright decision. Allowing a search robot is a distribution decision. Most sites made the first one by accident and never got around to making the second one at all.
The mistake: treating AI crawlers as one switch
Almost every robots.txt I open has one of two shapes. It names GPTBot and stops, or it names nothing at all. Both were written as though the phrase AI crawler pointed at a single thing.
It points at three, and the vendors are unusually clear about it. OpenAI's crawler documentation says a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot, and it spells out what the search setting costs you: sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot is described separately as the crawler for content that may be used in training. Two tokens, two unrelated outcomes, one phrase in the trade press covering both.
"We blocked the AI bots, so we are out of AI."
The training crawler is the only one most blocklists ever named. The search robot that decides whether an assistant can show your business carried on doing whatever it was already doing, and nobody checked which.
Sort every token by the job it does rather than the vendor that ships it. Training, search index, live fetch. Only the middle one is a visibility decision, and all three vendors document it as a separate setting.
"Which of these robots decides whether I appear in the answer?"
Every vendor ships a robot per job
Sort the tokens by job instead of by vendor and the decision gets simple. Training is GPTBot at OpenAI, ClaudeBot at Anthropic, Google-Extended at Google. Search indexing is OAI-SearchBot, Claude-SearchBot and PerplexityBot. Live fetching while a person waits for an answer is ChatGPT-User, Claude-User and Perplexity-User.
Anthropic states the consequence of each one in plain terms on its own help page, including that disabling Claude-SearchBot prevents indexing of your content for search, which may reduce your visibility in user search results. Perplexity draws the same line in a single sentence: PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for AI foundation models. If your objection to AI is about training, PerplexityBot was never the bot you meant.
Google is the odd one, and the reason is worth knowing. Google-Extended has no user agent string of its own. Google's documentation says the crawling is done with existing Google user agent strings and the robots.txt token is used in a control capacity, so it will never show up in your server logs no matter how carefully you grep them. It governs training for Gemini models and grounding, which is Google handing content from its Search index to the model at prompt time. And Google states plainly that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Google Search.
The fetch that happens mid-question barely reads your robots.txt
Here is the part that should change how you think about the whole exercise. The moment that matters commercially is not the crawl. It is the live fetch, the one that happens while a buyer sits there having just typed your category into an assistant. Those fetches are the least governed by robots.txt of the three.
OpenAI documents ChatGPT-User as the agent that may visit a page when a user asks a question, and says that because these actions are initiated by a user, robots.txt rules may not apply. Perplexity is blunter: since a user requested the fetch, this fetcher generally ignores robots.txt rules. Anthropic is the outlier in the useful direction, stating that disabling Claude-User prevents its system from retrieving your content in response to a user query.
Which means the file you edit is not the whole control surface. The rest of it is your firewall, and that is where most of the real blocking now happens quietly. Perplexity publishes step by step instructions for whitelisting its bots in Cloudflare WAF and AWS WAF, and a vendor does not write that page unless a lot of sites are dropping the requests without knowing. Anthropic warns that blocking its IP addresses may not work as an opt-out at all, because doing so stops them reading your robots.txt in the first place.
Read that last one twice. Block at the network layer and you can land in the worst version of both positions at once: no dependable opt-out, and no presence either.
Four checks to run on your own domain today
The bottom line
None of this makes an assistant recommend you. Crawler access is permission, not persuasion, and a page nobody would cite does not become citable because a robot can now reach it. But it is the only layer in this whole discipline where the failure is total, silent and free to fix.
There is a smaller irony worth sitting with. OpenAI and Perplexity both publish an llms.txt for their own documentation. Neither of them documents reading yours. That is a fair reminder to spend your effort on the parts the vendors actually write down and commit to.
The short version: allow the search robots, allow the user-triggered fetchers, and treat training as a separate business decision you make on purpose rather than one a plugin makes for you. Then go and look at your firewall, because your robots.txt may have spent two years telling the truth to nobody.
Book a 30 minute call: cal.com/zeeshanwaheed/30min or email [email protected]. I answer the first one myself, and you can see who you are talking to before you book.