If your site runs on Cloudflare, the setting to check is the one for AI training. Since 15 September 2026, choosing Block for Training also stops Googlebot, Bingbot and Applebot, so the click meant to keep your pages out of AI models also shuts out the crawlers behind Google, Bing and Apple search. Disallow is the option Cloudflare built to refuse training and keep search. All three AI bot controls come with every Cloudflare plan, Free included, and Cloudflare says 17% of its sites block training in some way, while fewer than 1% block search.
Advice written before that date describes a different switch. Until then, the one-click Block AI bots option skipped crawlers that do search and training at once, so it never touched Googlebot. Block, as Cloudflare defines it now, does not skip them, and Cloudflare says more than 20% of web domains sit behind it.
On Cloudflare, Block for AI training now reaches Googlebot. Disallow is the word you want.
Key numbers
| Item | Number |
|---|---|
| Date Block for Training began covering Googlebot (Bingbot and Applebot too) | 15 September 2026 |
| Crawlers that Block now stops for search as well (Googlebot, Bingbot and Applebot) | 3 |
| Cloudflare sites that block AI training in some way (by Cloudflare’s count) | 17% |
| Cloudflare sites that block search bots | Under 1% |
| Web domains behind Cloudflare (by Cloudflare’s count) | More than 20% |
| Growth in daily AI agent requests on Cloudflare in a year | More than 1,700% |
| AI bot controls per domain (Search, Agent and Training) | 3 |
| Options for Training (Block, Block on pages with ads, Disallow, Allow) | 4 |
| Microsoft target for robots.txt no-training support (until then, NOARCHIVE for Bing) | Early 2027 |
| Time for a robots.txt change to reach OpenAI | About 24 hours |
| imisofts.com robots.txt as Cloudflare serves it (byte for byte the file we wrote) | 673 bytes |
Read on 22 October 2026 from Cloudflare’s blog posts of 1 July, 15 September and 30 September 2026, Cloudflare’s robots.txt and AI Crawl Control documentation, OpenAI’s crawler documentation, Google’s common crawlers page and the imisofts.com Cloudflare dashboard.
What changed on 15 September
Cloudflare gives every domain three AI controls, one per behaviour: Search, for crawlers that build an index; Agent, for bots acting for a person in real time, such as ChatGPT-User; and Training, for crawlers that take content to train models. It counts Googlebot, Bingbot and Applebot as mixed-use crawlers: one crawler that serves both search and AI training.
On 15 September 2026, Cloudflare made Block and Block on pages with ads apply to those mixed-use crawlers, and added Disallow. With Bot Preference Sync on, Disallow writes a no-training preference into your robots.txt and blocks every other training crawler, including the training-only ones run by Amazon, Anthropic, Meta and OpenAI, while Google, Microsoft and Apple keep crawling for search. Cloudflare’s advice for anyone who wants those three gone is blunt: select Block, and “It will stop Applebot, Bingbot, and Googlebot from reaching your site”.
If you switched on the old Block AI bots before then, Cloudflare did not cut Googlebot off for you. It migrated that setting to Allow for Search and Disallow for Training, and it moved earlier Training blocks to Disallow too. The risk is a choice made from now on, by someone following an older guide that says Block.
We do not want AI training on our content, so set Training to Block.
Since 15 September that also stops Googlebot, Bingbot and Applebot reaching the site.
Set Training to Disallow and turn on Bot Preference Sync.
Crawlers that only train are blocked. Google, Microsoft and Apple keep crawling for search.
Which setting decides if ChatGPT can read you
For a business that wants AI assistants to recommend it, the three controls do different jobs. Search covers the crawlers behind AI search, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, as well as Googlebot and Bingbot. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Leave Search on Allow.
Agent is the control that matters in the moment a buyer asks. It covers bots acting for a person in real time, such as ChatGPT-User, and browser agents; Cloudflare’s bot reference puts Claude-User and Perplexity-User in the same AI assistant category. OpenAI says ChatGPT-User may visit a page when someone asks ChatGPT a question, and that robots.txt rules may not apply because a user started it. A Cloudflare block is enforced at the network, not requested in a file, so on Cloudflare the Agent setting can stop an assistant opening your page whatever your robots.txt says. Cloudflare says daily requests from AI agents on its network grew by more than 1,700% in a year.
Training is a rights decision more than a traffic one. If you object, choose Disallow. Google says Google-Extended, the token Disallow uses for Google, does not affect inclusion in Google Search and is not a ranking signal. Bing is the gap. Cloudflare says Microsoft is building support for a robots.txt no-training preference, targeted for early 2027, and until then Disallow does not pass that preference to Bing, so for Bing it points to the NOARCHIVE meta tag.
What I found on our own zone
This site runs on Cloudflare’s Free plan. I opened the zone’s Security settings on 22 October 2026: Search, Agent and Training all read Allow (do not block), and Cloudflare marks Allow as the recommended choice on all three. The Training menu offers four options: Block, Block on pages with ads, Disallow and Allow. Bot Preference Sync is off, and the robots.txt Cloudflare serves is byte for byte the 673-byte file we wrote, which welcomes GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot by name.
We leave all three open on purpose. For a services business, being read by assistants is the point, and we made the training call deliberately, in that file.
If your site sells services, not ads
New domains since 15 September are offered one of two presets. A site that does not run ads starts with Search, Agent and Training on Allow. A site that monetizes pages with ads starts with Training on Disallow and Agent on Block on pages with ads, which applies only to pages Cloudflare detects serving an ad. Sites that had the old Block AI bots switch on were moved to that same Agent setting. On a services site with no ad slots, that setting leaves assistants free to fetch your pages. The option to avoid on an ad-funded site is Block on pages with ads under Training: it now covers Googlebot, Bingbot and Applebot on every page Cloudflare detects serving an ad.
How I would set it
I have delivered 500+ projects, and on any site behind Cloudflare I would make these four checks:
Read the three values first
In the dashboard, open your domain, go to Security, then Settings, filter by Bot traffic and select Configure AI bot policies. It takes a minute, on any plan.
Use Disallow, not Block, for training
If you object to training, set Training to Disallow and turn on Bot Preference Sync, as Cloudflare’s own steps say. Block now stops Googlebot, Bingbot and Applebot.
Leave Search and Agent on Allow
Search feeds the indexes AI assistants answer from. Agent lets ChatGPT-User and similar fetchers open your page when a question needs it.
Read the robots.txt Cloudflare serves
Open yourdomain.com/robots.txt after saving. With Preference Sync on, Cloudflare puts its block above your own rules, and OpenAI says a change can take about 24 hours to reach its systems. To keep Bing from training, add NOARCHIVE.
Those four checks are where our AI search optimization work starts on any site behind Cloudflare. The robots.txt layer underneath has its own traps, covered in which AI crawlers to block, and Google’s separate switch for AI Overviews sits in Search Console: see Search generative AI control.
The bottom line
If you want ChatGPT, Claude or Perplexity to be able to recommend you, keep Search and Agent on Allow. If you object to AI training, choose Disallow, which Apple, Google and Microsoft honor or have committed to honor, by Cloudflare’s account. Do not choose Block unless you mean to shut out the crawlers behind Google, Bing and Apple search, because since 15 September that is what it does.