Cloudflare announced on September 15, 2026, that it is launching a new feature called "Disallow AI Training" (Disallow: directive in robots.txt). This setting allows website owners to prevent AI crawlers from using their content for training while still allowing search engines to crawl and index their sites.
Cloudflare Launches "Disallow AI Training" to Let Sites Block AI Scraping While Keeping Search
The feature addresses the challenge posed by "mixed-use crawlers"—single bots used by companies like Apple, Google, and Microsoft for both web search and AI training. Previously, blocking such crawlers to protect content from AI training often meant the same bots would be blocked from search engines, causing the website to lose visibility in search results.
Apple and Google already support this setting. Microsoft has committed to implementing similar capabilities by early 2027. Other major AI operators, including OpenAI, Anthropic, Meta, and Amazon, use separate crawlers for search and training, allowing Cloudflare to block only the training-specific bots without impacting search discoverability.
Sources
- AI学習は拒否、検索クロールは維持……Cloudflareの新機能「AI学習の不許可」 Google、Apple、Microsoftが対応 （ITmedia AI+, 2026-09-17）
- Cloudflareのブログ記事