Cloudflare launched a "Disallow AI Training" setting on 15 September that appends a no-training preference to robots.txt while explicitly allowing Googlebot, Applebot, and Bingbot to continue crawling for search indexing. The setting uses Google-Extended tokens for Gemini training opt-out and Applebot-Extended for Apple AI, with Bing support forthcoming. A separate "Block All" option halts all three major crawlers entirely for publishers who prefer complete disengagement. The mechanism solves the previously binary choice: publishers could either accept AI training on their content or remove themselves from search indexing. Cloudflare's accompanying accountability framework requires crawler operators to honour robots.txt training opt-outs, provide opt-out mechanisms for AI-generated summaries, and assure explicitly that training disallowance will not damage traditional search rankings. For publishers in AI licensing negotiations, the tool establishes a content-protection baseline that does not sacrifice SEO visibility — and a CDN-layer enforcement mechanism that does not require Google's cooperation to implement.
Google's AI Content Pilot Is Paying Most Publishers Under 0.1% of Their Ad Revenue
The Information, 29 September: Google's ~100-publisher pilot for content used in Gemini, AI Overview…