Cloudflare launched a "Disallow AI Training" setting on 15 September that appends a no-training preference to robots.txt while explicitly allowing Googlebot, Applebot, and Bingbot to continue crawling for search indexing. The setting uses Google-Extended tokens for Gemini training opt-out and Applebot-Extended for Apple AI, with Bing support forthcoming. A separate "Block All" option halts all three major crawlers entirely for publishers who prefer complete disengagement. The mechanism solves the previously binary choice: publishers could either accept AI training on their content or remove themselves from search indexing. Cloudflare's accompanying accountability framework requires crawler operators to honour robots.txt training opt-outs, provide opt-out mechanisms for AI-generated summaries, and assure explicitly that training disallowance will not damage traditional search rankings. For publishers in AI licensing negotiations, the tool establishes a content-protection baseline that does not sacrifice SEO visibility — and a CDN-layer enforcement mechanism that does not require Google's cooperation to implement.
'Google Zero': The Publishing Industry Reached Consensus at Digiday's Summit — Search Traffic Is Not Coming Back
Digiday Publishing Summit, 15–17 September: publishers stopped waiting for a search traffic rebound …