Cloudflare is rolling out significantly more granular controls for how website owners manage AI bot traffic, moving beyond the blanket-block approach it introduced in 2024. The new system, which went into effect on September 15, 2026, categorizes AI crawlers into three distinct groups—Search, Training, and Agent—and gives site owners the power to permit or deny access to each category independently. This update represents a major shift in the ongoing battle between content publishers and the automated systems that consume their data.
Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click. The new controls replace that blunt instrument with a more nuanced permissions model. Site owners can now allow search engine indexing to continue while blocking data collection for AI model training, or permit training crawlers while blocking bots that act on behalf of users, such as ChatGPT’s agent crawlers. This granularity allows publishers to preserve their presence in search results while protecting their content from being freely consumed as training data or accessed through AI-powered assistants.
Default Rules Favor Human-Focused Sites
Effective immediately, Cloudflare has set a significant default policy: Training and Agent bots are now blocked by default on any page that carries advertising. The logic is straightforward—advertising signals that a site intends to serve human visitors, and AI agents consuming that content undermine the publisher’s business model. Search crawlers, by contrast, remain permitted by default, acknowledging that search indexing drives human traffic that ads monetize.
This default rule applies across all Cloudflare plans, including the free tier, meaning every site owner benefits from the protection immediately without needing to configure anything. For multi-purpose crawlers like Googlebot, which performs both search indexing and AI training, Cloudflare applies whichever rule is strictest. If a site blocks training bots, Googlebot
