Cloudflare’s AI block could shut out Google next week, even if you change nothing

Cloudflare plans to remove an exemption that has kept search crawlers such as Googlebot outside its AI blocking rules, with the change scheduled to take effect on September 15.

The company says Googlebot, Microsoft’s Bingbot and Applebot will be blocked wherever applicable AI training restrictions are enabled, unless the site owner opts out of the change. That includes sites using its older “Block AI bots” setting. Cloudflare’s announcement

Currently, that setting excludes crawlers used for both search and training. Under the new policy, Cloudflare will assess each crawler against all its classified uses and apply the strictest restriction. A crawler classified for both purposes could therefore be blocked even when search crawling is allowed.

Announced in July, the policy is intended to push companies to separate their crawlers by purpose. Cloudflare argues that combining search with other AI uses gives established search companies an advantage over competitors whose bots publishers can block separately.

“We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training,” CEO Matthew Prince said in the announcement.

Google already offers Google-Extended, a control that lets publishers restrict certain Gemini training and grounding uses without affecting their inclusion in Google Search. It is a robots.txt instruction, however, rather than a separate crawler.

Googlebot still handles crawling for Search, including AI Overviews and AI Mode. Google-Extended does not provide a separate opt-out for those search features. Blocking Googlebot at the network level would also prevent it from fetching pages for conventional search results.

Cloudflare is also changing its defaults for pages that display advertising. Training bots and agents acting on users’ behalf will be blocked by default, while search remains allowed, subject to the rules for crawlers with multiple purposes.

Those defaults cover new customers, new sites added by existing customers, and existing free customers who have not changed their settings before September 15. The expansion of existing training blocks is a separate change, so paid customers with those restrictions enabled may also be affected.

The advertising condition reflects Cloudflare’s view of the economics: an ad indicates that the publisher expects a person to visit the page, generating revenue. The company says bots that substitute for that visit should face different defaults.

Alongside the restrictions, Cloudflare is developing Pay Per Use, an expansion of its Pay Per Crawl initiative. Its announced arrangements include payments when an opted-in publisher’s content appears in Ceramic.ai search results, and on-demand purchases of premium content by You.com agents. Those arrangements depend on participation by publishers and the relevant AI services.

Website owners can change the policies in Cloudflare’s Security settings and opt out of the September 15 changes before the deadline. Restrictions can apply across a site or only to pages where Cloudflare detects advertising.

The post Cloudflare’s AI block could shut out Google next week, even if you change nothing appeared first on Search Engine Watch.