September 15, 2026 is Cloudflare’s deadline to default ad-supported pages to allow search but block AI training and agent use, while mixed-use crawlers that do not separate those functions will be blocked.
More than half of web requests now come from bots, Cloudflare said, and over 50% of AI crawl traffic re-fetches unchanged pages, driving its push for tighter classifications, crawl controls and efficiency signals.
New analytics include an Attribution Business Insights dashboard and answer-engine optimization tools so site owners can see how AI bots use content, how often they are cited, and what traffic AI companies send back.
Cloudflare is also extending Pay Per Crawl into Pay Per Use, with partners including Ceramic.ai and You.com, so publishers can be paid when content appears in AI results or is accessed on demand.
The broader aim is to build a commercial framework for the 'agentic Internet' as publishers sign more than 50 major AI licensing deals and seek compensation without sacrificing discoverability.
As websites block AI training, are we heading toward a future of less intelligent and more biased AI assistants?
As tech giants are forced to pay for web data, is the foundational promise of an open internet collapsing?
Cloudflare’s September 2026 AI Crawler Shift: Ending Free Data Access and Launching Pay-Per-Crawl for Publishers
Overview
Cloudflare’s new AI crawler policy, effective September 15, 2026, marks a major shift in how online content is managed and valued. As the traditional 'crawl-for-traffic' model becomes less fair—especially with AI companies using content for training without consent or compensation—Cloudflare is giving publishers more control over their content. This policy addresses the imbalance where companies like Google dominate access and make it hard for publishers to stay visible without giving up their content for AI use. By rebalancing content access and empowering creators, Cloudflare aims to create a more equitable exchange between publishers and AI companies.