Cloudflare has introduced a new tool designed to block bot crawlers from accessing website content without permission or payment, aiming to help publishers generate revenue from artificial intelligence firms that collect and train on online material, the company announced on Tuesday.
The tool gives website owners the ability to decide whether AI crawlers can access their content and to set a price through a "pay per crawl" model. Cloudflare said this approach empowers content creators to manage how their work is used while ensuring they are compensated fairly. As more AI tools collect data without referring traffic back to source websites, publishers are seeking new revenue streams to offset the decline in ad income from search engine referrals.
The initiative has gained backing from prominent publishers, including Condé Nast and the Associated Press, as well as major platforms like Reddit and Pinterest.
Stephanie Cohen, Cloudflare’s Chief Strategy Officer, said the aim is to put control back in the hands of content creators and help build a sustainable online ecosystem. "The change in traffic patterns has been rapid, and something needed to change," Cohen said in an interview. "This is just the beginning of a new model for the internet."
Cloudflare cited data showing that Google’s ratio of site crawls to visitor referrals has dropped significantly, now standing at 18 to 1, compared to 6 to 1 just six months ago. This suggests Google continues to crawl content at the same pace but sends fewer visitors back to the original sites. The change may be due to new search features like AI Overviews, which display summarized answers directly in search results.
Even so, Google’s referral ratio remains far higher than that of other AI firms. For instance, Cloudflare reported that OpenAI's crawlers operate at a ratio of 1,500 to 1, meaning they crawl content far more frequently than they send users back to it.
For years, search engines helped sustain web publishers by directing traffic to their sites in exchange for access to their content. This model rewarded quality content creation. However, AI companies are now collecting and repurposing this content to power chatbots and other AI tools, often without crediting the source or driving traffic back. Many of these companies bypass the standard tools used by publishers to block such scraping and claim their actions are legally permissible.
In response, several media organizations have taken legal or commercial action. The New York Times has filed a lawsuit for copyright infringement against some AI developers, while others have entered licensing agreements.
Reddit, for example, has filed a lawsuit against AI startup Anthropic, accusing it of scraping user comments to train its chatbot. At the same time, Reddit has reached a content licensing deal with Google.
Found this article interesting? Follow us on X(Twitter) ,Threads and FaceBook to read more exclusive content we post.
