Cloudflare is giving websites a new tool to take back some control as artificial intelligence companies continue to gather info and scrape huge amounts of online content. The new system helps site owners decide if AI bots can access their material, and if so, at what price. For many publishers, this could offer a chance to turn the crawling of their work into something that pays.
Websites Are Now Setting the Rules
The rise of AI-driven crawlers has caused frustration among content creators. While these bots collect material to build large language models and other AI products, they often do it without returning traffic to the original pages. This has slowly chipped away at the visitor numbers that many websites once relied on to earn advertising revenue.
Cloudflare’s latest feature gives site owners a way to manage this. They can now decide whether to block AI crawlers completely or charge them through a system that prices each visit. This approach allows publishers to step into a position where they can control how their content is used instead of simply watching it get scraped.
This system is currently available as part of Cloudflare’s Pay Per Crawl beta program. Website owners who are interested can apply for early access here.
Publishers and Platforms Are Getting Involved
Several large publishers and social platforms have come forward to support this move. Companies behind well-known outlets and websites are paying close attention to how AI is reshaping the internet. Many of them are now pushing for ways to protect their work while finding paths to new revenue.
The internet has gone through a fast shift in how traffic moves. For years, search engines gathered content and sent users to the websites where the information came from. That model worked well for content creators. It helped build audiences and supported advertising businesses. But the flow of visitors is no longer the same.
Recent patterns show that Google’s web crawlers are still active, but the company now sends back far fewer visitors than it did just months ago. Data from Cloudflare suggests the gap between crawling and referrals has widened sharply. It used to be around six crawls for every visitor sent back to a site, but now, that gap has grown to about eighteen crawls per visitor. Some of this change seems tied to Google’s newer search features that provide answers right on the results page, which means fewer people click through to the original source.
Other AI companies pull content at even higher rates without sending traffic back. OpenAI, for example, has a much wider gap between what it takes and what it gives in return.
AI Crawlers Are Challenging Old Web Habits
For a long time, the web worked on a simple pattern. Search engines crawled the web, indexed the information, and passed visitors back to the sites they found. That cycle supported the people and companies who created content.
Now, with AI bots collecting material to train chatbots and language models, much of that balance has shifted. These AI systems often provide information directly to users without pointing them to the original websites. As a result, many publishers feel cut out of the process.
Some AI companies have also found ways to bypass technical tools that are meant to block content scraping. They argue that gathering public information in this way does not break any laws. On the other side, many publishers believe their rights are being ignored.
The clash is already playing out in courts. Some companies have filed lawsuits against AI firms, accusing them of using their work without permission. At the same time, other publishers are making deals to license their content to AI companies under agreed terms.
Legal Fights and Deals Are Now Reshaping the Space
The fight over how AI companies use online content is unfolding on several fronts. Reddit, for example, recently launched legal action against an AI company it claims scraped user posts without approval. Yet, Reddit has also struck a content-sharing deal with Google, showing that some companies are choosing both paths: to sue where needed and to partner where possible.
Cloudflare’s new tool arrives as publishers are urgently looking for ways to set boundaries and, if possible, to earn fair payment when AI firms rely on their work. The growing tension around AI crawlers is pushing the internet toward new rules, and this tool may be part of that shift.
Note: This post was edited/created using GenAI tools.
Read next: From Text to Talk: Meta Brings Voice Calling to Enterprises and Teases Future AI-Driven Customer Interactions
