Cloudflare, a major internet company, has launched a brand-new tool that gives website owners more power over who uses their content. The tool is designed to stop AI bots—those are automatic programs—from using website data without permission. These AI bots often copy content from websites to help train artificial intelligence systems like chatbots, without giving credit or paying the content creators.
With this new tool, website owners can either block these bots entirely or charge them a fee for access. Cloudflare calls it a “pay per crawl” system. That means every time an AI bot wants to look at and copy a website’s content, it has to pay.
This marks a big shift from how things used to work. For many years, websites allowed search engines like Google to collect their content freely. In return, those search engines sent people to those websites. This brought in visitors, which led to more ad views and more money for the site owners. But now, things are changing.
How AI Crawlers Are Hurting Website Revenue
In the past, content creators got paid mostly through ads when people visited their websites. But now, more and more AI bots are copying content without sending people to the original website. This means fewer visitors and less ad money. Even though the bots are using the content, the original creators don’t get paid or even recognized.
According to Cloudflare, the number of times Google crawls a website compared to the number of visitors it sends back has changed a lot. Just six months ago, the ratio was 6:1—meaning for every six times a Google bot looked at a site, one person would click through to that page. Now, it’s 18:1. This suggests Google is still crawling sites, but it’s not sending people back nearly as much.
🌐 Spy Games in The Hague? ICC Targeted Again as Cyber Intrusions Escalate
And it’s even worse with other AI companies. Some, like OpenAI, reportedly crawl websites up to 1,500 times for every one visitor they send back. That’s a massive gap. These companies take in lots of content to train their chatbots but don’t help the original websites in return. That makes it harder for the people who write or create that content to keep doing their work.
Some AI companies even ignore rules set by websites that say, “Don’t copy this.” These rules are part of a system called “robots.txt”, which tells bots which pages they’re allowed to access. But many AI bots bypass these limits, saying that what they’re doing isn’t illegal. This has led to several lawsuits, where publishers are suing AI companies for using their content without paying for it.
Publishers and Platforms Back the Initiative
Cloudflare’s new tool is getting strong support from big names in publishing and social media. Companies like Condé Nast (which owns major magazines), the Associated Press, Reddit, and Pinterest are all in favor of this move. They believe this tool is an important step in helping online creators take back control of their work.
This new tool from Cloudflare lets website owners make that choice themselves. They can say yes or no to AI bots. Or, if they’re open to sharing their content, they can set a price. The idea is to make the internet fairer for creators who spend time and effort making high-quality articles, images, and other information.
“Gemini AI Is Watching—Even When You Say No”: Google Update Sparks Privacy Panic
Cloudflare says it wants to help build a system where both content creators and AI companies can benefit. By making sure creators get paid when their work is used to train AI, this tool could lead to a more balanced online world.
Right now, many websites are seeing a drop in traffic from search engines. AI tools often show answers directly without linking back to the source, meaning fewer clicks and less revenue. With this change, content creators may finally have a way to earn back some of the money they’re losing. Cloudflare’s tool may be one of the first steps toward making that happen.
