Cloudflare, an internet infrastructure firm hosting a significant portion of the internet, is rolling out a new system to block AI bots from accessing content on millions of websites, inclu…
Cloudflare, an internet infrastructure firm hosting a significant portion of the internet, is rolling out a new system to block AI bots from accessing content on millions of websites, including major news outlets like Sky News and The Associated Press. This initiative aims to address the growing concerns of content creators who accuse AI firms of using their work without permission or payment to train their systems.
The system targets AI bots, also known as crawlers, which collect data from the web, and eventually, websites will have the option to charge AI companies for accessing their content, introducing a "Pay Per Crawl" system. The move is seen as a "game-changer" by some, like Condé Nast's CEO, who believe it's a critical step towards protecting creators and ensuring a fair value exchange on the internet.
The BBC recently threatened legal action against an AI firm for using its content without permission, highlighting the conflict between publishers and AI companies. Cloudflare argues that AI crawlers disrupt the unwritten agreement between publishers and search engines, as they generate answers without directing visitors to the original source, depriving content creators of revenue.
While the new system is a welcome development, some experts caution that stronger legal protections are still needed. They point out that the technology only protects content on websites that implement it, and it doesn't fully address the issue of AI companies using content without permission.
The UK is currently in a legislative battle over protecting creative industries from AI firms, reflecting the broader debate on how to balance innovation with the rights of content creators. The initiative by Cloudflare reflects a growing awareness of the impact of AI on content creators and the need for new economic models in the age of AI.
The company's CEO emphasizes the importance of giving publishers control and building a system that works for everyone. The system will initially apply to new Cloudflare users and sites that previously participated in blocking crawlers.