Cloudflare Gives AI Companies a Deadline to Separate Crawlers or Face Publisher Blocks

Cloudflare announced a new policy that gives AI companies until September 15 to distinguish their web crawlers used for search from those used for AI training and agents. If they don't comply, their crawlers could be blocked by default on many publisher sites. Web crawlers are automated programs that scan websites to gather information. This move aims to protect publishers' content and potentially force AI companies to pay for using it.

This policy shift could have significant implications for both AI companies and content creators. Publishers have long struggled with AI companies scraping their content without compensation. By allowing publishers to block AI training crawlers, Cloudflare is giving them more control over their content and potentially opening the door for licensing deals. For AI companies, this means they may need to negotiate with publishers for access to their content, which could increase their operational costs.

If you're a publisher concerned about your content being scraped, you can start by checking your website's settings on Cloudflare. Look for options to block specific crawlers or enable default blocking for AI training crawlers. This will help you take control of how your content is used by AI companies. For AI companies, it's crucial to review and update your crawling practices to comply with Cloudflare's new policy before the deadline.