Blocking AI crawlers with Cloudflare FAQs

Cloudflare, one of the world’s largest content delivery networks, recently announced that the default settings for its free plan will be set to block AI crawlers. Creators and publishers will be able to decide if they want to give AI crawlers access to their content, and if they do, how AI systems can use that content. It’s an important step towards protecting original content from unauthorized use and increased transparency around how AI companies are using it. 

FAQs

What is Raptive’s relationship with Cloudflare?

Cloudflare is one of the largest and best-known content delivery networks, and they provide their services to web hosts used by many Raptive creators, so it’s often built into your hosting package. We work with Cloudflare in multiple ways; this initiative impacts the entire ecosystem, so we got involved with discussions to represent our creators and the broader community. 

Does Raptive recommend blocking AI crawlers through Cloudflare? 

Yes. Raptive strongly recommends blocking major AI crawlers such as OpenAI, Anthropic, and others. These crawlers collect and use web content to train AI models, without permission or compensation. We believe your content should not be used in this way unless there’s a clear, fair value exchange. Blocking sends a signal to AI companies that consent and compensation matter.

Are there any downsides to blocking AI crawlers through Cloudflare? 

At this time, there are no known negative effects on revenue or performance from blocking AI crawlers. The crawlers being blocked are not tied to search engines or ad systems. However, because this technology is evolving quickly, we’ll continue to monitor the ecosystem closely and keep you informed if anything changes. By default Cloudflare does not block Google-Extended or GoogleBot.

Will this have a negative impact on search traffic? 

No. Blocking AI crawlers will not affect your visibility in Google Search or other major search engines. This is separate from the crawlers used to index your content for search results. You can continue to be discovered by users while protecting your work from unauthorized use in AI tools.

Does this setting block AI search bots? 

It does not block AI search bots. 

How does this impact Raptive’s robots.txt bot blocking? 

Raptive already helps block many AI crawlers using your site’s robots.txt file. The Cloudflare feature adds an additional, more technical layer of defense by filtering traffic at the network level, potentially stopping bots that ignore or bypass robots.txt.

What is pay per crawl? I read that Raptive was one of many companies who opted into it.

Pay per crawl is Cloudflare’s private beta program that would allow creators to charge AI crawlers for access to their content. It’s still in early days of testing and we don’t yet know how it will evolve. Creators interested in participating can reach out to Cloudflare for more information.

Early versions of some published articles incorrectly stated that Raptive was part of a group of publishers who had opted in to pay per crawl. While we—and other large publishers—are in conversations about how to make this a viable solution for creators, it's not something we’ve opted into at this time. We don't believe it's the right payment model for the longer term, but are excited to get data from those who do test it.

If I give AI crawlers permission to access my content, will I be reimbursed for that?

Currently, there is no standard compensation model for creators who allow AI companies to use their content. Some large publishers have signed individual deals, but those terms are not public and are not broadly available. We are working on this, but until there's a clear path for creators to benefit, we recommend blocking access. 

I'm an existing free user, will my settings automatically change?

Yes. Cloudflare has announced that free-tier users will have AI bot blocking enabled by default. If you’re using Cloudflare’s free plan, your site will begin blocking the listed AI crawlers automatically. You can change this behavior in your Cloudflare dashboard if you choose.

I’m an existing paid user, how does this affect me? 

Paid Cloudflare users will not have AI bot blocking enabled automatically. You’ll need to log in to your Cloudflare dashboard and turn it on manually. We recommend enabling this feature to protect your content unless you have a specific reason not to.