Cloudflare Pay Per Crawl: What Changes On 15 September

On 15 September 2026, Cloudflare starts blocking a category of AI crawlers by default. If you have never opened a robots file and are on a free plan, this applies to you and nobody has told you.

The change has been reported almost entirely as a publishers-versus-AI story. That framing is how it was pitched, and it obscures who is actually affected.

What is changing

From 15 September, Cloudflare's default settings block "mixed-use" crawlers from any page that hosts ads. Mixed-use means bots that serve both traditional search indexing and AI training or AI agents, rather than crawlers dedicated to one purpose.

The new defaults apply to three groups:

• New Cloudflare customers
• New sites set up by existing customers
• All existing customers on the free plan

At the same time, Pay Per Crawl becomes Pay Per Use. Rather than charging AI companies per page fetched, the model pays publishers when their content is actually used. Launch partners include Ceramic.ai, which pays when content appears in search results, and You.com, which pays when premium content is accessed by an agent.

Why the free tier is the real story

Large publishers already had a position on crawlers and the staff to hold it. They were in the conversation, they lobbied for exactly this, and their defaults were already deliberate.

The group whose settings are about to change without their involvement is much larger and much quieter: small businesses on a free plan who have never made a crawler decision in their lives.

An infrastructure default deciding your machine visibility is a genuinely new situation. Your visibility to machines is about to be set by an infrastructure vendor's default rather than by a decision anyone in your business made.

The underlying policy is defensible, and worth saying so clearly. Unpaid extraction at scale was never sustainable, and somebody had to move first. The objection is not to the direction. It is to a setting flipping on for people who were never asked.

Does blocking AI crawlers help or hurt my business?

Whether you want the traffic is the question the default cannot answer for you, because the right answer depends entirely on how your business makes money.

Crawler decisions seen from a traffic monetization goal and from a discoverability goal

	If you monetize traffic	If you need to be found
Being harder to crawl	Leverage in a negotiation	Simply harder to find
Content in an AI answer	A visit you did not get paid for	A referral you wanted
Pay Per Use	A revenue line	Irrelevant, no meaningful volume
Right default	Restrict	Allow

A publisher with real traffic gains leverage by being harder to crawl. A service business trying to get found gains nothing, and loses the chance to be the answer when a potential customer asks an assistant who does this kind of work.

Same default. Opposite consequence. And nobody sorted anyone into the right bucket.

For most small and mid-sized service businesses, the honest answer is that you want to be crawled and cited. Your content is not the product. It is how you get discovered, and the assistants are increasingly where discovery starts.

What a "mixed-use" crawler is, and why the distinction is new

The word doing the work in this policy is "mixed-use," and it is worth understanding because it is the mechanism, not a detail.

For most of the web's history a crawler had one job. Googlebot indexed pages so they could appear in results, and the bargain was legible: you let it in, it sent you visitors. Blocking it meant disappearing, so almost nobody blocked it.

AI broke the bargain by making one crawler serve two purposes. The same fetch can index a page for search and feed a model that answers the question without sending anyone anywhere. From the site's side those two uses are indistinguishable, because it is the same request from the same agent.

Serving two purposes at once is what "mixed-use" names. Cloudflare's move is to treat a crawler that will not separate the two purposes as one you should be able to refuse by default, which pressures AI companies to split their crawlers so site owners can make a real choice.

The strategy is coherent. The side effect is that until the separation happens, refusing the mixed crawler can also cost you the search-adjacent benefit, and small sites will absorb that trade without knowing they made it.

Why "ads on the page" is a strange trigger

The default applies to pages that host ads. That condition sounds narrow and technical, and it quietly decides who is affected.

The ad-presence test makes sense as a proxy for a publisher. If a page carries ads, its owner monetizes attention, so an AI answering from that page without sending a visitor is taking revenue.

The trouble is that plenty of businesses carry ad scripts for reasons that have nothing to do with monetizing content. Retargeting pixels, analytics with an ad-network dependency, a conversion tag left over from a campaign. A site can qualify as ad-hosting while earning nothing from ads at all.

If that describes you, a default designed for publishers is about to be applied to a lead-generation site, where the economics run the opposite way. That is worth checking specifically rather than assuming the trigger does not apply to you.

What Pay Per Use means if you are not a publisher

Realistically, very little, and it is worth saying plainly so nobody builds a plan around it.

Pay Per Use pays when content is used: when it appears in a partner's AI search results, or when premium content is accessed by an agent. That is a genuine revenue line for someone producing high volumes of content that assistants want to draw on.

A service business with thirty pages and a blog is not that. The payments would be negligible, and pursuing them would mean optimizing for being consumed rather than being contacted, which is the wrong goal for a business whose website exists to start conversations.

The useful read is that Pay Per Use tells you where the platform is heading. Content access is becoming a metered, paid relationship between large parties. Small sites are not participants in that market, they are just subject to its defaults.

One more asymmetry is worth naming. The businesses most likely to be caught by this are the ones least likely to be reading about it, because following CDN policy changes is not a job anyone at a twelve-person firm has been given.

What to check this week

Checking your own robots file is a twenty-minute job and almost nobody will do it.

1. Find out what sits in front of your site. Many businesses genuinely do not know whether they are behind Cloudflare, because a developer set it up years ago.
2. Check your plan. The free plan is the one where existing sites get the new defaults applied. Paid plans keep their current configuration.
3. Look at your current bot settings. Whatever is there now is what you are choosing to keep or change, and doing nothing is also a choice from 15 September onward.
4. Decide deliberately, and write the decision down. Whichever way you go, the point is that a person in your business made the call, so it can be revisited when the model changes again.
5. Check your robots file separately. Cloudflare's defaults and your own robots directives are two different controls, and they can disagree.

What this does not change

Three things are worth ruling out, because the coverage has blurred them together.

• Googlebot is not being blocked. Cloudflare has said new sites keep letting search engines index their pages. Traditional search indexing is not the target of this change.
• Your robots.txt still does what it did. Cloudflare's defaults operate at the network edge, ahead of your file. They are a second, separate control, which means the two can disagree and the edge wins.
• Nothing is retroactive. Content already ingested by models that trained on it is not recalled by a crawler block. This affects future access, not the past.

That last point matters for expectations. Blocking crawlers now does not remove you from an assistant's existing knowledge, and allowing them does not immediately put you into one. Both directions take time to show up.

The wider pattern

Strip out the specifics and what is left is a governance question that keeps recurring.

Decisions about whether your business is legible to machines are increasingly being made by intermediaries: your CDN, your CMS, your hosting provider, your platform. Each of them ships defaults, and defaults become policy for everyone who never opens the settings.

Delegating the decision is fine when your interests and theirs align. Cloudflare's interest here is in building a marketplace where AI companies pay for access. That is a reasonable business, and it is not the same as your interest in being discoverable.

The practical defense is not technical. It is knowing which of your defaults were chosen and which were inherited, and revisiting that list once a year rather than once a crisis.

Do you actually know who controls whether an AI assistant can read your website right now, and when you last made a deliberate decision about it rather than inheriting one?