Learn / What is pay-per-crawl?
What is pay-per-crawl?
Pay-per-crawl is a pattern where a website, instead of blocking an AI bot or serving it for free, quotes it a price — a machine-readable payment demand (typically via HTTP 402) that the crawler can pay automatically to access content. It converts scraping pressure into revenue.
Who ships it
Cloudflare launched pay-per-crawl controls letting site owners return x402-style pricing to bots at the edge; AWS added machine-readable price and payment terms to CloudFront in 2026. Together they put the capability in front of a large share of the web.
Why publishers care
AI training and answering consumes content while sending fewer clicks. Pay-per-crawl is the first mechanism that prices machine consumption directly — every crawl becomes a micro-transaction rather than a lost pageview.
The mechanics
The crawler requests a page; the edge responds 402 with amount, asset (usually USDC), and address; the crawler pays and retries with a receipt; the edge verifies and serves. It is x402 applied to content instead of APIs.
Frequently asked questions
Does pay-per-crawl block Google?
Site owners choose which bots see prices, which pass free, and which are blocked — search crawlers can be exempted while AI scrapers are charged.
What do crawls cost?
Prices are set by the site owner and are typically fractions of a cent to a few cents per request — high-value content can charge more.
Is this live today?
Yes. Cloudflare's implementation is live, AWS CloudFront supports machine-readable pricing, and the x402 Foundation standardizes the payment leg.
Live, curated, and machine-readable.
Track the x402 standard