Cloudflare Inc. today debuted a tool that enables website operators to charge artificial intelligence developers for access to their content.

The new pay per crawl feature, as it’s called, is in private beta test. 

Cloudflare operates a popular content delivery network, or CDN, that website operators use to speed up page loading times. The company also provides related services that ease tasks such as blocking cyberattacks. Cloudflare estimates that its infrastructure processes about 20% of the world’s web traffic.

After enabling the new pay per crawl feature, website operators can define a flat fee that will be charged from AI bots for every request. If necessary, different settings can be applied to specific crawlers. “This is particularly helpful if you want to allow a certain crawler through for free or if you want to negotiate and execute a content partnership outside the pay per crawl feature,” Cloudflare executives Will Allen and Simon Newton wrote in a blog post.

Websites exchange data with visitors’ browsers in the form of HTTP messages. When a user visits a webpage, the user’s browser sends an HTTP message that asks to download the contents of the page. The website then responds with HTTP messages of its own that contain the requested files.

When an AI crawler visits websites connected to Cloudflare’s pay per crawl tool, it receives an HTTP message indicating that payment is required for access. The crawler can optionally send an HTTP message that broadcasts the maximum price its developer is willing to pay. Access is granted if the website operator’s pricing is accepted.

Cloudflare has equipped its pay per crawl feature with technology designed to block malicious activity. In particular, it prevents hackers from gaining access to a website by impersonating an AI crawler whose developer paid for content. Cloudflare will mitigate such spoofing campaigns using public key cryptography.

With public key cryptography, an AI crawler generates an encrypted message that only it can decrypt. Website operators then receive a copy of this encrypted message. To verify an AI crawl request, website operators ask the bot that sent the request to decrypt the encrypted message.

The pay per crawl feature uses a public key cryptography algorithm called Ed25519. It encrypts data by encoding it into mathematical structures known as twisted Edwards curves that were discovered in 2008. Besides Cloudflare’s AI crawler verification mechanism, the algorithm is also used in the popular SSH protocol for logging into servers and other cybersecurity tools.

“Given that Cloudflare protects the majority of the world’s most popular websites, as well as millions of smaller websites that publish academic and scientific content, this security feature will elegantly prevent data-greedy bots from unwarrantedly scraping human-created content without permission and without paying for it,” said Ilia Kolochenko, chief executive of cybersecurity company ImmuniWeb.

Cloudflare debuted the pay per crawl feature alongside several other new AI-focused capabilities. Going forward, the company’s CDN will block AI crawlers from accessing websites by default. There’s also a new setting that allows publishers only to block crawlers from accessing pages with ads.

Image: Cloudflare