Creative Commons announces tentative support for AI ‘pay-to-crawl’ systems

Now, the nonprofit is tentatively backing pay-to-crawl systems, saying it is “cautiously supportive.”

In the past, websites freely allowed web crawlers to index their content for inclusion into search engines like Google. They benefited from this arrangement by seeing their sites listed in search results, which drove visitors and clicks. With AI technology, however, the dynamic has shifted. After a consumer gets their answer via an AI chatbot, they’re unlikely to click through to the source.

CC offered several caveats to its support for pay-to-crawl, noting that such systems could concentrate power on the web. It could also potentially block access to content for “researchers, nonprofits, cultural heritage institutions, educators, and other actors working in the public interest.”

It suggested a series of principles for responsible pay-to-crawl, including not making pay-to-crawl a default setting for all websites and avoiding blanket rules for the web. In addition, it said that pay-to-crawl systems should allow for throttling, not just blocking, and should preserve public interest access. They should also be open, interoperable, and built with standardized components.

Cloudflare isn’t the only company investing in the pay-to-crawl space.