Major AI labs adopt unified opt-out standard for data scraping

Major AI developers including OpenAI, Google, Microsoft, and Anthropic have committed to a unified standard for website owners to opt out of data scraping. The proposed policy token, introduced by Kevin Kelley, a technical lead for standards at Anthropic, aims to streamline how publishers manage AI access. According to The Verge, this signal will reside within the standard robots.txt file used to direct web crawlers.

Streamlining publisher control

Currently, website administrators must manually update permissions for each individual AI company to prevent their content from being ingested for model training. This new standard creates a single point of entry to communicate preferences across the industry. While Anthropic has already integrated the signal, Google and Microsoft expect to implement the feature next year, with OpenAI targeting a rollout in the coming months.

Industry implementation and stakes

The Scale Report understands that while the adoption of a common signal is a step toward standardization, its efficacy depends on broad industry compliance. Meta communications head Andy Stone expressed support for the initiative but has not yet confirmed whether the company will formally honor the token. This development follows a period of increasing friction between content creators and AI labs, with some entities turning to tools like Cloudflare to block automated scrapers entirely.

This move signifies a shift from the ad hoc blocking methods that have defined the early AI era. As publishers balance the desire for search engine visibility with the need to protect intellectual property from unauthorized model training, a standardized signal reduces the technical barrier to exercising digital sovereignty. The distinction between search indexing and data training remains the core conflict for publishers managing their online presence.