OpenAI, Google, Microsoft and Anthropic adopt new publisher opt‑out standard against AI crawlers
anthropic google microsoft openai training
| Source: Techmeme | Original article
OpenAI, Google, Microsoft and Anthropic will adopt a new standard that lets publishers opt out of AI scraping across multiple crawlers.
OpenAI, Google, Microsoft and Anthropic announced they will adopt a new industry‑wide standard that lets website publishers opt out of having their content scraped for AI training. The move, reported by The Verge and corroborated by several tech outlets, introduces a control that can be applied across multiple crawlers, giving publishers a single mechanism to express “no‑scrape” preferences.
The development matters because the rapid growth of large language models has intensified debate over how training data is sourced. Publishers have long complained that their articles, images and other digital assets are harvested without consent, potentially eroding revenue and raising copyright concerns. By backing a unified opt‑out protocol, the four AI leaders signal a willingness to address those concerns and to create a more predictable data‑collection environment. The standard also promises technical consistency: a single signal embedded in a site’s robots.txt or similar file can be honoured by the participating crawlers, reducing the need for each company to implement its own bespoke filters.
What to watch next includes the timeline for rolling out the protocol and whether other AI developers will join the coalition. Regulators in Europe and North America are monitoring data‑use practices, so the standard could become a reference point in future policy discussions. Additionally, publishers will be looking for guidance on how to implement the opt‑out signal and on any verification tools that will confirm compliance. The effectiveness of the standard in actually limiting data ingestion will likely shape the next round of negotiations between the AI industry and the publishing sector.
Sources
Back to AIPULSEN