Google, Anthropic and others back new AI crawling opt‑out standard, says SiliconANGLE
anthropic google microsoft openai training
| Source: Mastodon | Original article
Google, Microsoft, Anthropic and OpenAI announced a joint effort to create a new robots.txt extension that lets website owners block generative‑AI crawlers with a single directive. The proposal, first reported by SiliconANGLE, would add a specific entry to the long‑standing robots.txt file, signalling that the site does not wish to be scraped for AI‑training data. Microsoft and OpenAI have committed to implement the control immediately, including a feature dubbed “AI Grounding” that ties scraped content to its source. Google said it will roll out support for the standard in the following year.
The move builds on the industry‑wide push for clearer data‑use policies that we covered earlier this month, when Google joined OpenAI and Anthropic in promising new scraping controls [2026‑11‑01]. By formalising an opt‑out mechanism, the consortium aims to address growing pressure from regulators and content creators who argue that large‑scale model training often proceeds without explicit permission. A standardized signal could reduce legal uncertainty, improve public trust, and give publishers a practical tool to protect their intellectual property.
What to watch next: the technical specifications of the proposed robots.txt entry, and how quickly the major AI providers integrate it into their data‑collection pipelines. Industry observers will also monitor whether other players—such as smaller model developers or cloud providers—adopt the standard, and how regulators respond to a de‑facto industry‑led solution. The timeline for Google’s implementation will be a key indicator of how quickly the broader ecosystem can align around the new opt‑out framework.
Sources
Back to AIPULSEN