Google Teams Up with OpenAI and Anthropic on New Scraping Controls
anthropic google microsoft openai training
| Source: Mastodon | Original article
Google, together with OpenAI, Anthropic and Microsoft, pledged to support a common robots.txt signal that blocks scraping for AI training.
Google, OpenAI, Anthropic and Microsoft announced a joint pledge to adopt a new “common signal” in robots.txt that would let website owners explicitly block their content from being scraped for AI‑training purposes. The initiative, reported by Engadget, aims to create a standardized, machine‑readable directive that crawlers can recognise and respect, signalling that the site’s data should not be harvested for large‑scale model training.
The move matters because the rapid expansion of generative AI has intensified scrutiny over how training data are sourced. Content creators, publishers and regulators have raised concerns that indiscriminate web crawling may infringe copyright and bypass consent mechanisms. A unified robots.txt signal could give owners a practical tool to protect their material, reduce legal exposure for AI firms, and demonstrate industry willingness to self‑regulate amid growing policy pressure.
What to watch next includes the technical specifications of the signal, how quickly major search‑engine and AI crawlers will implement it, and whether other players—especially smaller AI startups and third‑party data aggregators—will follow suit. Regulators may also look to the pledge as a benchmark for future guidance on data‑use transparency. This development follows Anthropic’s recent steps to limit training on certain data sets, underscoring a broader shift toward tighter control over AI‑training pipelines.
Sources
Back to AIPULSEN