Tech giants back shared standard to let websites opt out of AI training
OpenAI, Google, Microsoft, and Anthropic have agreed to support a unified robots.txt standard allowing web publishers to block AI crawlers from scraping their content for model training with a single command.
OpenAI, Google, Microsoft, and Anthropic announced on Oct. 30 that they will honor a single policy token in website robots.txt files, creating a unified opt-out mechanism for artificial intelligence training scraping across multiple crawlers, according to reports by The Verge and AGTP.[1][2][3]
The shared token, reported as __usage_ai-training, originated as an Internet Engineering Task Force proposal drafted by Anthropic standards lead Kevin Kelley. The IETF is currently considering the proposal, and the agreement remains a voluntary industry pledge rather than a legal mandate, while existing crawler-specific controls will remain functional.[3]
Adoption across the tech companies will follow a staggered timeline. Anthropic has already implemented the token, while OpenAI stated it will roll out support in the coming months. Both Google and Microsoft are targeting 2027 for implementation. Additionally, Meta has expressed strong support for a common opt-out signal.[3]
The standard will interact differently with each platform's tools. Microsoft Bing is introducing more granular controls to differentiate AI training from search-generated AI answers. Google stated that implementing the token will be treated the same as blocking Google-Extended, which restricts model training while maintaining site eligibility for Gemini citations.[3]
Key facts
- OpenAI, Google, Microsoft, and Anthropic announced an agreement on Oct. 30 to honor a single shared robots.txt policy token to opt out of AI training.
- The shared robots.txt token is __usage_ai-training and stems from an IETF proposal authored by Anthropic standards lead Kevin Kelley.
- Anthropic has already adopted the standard, OpenAI plans to implement it in the coming months, and Google and Microsoft are targeting 2027.
- Adoption is voluntary rather than legally required, and previous crawler-specific control methods will remain active.
- Google treats the token as equivalent to blocking Google-Extended, allowing participating sites to remain eligible for Gemini citations.
- Microsoft Bing is adding separate granular opt-outs that distinguish AI model training from AI answers.
- Meta has indicated strong support for a common AI scraping opt-out signal.
Sources · 3 sources
- TE
Techmeme@TechmemePost on X ·
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI across multiple crawlers (@emroth08 / The Verge) (Visit Techmeme dot com for the link and full context!)
Open source - TE
TechmemeArticle ·
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI across multiple crawlers (Emma Roth/The Verge) Emma Roth / The Verge : OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI across multiple crawlers — The new control will allow publishers to apply their AI training preferences across multiple crawlers.
Open source - AG
AGTP@AGTPinsightsPost on X ·
OpenAI, Google, Microsoft and Anthropic just agreed to back one shared AI training opt-out for websites. Here's what you need to know. The four companies announced on October 30 that they will honor a single "policy token" in robots.txt. Publishers can set one broad preference for AI training across all participating providers. They no longer have to manage separate instructions for every AI developer. If a crawler sees the shared token, the companies say they will restrict their AI crawlers from accessing that site. The idea is an IETF proposal, from the group working on AI preferences. It came from Anthropic standards lead Kevin Kelley. The token is reported as `__usage_ai-training`. Adoption is voluntary, and the IETF is still considering the proposal. The rollout is staggered. Anthropic has already adopted it. OpenAI says it will implement it in the coming months and will share details later. Google and Microsoft are targeting 2027. Existing crawler controls stay in place. There are extra details. Microsoft's Bing adds more granular opt-outs, splitting AI training from AI answers. Google says using the line will be treated as blocking Google-Extended, which keeps a site eligible for Gemini citations. Some SEOs are warning about that trade-off. Meta has also said it strongly supports a common signal like this. Key numbers: - 4 companies committed: OpenAI, Google, Microsoft, Anthropic - 1 shared robots.txt token instead of separate blocks per company - Anthropic: already adopted - OpenAI: coming months - Google and Microsoft: 2027 For now, publishers get one switch to say no to AI training, but it is a voluntary pledge, not a legal requirement.
Open source