The distilled version
- OpenAI, Google, Microsoft and Anthropic plan to support a shared signal, placed in a site’s robots.txt file, letting publishers opt out of AI scraping across all of them at once.
- If you run a business website or blog, this could replace the need to set separate AI crawler rules for each company.
- The rollout is staggered and Meta has not committed to honoring the signal, so it isn’t fully in place yet.
What happened
OpenAI, Google, Microsoft and Anthropic say they will adopt a new shared standard, called a policy token, that lets website publishers set their AI training preferences once and have it apply across multiple AI companies’ crawlers, instead of configuring each one separately. The signal was proposed by Anthropic’s Kevin Kelley through the Internet Engineering Task Force and will sit inside a site’s existing robots.txt file, which already tells web crawlers what they can and can’t access.
Microsoft says it will offer granular controls so publishers can opt out of AI training or AI grounding (where a model checks a verified source to reduce errors) separately, and that sites will still show up in Bing search results regardless of their choice. Google has already rolled out similar controls that separate AI use from search visibility.
The four companies are not moving at the same pace: Anthropic has turned the signal on already, while Google and Microsoft say they will add support next year and OpenAI says its rollout will happen within a few months. Meta’s communications head said the company backs the idea of a shared signal but stopped short of saying Meta will actually follow it.
Why it matters
If you run a website or blog, you currently have to set AI scraping preferences separately for each AI company, if that option even exists on your site. A single shared signal would mean one setting to manage instead of several, saving time and reducing the chance you miss one.
What it means for your business
If you run the business: this isn’t live everywhere yet, so there’s nothing to change today. But once your website platform supports the shared signal, it’s worth deciding whether you want your content used to train AI models, or kept out, and setting that preference in one place.
What to do
Worth watching- Check which AI crawlers your website currently allows or blocks via robots.txt, if you manage that yourself or with a developer.
- Note that this shared signal isn’t fully available yet: the four companies are rolling it out at different speeds, from already live to sometime next year.
- Ask your website or CMS provider whether they plan to support the new policy token once it’s live.
- Remember that Meta has not committed to honoring the signal, so a separate opt-out may still be needed for its crawlers.
Worth watching: no action needed yet; revisit when it matures.
Jargon in this story
Sources
- 1The biggest names in AI are backing a new way to opt out of scraping (opens in a new tab) · The Verge · News report · Oct 30, 2026