OpenAI and Major AI Firms Adopt Shared Policy Token in robots.txt to Simplify AI Training Preferences
Current overview (not historical)
OpenAI has announced that it will honor a shared policy token embedded in a website’s robots.txt file, allowing site owners to communicate their AI training preferences to all participating AI providers in a single configuration. The token, proposed by the IETF, lets publishers set a company‑wide rule for AI crawlers instead of configuring each platform separately. Following OpenAI’s lead, Google, Microsoft, and Anthropic have also committed to adopting the standard. This move is intended to give website operators greater control over how their content is used for AI training, while reducing the administrative burden of managing multiple AI‑specific directives.
Written with AI from the reporting · updated 10/30/2026
Expand the detailed fact route and sources
Story route
A station-by-station route from the first public evidence onward; every station carries its own evidence and uncertainty.
- 1Unclassified fact stationOct 31, 02:57single sourcereport-time only
OpenAI 将尊重共享的 robots.txt 信号帮助网站运营者传达 AI 训练偏好
Read original source ↗ OpenAI NewsReplay this moment in the explorerEvidence & revisions · 1 source records
- Occurred (fact side)：
- not recorded
- Published (valid side)：
- Oct 30, 17:30
- First seen (known side)：
- Oct 31, 02:57
Fact stationSingle source · one-sided claimReport-time onlyCopy may have been revised later; time replay is bounded. Layer labels group reading only, not causation between layers.
- 2Unclassified fact stationOct 31, 04:58single sourcereport-time onlyrevised
OpenAI、Google、Microsoft、Anthropic 采用新标准支持网站AI爬虫退出
Read original source ↗ The Verge · AIReplay this moment in the explorerEvidence & revisions · 1 source records
- Subject：
- openai
- Action：
- 将在 robots.txt 中采用共享策略标记以支持退出爬虫
- Object：
- AI训练爬虫
- Occurred (fact side)：
- not recorded
- Published (valid side)：
- Oct 31, 04:26
- First seen (known side)：
- Oct 31, 04:58
Fact stationSingle source · one-sided claimRevisedReport-time onlyCopy may have been revised later; time replay is bounded. Layer labels group reading only, not causation between layers.
Uncertainty marks flag where to check the sources; they are not proven causation.
Report timeline
Explore every side through dated source reports.
- The Verge · AIRelated fact: single sourceFact occurrence time unverifiedSource report revisedOpenAI, Google, Microsoft and Anthropic to Adopt IETF‑Proposed Policy Token for AI Scraping Opt‑Out
Report first recorded Oct 31, 04:58
OpenAI, Google, Microsoft and Anthropic announced they will support a new IETF‑proposed “policy token” that publishers can embed in their robots.txt files to signal AI‑training preferences. The token allows a single opt‑out setting that applies across multiple AI companies, eliminating the need for publishers to configure each crawler separately. Anthropic has already adopted the signal; Google and Microsoft plan to support it next year, while OpenAI will implement it in the coming months. Microsoft will also offer granular controls to opt out of AI training or AI grounding, though sites will still appear in Bing search results regardless of preferences.
Read original source ↗
- OpenAI NewsFirst-party sourceRelated fact: single sourceFact occurrence time unverifiedOpenAI Will Honor Shared robots.txt Signals to Let Sites Communicate AI‑Training Preferences
Report first recorded Oct 31, 02:57
OpenAI announced that it will respect a shared robots.txt signal, allowing website operators to specify their preferences for AI training data. The signal lets site administrators indicate which content should or should not be used by participating AI providers, simplifying the process of managing data usage for training large language models. This change is intended to give web owners more control over how their content is leveraged in AI development and to streamline compliance with data‑usage policies.
Read original source ↗
Event activity trend
- Latest comparable
- 3
- Comparable peak
- 17Oct 31, 05:00
- 24h change
- -72%
The chart compares consistently observed participants only. Hover or tap to inspect each hour; use arrow keys to move between observations.
Event discussion
Reader opinions are hosted on GitHub Discussions; posting requires a GitHub account. Comments are not verified facts.