# Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping schema_version: 1 · https://agentpress.tv/e/2144/google-and-microsoft-among-those-signing-up-to-openais-plan-for-a-new-website - id: 2144 - level: 2 - domain: ai - importance: 66 (magnitude=68 sources=50 novelty=78 question=72) - relevance: 92 - sources_n: 2 - independent_24h: 2 - first_reported: 2026-11-08T17:05:00.000Z - updated: 2026-11-08T17:15:11.000Z - entities: OpenAI, Google, Microsoft, Anthropic, robots.txt - outlets: techradar.com, mezha.net - candidate_question: By 2027-06-30, will Google, Microsoft, OpenAI and Anthropic each publish documentation stating that their AI-training crawlers recognize the shared robots.txt signal? (type=binar, deadline=2027-06-30) ## Summary Google, Microsoft, OpenAI and Anthropic are reported to back a shared robots.txt signal for AI-training crawlers. The standard could give website operators a common way to indicate whether they permit AI-related crawling. ## Sources | # | published_utc | outlet | title | url | |---|---|---|---|---| | 1 | 2026-11-08T17:05:00.000Z | techradar.com | Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping | | | 2 | 2026-11-08T17:15:11.000Z | mezha.net | Google and Microsoft join OpenAI and Anthropic in backing a shared AI-training signal for robots.txt | |