EVENT lvl=2 · domain=ai · importance=66
Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping
# title from source: techradar.com
Google, Microsoft, OpenAI and Anthropic are reported to back a shared robots.txt signal for AI-training crawlers. The standard could give website operators a common way to indicate whether they permit AI-related crawling.
- id
- 2144
- level
- 2
- domain
- ai
- importance
- 66 magnitude=68 sources=50 novelty=78 question=72
- relevance
- 92
- sources_n
- 2
- independent_24h
- 2
- first_reported
- updated
- outlets
- techradar.com, mezha.net
QUESTION
By 2027-06-30, will Google, Microsoft, OpenAI and Anthropic each publish documentation stating that their AI-training crawlers recognize the shared robots.txt signal?
type=binar · closes=2027-06-30 · resolves_on=Official crawler documentation from Google, Microsoft, OpenAI and Anthropic · lobby: scheduled when available
SOURCES // 2 · chronological · titles only
| # | published_utc | outlet | title | url |
|---|---|---|---|---|
| 1 | techradar.com | Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping | https://news.google.com/rss/articles/CBMi-wFBVV95cUxQd29saUJrSjhXOHFscWJhSVBHbkVVNTdSOUZiV3lUalRBOHdfdTVCdHBRcHNkenMtSjI2cHNuUzgzVnFOdWxQYlVyR3ZodnlwNjBRalhhWS1wQlBTUC1WYzJDUF9vNUpubmJCNzJvaEljS3o1dDlCaTdTX0pHTHlrakRPMWZST3czUGFONERqeGRyLTFld2tRMnRMU1J2NzJzb2dsdHdzZzZtUHk0U3N4ajhyWWpLZHJEMkpxdW8xODlRSHJHNHNQeGFHSlM5RldHdzlqaG1aQzhCeEJSajRYQjgtTlIwaW9RcHRKYkl1ZlQ5VGhqUGh1cDVSOA?oc=5 | |
| 2 | mezha.net | Google and Microsoft join OpenAI and Anthropic in backing a shared AI-training signal for robots.txt | https://news.google.com/rss/articles/CBMihgFBVV95cUxNZjlIcFZVMGFhVnRxeHNBc2ROcFRkTzZ6WlpyMEt5eXJOajk2TUlMNmc1Q25fQXpUSkZsWlR2aTV1akU0UUdLdV85ZnZnenRfVXhBcUkzckN0TXJpVm9GX1Q2bEtLNmM4MVZxQXAyOUZzTDRhQl9xZ3d3QlE2aDZacGE4Z29jZw?oc=5 |