Google and Microsoft back shared AI-training signal for robots.txt

Google and Microsoft have joined OpenAI and Anthropic in backing a common robots.txt signal that tells AI developers whether a site permits its content to be used for model training, TechRadar reported. The proposal is intended to replace separate instructions for individual companies’ crawlers with one shared entry that participating providers interpret consistently, according to the outlet.

The proposed signal would not itself prevent a bot from accessing a website. TechRadar reported that it communicates a preference about how content may be used, rather than blocking a crawler from fetching that content. The outlet also noted that the mechanism would not affect material that companies had already collected. It said the approach is not an approved standard and that none of the four companies fully honors it yet.

The effort is part of work associated with the Internet Engineering Task Force, rather than a mechanism created solely by OpenAI, TechRadar reported. The outlet said an Internet-Draft published by Anthropic’s Kevin Kelley describes a policy token, “__usage_ai-training,” for use in the User-Agent line of a robots.txt group.

The proposal’s definition of training is also limited. TechRadar said the IETF’s AI Preferences working group defines it as changing the learned parameters of a generative model, while excluding models used only for tasks such as classification or ranking. That distinction means the signal concerns a specific type of AI development, rather than every use of website material by automated systems, according to the report.

A shared instruction could make it simpler for website operators to express a preference across participating providers, TechRadar reported. But the signal’s practical effect depends on companies choosing to follow it; the outlet’s account says it neither blocks access nor covers previously gathered material. Google, Microsoft, OpenAI and Anthropic are the companies identified as backing the approach, according to TechRadar and Mezha.