One choice, fewer separate instructions

Google announced on 30 October that it plans to support a shared AI-training control in robots.txt, the text file websites use to communicate with automated crawlers. Its stated target is 2027. Microsoft announced related support the same day and gave a more specific date: it will begin honouring its new preferences as it crawls from 1 February 2027.

The distinction matters: these are announced implementation plans. They are not evidence that a single instruction already works across every AI company. The proposal is an individual Internet-Draft discussed through the Internet Engineering Task Force’s AI Preferences work, not a finished, endorsed IETF standard.

The practical aim is straightforward. Instead of repeating an instruction for a long list of companies, a website owner could express a shared preference about a particular use: generative AI training. Participating providers would interpret that common signal.

What the proposed signal actually says

The draft calls its shared identifier __usage_ai-training. It places that identifier in an ordinary robots.txt group and uses familiar Allow and Disallow rules to express preferences for particular website paths. A path is simply the part of a web address identifying a page or section.

Under the draft, a disallowed path expresses a preference against the covered training use; an allowed path expresses a preference in favour. If the shared group is missing, the draft says no preference has been expressed through this mechanism. Silence is not described as an affirmative instruction from the publisher.

The draft separates content-use preferences from rules about crawling. An instruction about training is not itself an instruction to stop fetching a page. A provider might decide not to fetch material it cannot use for its purpose, but that is different from the protocol creating a technical access restriction.

Training and answering a question are different uses

Training changes the learned parameters of a model: the internal numerical settings it uses to produce outputs. Retrieving a web page while preparing an answer is a different activity. It can provide current context without that request itself retraining the model. Microsoft calls the latter category grounding.

Microsoft’s announcement distinguishes its shared training preference from separate page-level grounding preferences. It says the new controls will not change eligibility for Bing search results. These are Microsoft’s announced rules for the covered systems, not a promise about every search engine or AI service.

Google says its planned handling will map a shared training refusal to Google-Extended when that company-specific control is not explicitly specified. Existing Google-Extended instructions will continue alongside the shared signal. Website owners therefore cannot assume that every provider will implement exactly the same combination of training and answer-related settings.

A preference is not a locked door

The Robots Exclusion Protocol standard, RFC 9309, explicitly distinguishes its rules from access authorisation. A publicly accessible robots.txt file communicates instructions; it does not require someone requesting a public page to authenticate or obtain a key.

SIGNAL analysis: imagine putting a notice beside a document saying it may be read but should not be used for a particular purpose. The notice can make the choice clear to a cooperative reader. It does not physically prevent someone else from copying the document. That analogy describes the difference between a preference and an access barrier; it does not settle anyone’s legal rights.

A common signal could improve consistency among providers that choose to support it. It cannot establish that an unknown crawler follows the rules, erase data already collected, or demonstrate what a model was trained on. None of those outcomes should be inferred from the announcement.

Why this matters beyond website administration

SIGNAL analysis: the internet’s usefulness depends on people making material available, including reporting, research, documentation and creative work. Discoverability and reuse are related, but they are not identical choices. A publisher may want readers to find a page while objecting to a separate use of its contents.

A common vocabulary could reduce the work involved in stating that position. It could also make different providers’ public commitments easier to compare. But a simpler instruction is useful only if its meaning, scope and implementation are understandable. Otherwise, one short line can create an impression of control stronger than the evidence supports.

For readers, the important question is not whether a website has added a fashionable file entry. It is whether the provider explains what it honours, when it applies the preference and which activities remain outside it. Those details determine what the promise actually covers.

The next evidence to watch

Google’s documentation already distinguishes Google-Extended from Googlebot, the crawler associated with Google Search. Its Google-Extended description covers training and certain grounding uses, while saying that the control does not determine inclusion or ranking in Google Search. These provider-specific definitions remain important alongside any shared signal.

The immediate milestones are the evolution of the draft, published implementation guidance and the providers’ announced 2027 rollouts. Microsoft’s February date is more specific than Google’s broader 2027 target (its crawler documentation specifies early 2027); neither should be converted into a claim of universal availability today.

SIGNAL analysis: stronger evidence would show that different providers interpret the same preference consistently, document exceptions and give publishers a way to understand their actual behaviour. A shared opt-out could become a useful piece of internet plumbing. The achievement to measure is meaningful respect for the preference, rather than the existence of the line itself.

Original sources

Sources checked 1 November 2026. Send a correction.