The Shared AI Training Opt-Out in robots.txt: How __usage_ai-training Works and What It Costs You in AI Search

TL;DR: Google, Microsoft and OpenAI announced on October 30, 2026 that they will honour one shared robots.txt group, User-Agent: __usage_ai-training, as an AI-training opt-out. Anthropic wrote the draft and says it is building support. Nobody reads it yet: Microsoft starts February 1, 2027, Google in “early 2027,” and OpenAI “in the coming months.” The token controls training, not crawling, and User-agent: * does not count. The trap is AI-search visibility. At Google, the shared opt-out also removes you from Gemini grounding unless you set Google-Extended explicitly. At Microsoft, grounding is a separate page-level setting. Decide training and visibility separately, provider by provider.

What was announced

On Friday, October 30, three posts went up on the same day: Google’s “Supporting a shared AI training control for websites” by John Mueller, Microsoft Bing’s “Your content, your choice” by Krishna Madhavan, and OpenAI’s “Making it easier for websites to express AI training preferences.” All three point to one document: draft-kelley-aipref-training-preference-signal-00, an Internet-Draft by Kevin Kelley of Anthropic, submitted to the IETF’s AI Preferences (AIPREF) working group. Search Engine Journal notes it is listed as an individual draft, not yet official IETF standardisation work. It expires May 3, 2027 unless updated.

The appeal is simple. In Google’s words, site owners will no longer be “managing separate instructions for every AI developer.” Today, an opt-out means one rule for GPTBot, one for Google-Extended, one for ClaudeBot, and so on.

How the token works

The draft adds no new directives. You write a normal robots.txt group, but the User-Agent line names a “policy token” instead of a crawler. The draft’s own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against AI training for the whole site except /press-kit/. Path matching and longest-match precedence follow RFC 9309, the standard robots.txt rules. Four details trip people up:

1. Wildcards don’t count. The draft says User-Agent: * groups “MUST NOT be interpreted as matching the policy token.” If your file has no __usage_ai-training group, you have expressed no training preference, even if you block every bot with *. That is true of every robots.txt published before this draft.
2. It governs use, not access. The token “is not a directive for crawling behavior,” per the draft. To stop a crawler fetching pages, you still need a group for that crawler’s own name.
3. “Training” has a narrow definition. The draft borrows the AIPREF vocabulary: using content “to modify the learned parameters of a generative AI model.” Models used only for classification, ranking or scoring are excluded. So is answer-time retrieval, because it changes no parameters (PPC Land’s analysis walks through this).
4. Timing and deals matter. Preferences apply to content as of the robots.txt in effect when it was retrieved, and “private agreements MAY override” them. The draft does not address content collected before you add the token.

Who honours it, and when

Provider	Start date	How the shared opt-out is applied	AI search / grounding control
Microsoft (Bing, Copilot)	February 1, 2027, as pages are crawled	Training only; page-level override via meta tag or header	Separate page-level AI Grounding preference: Allow, DisallowExtended, Disallow
Google (Gemini)	“In 2027” (blog); “early 2027” (crawler docs)	Treated as a Google-Extended disallow when Google-Extended isn’t specified	Google-Extended covers training and Gemini grounding together
OpenAI	“In the coming months”	Covers “foundational generative AI training”; details to follow	OAI-SearchBot (ChatGPT search) stays a separate control
Anthropic	No date; implementation started	Draft author; Kelley says support is being built	Existing ClaudeBot, Claude-User and Claude-SearchBot controls (per SEJ)

Existing company-specific controls keep working at all four. Google says so explicitly, OpenAI says its current crawler controls “will continue to apply,” and the draft says proprietary tokens such as Google-Extended “keep whatever meaning they have today.” It also says the reverse: a Google-Extended rule on its own does not count as a shared opt-out for OpenAI or Microsoft.

The real trade-off: training opt-out vs AI-search visibility

Most publishers do not want “no AI at all.” They want to avoid training while still being cited in AI answers that send clicks. The shared token handles training cleanly. Visibility is handled differently at each company.

Google: training and grounding are bundled

Google’s blog says that if you use the shared token to opt out “and Google-Extended is not explicitly specified,” Google will treat it “as if the publisher had disallowed Google-Extended.” Google’s crawler documentation, as summarised by SEJ and PPC Land, says Google-Extended controls both Gemini training and grounding in Gemini Apps and Vertex AI. So a blanket shared opt-out also takes you out of Gemini grounding. Google adds that “websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” SEJ found no documented way to allow Gemini grounding while blocking Gemini training. At Google, you choose both or neither.

Google-Extended does not affect Google Search rankings. AI Overviews and AI Mode have their own opt-out in Search Console, which SEJ covered separately.

Microsoft: training and grounding are separate

Microsoft is the most granular. Training follows the shared robots.txt token or a page-level tag. Grounding is a separate page-level preference with three settings. The default is Allow. DisallowExtended keeps your content in Microsoft’s own experiences (Copilot, Bing AI Mode, Microsoft 365) but out of grounding services Microsoft sells to third parties. Disallow blocks grounding entirely. The page-level syntax from Microsoft’s guide:

<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">

Microsoft warns that restricting grounding “will significantly reduce your content’s visibility in AI-generated experiences,” without affecting Bing search eligibility.

OpenAI: training and search crawlers are already separate

OpenAI’s crawler docs say GPTBot governs training on foundation models, and OAI-SearchBot governs whether you appear in ChatGPT search answers. Each setting is independent. Sites that block OAI-SearchBot “will not be shown in ChatGPT search answers,” though they can still appear as navigational links.

Copy-ready recipes

These are illustrations based on the published draft and provider docs. Nothing is enforced yet, so test against each provider’s documentation when they go live.

Recipe 1: no training anywhere, stay visible in AI search where possible

User-Agent: __usage_ai-training
Disallow: /

User-Agent: OAI-SearchBot
Allow: /

Leave Microsoft’s grounding preference unset, because Allow is the default. Accept that Google will treat this as a full Google-Extended disallow, which removes Gemini grounding as well as training.

Recipe 2: same, but keep Gemini grounding (accepting Google training)

User-Agent: __usage_ai-training
Disallow: /

User-Agent: Google-Extended
Allow: /

The explicit Google-Extended rule is more specific and wins at Google, so Google may train on your content and ground answers with it. Everyone else gets the training opt-out.

Recipe 3: protect premium content only

User-Agent: __usage_ai-training
Disallow: /members/
Disallow: /research/

Free pages keep the default (no preference). Path-level control is one of the draft’s stated goals.

Recipe 4: allow everything

Do nothing. No __usage_ai-training group means no preference, and both Microsoft preferences default to Allow. That is the stance BetOnAI takes: our robots.txt welcomes AI crawlers, because being cited in AI answers is how many readers find us.

Deadlines to put in your calendar

• December 15, 2026: Bing Webmaster Tools opens a temporary grounding preference, only for sites still using NoArchive or NoCache on that date.
• February 1, 2027: Microsoft starts honouring the new training and grounding preferences as it crawls.
• March 1, 2027: Microsoft stops reading NoArchive and NoCache as training or grounding signals. If you rely on those tags, publish the new preferences before then.
• September 30, 2027: any temporary Bing grounding preference expires.

What we don’t know yet

• Exact go-live dates for Google, OpenAI and Anthropic.
• How OpenAI resolves a conflict between a shared-token rule and a GPTBot rule. SEJ found its docs silent as of October 30.
• Whether the draft will change. It is an individual draft, and the IETF process may revise the token or its semantics.
• What happens to content collected before you opt out. The draft ties preferences to retrieval time and does not address earlier collection.
• Which other AI companies will adopt it. OpenAI and Kelley both invite others to join.

BetOnAI Verdict

This is the first opt-out that four major AI companies have agreed to read, and it is worth adopting if you object to training. The token is not the hard part. The hard part is that “AI visibility” means something different at each company, and at Google a training opt-out costs you Gemini grounding unless you accept Google training. Make two decisions, not one: do you want your work in training data, and do you want to be cited in AI answers? Then write rules per provider. If AI answers drive your traffic, lean towards Recipe 2 or 4. For building AI-search visibility as a service, see our AI search optimisation agency guide.

Frequently Asked Questions

What is __usage_ai-training in robots.txt?

It is a shared policy token from an IETF Internet-Draft written by Anthropic’s Kevin Kelley. A robots.txt group with User-Agent: __usage_ai-training and Disallow rules tells participating AI providers not to use those paths to train generative AI models. Google, Microsoft and OpenAI have said they will honour it.

Does User-agent: * block AI training under the new standard?

No. The draft says wildcard groups must not be read as matching the policy token, and a file without a __usage_ai-training group expresses no training preference.

Will opting out of AI training hurt my Google or Bing rankings?

Google says Google-Extended does not affect inclusion in Google Search, and Microsoft says its preferences do not change Bing search eligibility. AI answer visibility can drop, though: Google treats the shared opt-out as a Google-Extended disallow, which also removes Gemini grounding.

When does the shared AI training opt-out take effect?

Microsoft begins February 1, 2027. Google plans early 2027, OpenAI says in the coming months, and Anthropic says implementation has started without giving a date.

Sources

• IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00: https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
• Google Search Central Blog, October 30, 2026: https://developers.google.com/search/blog/2026/10/ai-training-control
• Microsoft Bing Blog, October 30, 2026: https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
• OpenAI, Making it easier for websites to express AI training preferences: https://openai.com/index/helping-websites-express-ai-training-preferences/
• OpenAI crawler documentation: https://developers.openai.com/api/docs/bots
• Search Engine Journal, November 2026: https://www.searchenginejournal.com/google-microsoft-openai-shared-ai-training-opt-out/595387/
• PPC Land, October 31, 2026: https://ppc.land/google-microsoft-openai-and-anthropic-back-one-ai-training-opt-out/