Google to Support a Shared AI Training Control in robots.txt

November 2, 2026. On 30 October 2026 Google announced plans to support "a shared AI training control in robots.txt," a single "policy token" that tells participating AI companies at once whether a site's content may be used to train models. Google calls it a draft proposal, aims to support it "along with other AI providers in 2027," and its crawler documentation names a robots.txt user-agent for the signal, "user-agent: __usage_ai-training," for "early 2027." The detail that matters for a business site is the default Google chose: if a site uses the shared token to opt out and says nothing about Google-Extended, "Google plans to interpret that as if the publisher had disallowed Google-Extended." Google-Extended governs both training of Gemini models and grounding, the step where Gemini pulls pages from the Search index into an answer, and Google's post adds that "websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So a blanket training opt-out, left alone, would also take a site out of grounding in Gemini Apps. The surfaces at stake are Gemini Apps and Vertex AI grounding; Google's documentation says Google-Extended does not affect inclusion in Google Search, and says nothing about AI Overviews or AI Mode. Here is what was announced, what Google-Extended already does, and how to set the two so you keep the visibility you want.

What Google announced on 30 October

The post on the Google Search Central Blog, posted by John Mueller, Google's Search Advocate, describes a control "based on work from the AIPREF working group at the Internet Engineering Task Force (IETF)." Instead of writing a separate robots.txt group for every AI company, "website owners will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers." The announcement positions it as an addition, not a replacement: "existing company-specific controls will continue to function alongside this new joint signal," so a site can "express a broad choice using the industry signal, while also tailoring their instructions to individual AI developers like Google through controls such as Google-Extended."

Google frames the move as a response to "publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training," and says the proposal "is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences." The timeline is the soft part: "We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress."

What Google-Extended already controls

Google's crawler documentation is precise about the existing token, which Google has offered since 2023. Google-Extended "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." It lets publishers "manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI."

Two sentences in that documentation decide most business cases. First: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking it costs nothing in Google Search inclusion or ranking, per the documentation, which does not mention AI Overviews or AI Mode either way, so it is not a promise about those surfaces. Second, training and grounding travel together under one token: the documentation describes one token for both uses and no separate switch that would keep a site out of training while leaving it in grounding. The same page already carries the note that "Google intends to support the AIPREF shared machine-readable training preference signal (with 'user-agent: __usage_ai-training') in early 2027."

The default that changes the Gemini math

Put the two documents together. A site that writes the shared opt-out token into robots.txt and leaves Google-Extended unmentioned would be, in Google's words, interpreted "as if the publisher had disallowed Google-Extended." Disallowing Google-Extended removes the site from training and from grounding in Gemini Apps and Vertex AI. And the post is explicit that eligibility to "appear in responses and receive clicks via grounding in Gemini Apps" belongs to sites that "specifically allow Google-Extended."

That is the trade a small business site would be making in 2027 if it adopts the industry opt-out without a second line. The intent, keeping your pages out of model training, is reasonable. The side effect, disappearing from Gemini's grounding sources, is the opposite of what a business chasing AI-search visibility wants. We cover the difference between being cited and being ranked in AEO vs SEO, and the tradeoffs by crawler in should you block AI crawlers.

Three robots.txt postures for 2027

Because company-specific groups keep working beside the shared token, the combinations are small. Google's post describes two of them in words ("opt out across all providers, or to pick and choose"); the third is the status quo.

The third row is the one Google's post exists to make possible. Its sentence is: "So website owners can easily express a broad choice using the industry signal, while also tailoring their instructions to individual AI developers like Google through controls such as Google-Extended." The exact syntax of the shared token's robots.txt line is not in the announcement; the crawler documentation gives only the user-agent name. Wait for the published spec before editing a live file.

What to do before the token goes live

1. Write down what you want from each AI surface. Training and grounding are one switch at Google today. If Gemini citations bring you enquiries, you want Google-Extended allowed, whatever you do about training elsewhere.
2. Check who currently blocks what. A CDN setting can block AI crawlers above your robots.txt; our guide to Cloudflare's AI bot settings covers how to check it.
3. Keep the explicit Google-Extended group if you adopt the shared opt-out. The default only applies when "Google-Extended is not explicitly specified," so one extra robots.txt group, 1 in all, preserves your choice.
4. Do not expect a ranking effect either way. Google's documentation says Google-Extended neither affects inclusion in Search nor acts as a ranking signal.
5. Watch the IETF draft. The post says the "AI training definition developed at the IETF" ships first "while multistakeholder discussions continue on the rest of the AIPREF vocabulary," so more tokens may follow.

If you want your pages to be the ones an assistant cites, most of the work is on the page rather than in robots.txt: clear answers, dated facts and sources an engine can verify. That is what our AI search optimization service works to get you named for, and what does llms.txt work and how to correct an AI Overview cover from the publisher's side.