Google plans a shared AI training robots.txt control: what HK SMEs should do now

On 30 October 2026 the Google Search Central Blog announced plans to support a shared AI training control in robots.txt. The approach is based on work from the IETF AIPREF working group. The idea is that a website can use one shared policy token to express training preferences across participating AI providers. Google said it aims to support the new control with other AI providers in 2027 and will share progress updates.

Why this matters for Hong Kong SMEs

Most small Hong Kong business sites do not edit robots.txt every week. Still, “may this content be used to train AI models?” and “may this content appear in Google Search or AI summaries?” are different questions. The 30 October post is a forward-looking plan, not a same-day rule change. The Google-specific control that already works today is the Google-Extended token in robots.txt. Separating the two controls helps you avoid cutting Search visibility by mistake—or assuming nothing needs a decision.

How the shared control is described

According to Google’s blog post, site owners will be able to use a shared policy token in robots.txt for broad preferences across participating AI providers. When a site uses that token to opt out of training and Google-Extended is not explicitly specified, Google plans to interpret the choice as if Google-Extended were disallowed. Existing company-specific controls will keep working alongside the joint signal, so a site can set a broad industry preference and still fine-tune Google via Google-Extended. Google also states that sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

What still applies today: Google-Extended

Google’s common-crawlers documentation lists Google-Extended as a standalone product token (it does not use a separate HTTP user-agent string). Publishers can use it to manage whether content Google crawls from their sites may be used to train future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google states that Google-Extended does not impact inclusion in Google Search and is not used as a ranking signal. The same crawler page notes Google’s intent to support the AIPREF shared machine-readable training preference signal (documented example: user-agent: __usage_ai-training) in early 2027.

Do not confuse it with Search Console’s Search generative AI control

The Search generative AI control in Search Console settings manages whether your site’s links and content can appear in generative AI features on Google Search, including AI Overviews, AI Mode, and generative AI features in Discover. Google says the control was rolled out to all websites worldwide as of 31 August 2026. It is not used as a ranking or inclusion signal for other parts of Search, and it does not affect AI training—Google points publishers to Google-Extended to limit training of models used to generate responses in those Search generative AI features. To keep a page out of Google Search entirely, use indexing controls such as noindex, not Google-Extended alone.

Practical checklist for Hong Kong SMEs

• Open your site’s robots.txt (usually https://your-domain/robots.txt) and check whether a User-agent: Google-Extended group exists and what Allow/Disallow rules it uses.
• Decide separately: whether content may be used for Gemini-related training/grounding, versus whether content may appear in AI Overviews, AI Mode, or Discover generative features—these use different controls.
• To limit Google-side AI training / some grounding uses, follow Google’s crawler documentation for Google-Extended robots.txt rules, then allow time for recrawl and processing.
• To adjust visibility only in Search generative AI features, use Search Console → Settings → Search generative AI; do not assume Google-Extended alone opts you out of AI Overviews.
• Do not use a Googlebot Disallow or a site-wide noindex as a substitute for an AI-training preference—those affect ordinary crawling or indexing more broadly.
• Record your policy choice and change date. Until the shared policy token is supported in practice, rely on Google-Extended and Search Console, and watch Google’s 2027 progress updates.

robots.txt pattern notes (structure only)

The following only illustrates the group structure commonly shown in Google’s documentation. It is not a one-size-fits-all policy. Choose Allow/Disallow paths to match your content policy and hosting setup; back up the current robots.txt before editing, and avoid blocking Googlebot by accident.

• To restrict Google-Extended: use a separate User-agent: Google-Extended group with Disallow: / (or Disallow for selected paths).
• To allow Google-Extended: use a separate User-agent: Google-Extended group with Allow: / (or Allow for selected paths).
• Shared AIPREF token: Google targets support around 2027; do not assume a draft token is already enforced universally—follow the latest Search Central and crawler documentation.

What not to do

• Do not treat “opt out of AI training” as “opt out of Google Search”; Google states Google-Extended does not affect Search inclusion or ranking.
• Do not rush to add an unverified new token because of rumours—the shared control is still a planned rollout with timing subject to Google’s updates.
• Do not promise that changing robots.txt or Search Console will raise or lower traffic, AI visibility, or sales; Google only documents the scope of each control and typical propagation delays (for example, generative AI control changes often take a few days).

Short takeaway

The 30 October 2026 announcement moves Google toward a cross-provider AI training preference expressed in robots.txt, with support targeted around 2027. For Hong Kong SMEs today, the practical steps are to review Google-Extended, keep it distinct from the Search Console generative AI control, and set each according to your policy on training versus Search AI visibility. Primary sources: Google Search Central Blog, Google common-crawlers documentation, Search Console’s Search generative AI control help, and the AI features guide. This article does not guarantee ranking, traffic, or business outcomes.