# One robots.txt signal to opt out of AI training: what to change on your site

> OpenAI, Google, Microsoft and Anthropic will read __usage_ai-training, one robots.txt group to decline AI training. Syntax, dates, and what not to block.

URL: https://geosnap.ai/en/blog/segnale-unico-robots-txt-addestramento-ai-cosa-fare-sul-tuo-sito
Language: English
Version IT: https://geosnap.ai/blog/segnale-unico-robots-txt-addestramento-ai-cosa-fare-sul-tuo-sito
Author: Rinald Sefa, CMO Geosnap · Category: Guides
Publisher: Geosnap (Maind Group S.r.l.), https://geosnap.ai

## In short

On October 30, 2026 OpenAI, Google, Microsoft and Anthropic announced they will honor a shared robots.txt signal: a User-Agent: \_\_usage\_ai-training group that, with the usual Disallow and Allow lines, says whether your content may train generative models. It covers training only: it does not remove you from search or AI answers, but Google will read it as a Google-Extended block unless you name that token, and Google-Extended also governs Gemini citations. It arrives between December 2026 and early 2027: you can write it today, but decide separately what to do about search and grounding.

- The \_\_usage\_ai-training token uses the robots.txt syntax you know: Disallow means "no training", Allow means "yes", and the User-Agent: \* group does not count.
- The signal is about training only: search crawlers (Googlebot, Bingbot, OAI-SearchBot) must stay open if you want to be cited.
- At Google, the token without Google-Extended equals a Google-Extended Disallow, which also blocks grounding in Gemini: choose explicitly.
- Bing separates training and grounding with ai-training and ai-grounding tags; sites using noarchive or nocache have until March 1, 2027.
- The dates: Anthropic already started, OpenAI in the coming months, Microsoft from February 1, 2027, Google in early 2027. Current rules remain valid.

Written by the Geosnap AI agent, reviewed and approved by Rinald Sefa.

Translated from the Italian original. [Read the original](https://geosnap.ai/blog/segnale-unico-robots-txt-addestramento-ai-cosa-fare-sul-tuo-sito)

On October 30, 2026, four companies that train AI models said the same thing on the same day: they will honor a single robots.txt signal that tells them whether a site's content may be used to train generative models. Until yesterday you had to know the name of every crawler (GPTBot, Google-Extended, ClaudeBot and so on) and write a rule for each. Here is what was announced, how to write the signal, when it will be read and, above all, what you must not block if you want to keep showing up in ChatGPT, Gemini and Google AI Mode answers.

## What did OpenAI, Google, Microsoft and Anthropic announce?

That they will all read the same token in robots.txt: `__usage_ai-training`. The signal is described in a draft submitted to the IETF by Kevin Kelley of Anthropic, ["Machine-Readable Training Preference Signal"](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/), dated October 30, 2026 and expiring May 3, 2027. The draft states that it introduces "no new directive names and no new grammar": it uses robots.txt groups, paths and precedence exactly as they are. It is not a standard yet: the IETF datatracker lists it as an individual submission with no formal standing, and the AI Preferences working group is still debating it.

The commitments arrived within hours. Google updated its [official crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) on October 30 with a note: it intends to support the AIPREF signal with `user-agent: __usage_ai-training` "in early 2027". Microsoft, as [Search Engine Roundtable](https://www.seroundtable.com/microsoft-bing-ai-controls-42429.html) reports, will start honoring it on February 1, 2027, together with new controls that separate training from grounding. OpenAI, according to [Unite.AI](https://www.unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences/), will implement it "in the coming months" and says its existing GPTBot and OAI-SearchBot controls remain in force. Anthropic, which wrote the draft, has already started implementing it, as [Relevant Audience reports](https://www.relevantaudience.com/geo/google-shared-ai-training-robots-txt-signal-2027/) from Kelley's email to the working group. Google's John Mueller added, speaking to [Search Engine Roundtable](https://www.seroundtable.com/google-ai-training-control-42430.html), that the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the AI Act, which recognizes robots.txt as a primary method for machine-readable preferences.

## How do you write the signal in robots.txt?

Like any other group, with the token in place of the crawler name. This is the draft's own example:

```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```

It means: a preference against training for the whole site, except the `/press-kit/` folder, where training is welcome. `Disallow` expresses a "no", `Allow` a "yes", with the longest-match rule you already know. Four details in the [draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) change the outcome:

- The `User-Agent: *` group does not apply to the token. If your rules only live under the asterisk, you have expressed nothing for this signal.
- If the file has no group with the token, you have expressed no preference. That is not an implicit "yes": it is silence.
- It is not a crawl directive. A crawler may keep fetching your pages and simply not use them for training. To limit crawling you still need groups named after the crawler.
- Specific expressions take precedence over general ones, and private agreements (a content license, say) may override the signal.

The token sits alongside the rules you already have. You can keep your `GPTBot` and `Google-Extended` groups and add the shared one: whoever reads it will apply the most specific rule.

## Does blocking training remove you from AI answers?

No, if you only block training. Yes, if you also block search crawlers or grounding. These are three different uses of your content and they need to be kept apart:

| Use | What it decides | How you control it |
| --- | --- | --- |
| Crawling for search | Whether your pages enter the index that citations and clicks come from | robots.txt groups named after the crawler (Googlebot, Bingbot, OAI-SearchBot) |
| Grounding | Whether an AI answer may read your page at the moment someone asks | Google-Extended for Gemini; Bing's new `ai-grounding` tags; Search Console controls for Google |
| Training | Whether your text ends up in the weights of future models | `__usage_ai-training` plus provider tokens (GPTBot, Google-Extended, Applebot-Extended) |

The tricky part is Google. Google-Extended is not a crawler but a control token and, according to the [official documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), it governs two things at once: training of future Gemini models and grounding, meaning the use of your content in Gemini Apps and Vertex AI answers. It does not affect inclusion in Google Search or ranking. Mueller told [Search Engine Roundtable](https://www.seroundtable.com/google-ai-training-control-42430.html) that if you use the shared token without naming Google-Extended, Google will read it "as if the publisher had disallowed Google-Extended". In plain terms: a generic "no to training" can, at Google, also cost you citations in Gemini. If you want to say no to training but yes to Gemini answers, you need an explicit Google-Extended group with `Allow`, accepting that at Google the two uses travel together. Google AI Mode and AI Overviews are not mentioned in the announcement.

Microsoft made the opposite choice and separated them. Since [October 30](https://www.seroundtable.com/microsoft-bing-ai-controls-42429.html) Bing offers three preferences: AI Training, AI Grounding and a middle option that allows grounding only in Microsoft products (Copilot and Bing's AI answers) but not in grounding services sold to third parties. Grounding is set page by page with the Bingbot meta tag or the X-Robots-Tag header, for example `<meta name="Bingbot" content="ai-grounding:disallow">`, while training can be declared site-wide with the shared token in robots.txt, as [Relevant Audience reconstructs](https://www.relevantaudience.com/geo/bing-ai-grounding-controls-replace-noarchive-nocache/) from Microsoft's announcement. Microsoft's warning is clear: blocking grounding significantly reduces visibility in AI experiences, while classic Bing results do not change.

## When does it take effect, and what happens to today's rules?

Between December 2026 and early 2027, on a different calendar for each provider. Today's rules stay valid: nobody has announced it will stop reading GPTBot, Google-Extended or the other tokens.

| When | Who | What happens |
| --- | --- | --- |
| October 30, 2026 | Anthropic | Started implementing the signal described in its draft |
| "In the coming months" | OpenAI | Token implementation; GPTBot and OAI-SearchBot keep working |
| December 15, 2026 | Microsoft | Temporary grounding preferences appear in Bing Webmaster Tools for sites using NoArchive or NoCache |
| Early 2027 | Google | Token support, read as a Google-Extended Disallow unless you specify it |
| February 1, 2027 | Microsoft | Bing starts honoring AI Training and AI Grounding as it crawls |
| March 1, 2027 | Microsoft | NoArchive and NoCache no longer count as AI preferences |
| September 30, 2027 | Microsoft | Temporary grounding preferences expire |

The Microsoft dates are those reported by [Search Engine Roundtable](https://www.seroundtable.com/microsoft-bing-ai-controls-42429.html). If your site uses `noarchive` or `nocache` to keep content out of Copilot, you have a real deadline: by March 1, 2027 you must rewrite them with the new tags, otherwise Bing falls back to "Allow" on the next crawl.

Then there is the CDN layer. Cloudflare explained on [September 15, 2026](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) that its "Disallow AI Training" setting publishes a no-training preference in robots.txt through Bot Preference Sync and lets mixed-use crawlers (Googlebot, Bingbot, Applebot) through for search; on its network fewer than 1% of sites block search bots, while 17% have turned on some training block. That post came six weeks before the shared token: if you use Cloudflare, check over the next few weeks what it actually writes into your robots.txt.

## What to do on your site this week

1. **Make two separate decisions.** Do you want your content to train models? Do you want to appear in AI answers? For a brand that lives on visibility the usual combination is no to the first and yes to the second.
2. **If you want to say no to training, add the shared group.** It costs two lines, is harmless for crawlers that do not know it, and will be read from early 2027. Meanwhile the provider tokens keep working.
3. **Leave the search crawlers alone.** Googlebot, Bingbot, OAI-SearchBot and the Perplexity and Anthropic equivalents must stay open, or you disappear from citations. Which ones exist and what they do is in our [guide to AI crawlers](https://geosnap.ai/en/blog/il-tuo-sito-blocca-chatgpt-guida-crawler-ai).
4. **Choose explicitly for Google-Extended.** `Allow` if you want to stay in Gemini answers, `Disallow` if you would rather give up Gemini than train. Do not let the shared token decide for you.
5. **Search for `noarchive` and `nocache`** in templates and HTTP headers. If you use them, plan the move to the `ai-grounding` tags before March 1, 2027 and verify the site in Bing Webmaster Tools before December 15.
6. **Check what the server really publishes.** Declared and enforced robots.txt often differ, as [the 2026 data shows](https://geosnap.ai/en/blog/blocchi-ai-robots-txt-quanti-siti-li-applicano-davvero-dati-2026). Geosnap's technical analysis also checks the robots.txt of the sites it monitors.

A robots.txt consistent with these choices can look like this:

```
# Search and AI answers: keep the search crawlers open
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
Allow: /

# Training: one signal for every provider that reads it
User-Agent: __usage_ai-training
Disallow: /

# Google: explicit choice (Allow = you stay in Gemini answers,
# but your content may train Gemini models)
User-agent: Google-Extended
Allow: /
```

One last thing to remember: the signal is a voluntary preference, not an enforcement mechanism. Providers that do not take part are not bound by it, and Microsoft writes that no choice "guarantees inclusion, traffic, or payment". But for the first time the four largest providers read the same line, and that makes the thing that matters for visibility much simpler: separating what you do not want (training) from what you do want (being read and cited).

## Sources

- [Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) (IETF Datatracker, 2026-10-30)
- [Google's common crawlers: Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (Google Search Central, 2026-10-30)
- [Google Adopts New AI Training Control](https://www.seroundtable.com/google-ai-training-control-42430.html) (Search Engine Roundtable, 2026-10-30)
- [Microsoft Bing Separates AI Training From AI Search Grounding With New Controls](https://www.seroundtable.com/microsoft-bing-ai-controls-42429.html) (Search Engine Roundtable, 2026-10-30)
- [OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences](https://www.unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences/) (Unite.AI, 2026-10-30)
- [Bing replaces NoArchive and NoCache with separate AI training and grounding controls](https://www.relevantaudience.com/geo/bing-ai-grounding-controls-replace-noarchive-nocache/) (Relevant Audience, 2026-10-31)
- [Google plans to honour a shared AI training opt-out in robots.txt from 2027](https://www.relevantaudience.com/geo/google-shared-ai-training-robots-txt-signal-2027/) (Relevant Audience, 2026-10-31)
- [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) (Cloudflare, 2026-09-15)

## Frequently asked questions

### Does the \_\_usage\_ai-training token work today?

Anthropic says it has started implementing it; OpenAI talks about "the coming months", Microsoft will honor it from February 1, 2027 and Google in early 2027. You can add it now: crawlers that do not know it ignore it, and the provider tokens you already use keep working.

### If I add the token, do I disappear from ChatGPT, Gemini and Google AI Mode?

No, as long as you keep the search crawlers open (OAI-SearchBot, Googlebot, Bingbot). The exception is Google: if you do not name Google-Extended, the token is read as a Google-Extended Disallow, which also governs grounding in Gemini answers. To stay in Gemini, write a Google-Extended group with Allow.

### Should I remove my GPTBot, Google-Extended or ClaudeBot rules?

No. No provider has announced it will drop its own tokens, and the IETF draft says specific expressions take precedence over general ones. The shared group is an addition, not a replacement.

### I use noarchive or nocache: what changes?

A lot, for Bing. From March 1, 2027 those tags no longer count as AI preferences: you need to move to the new ai-training and ai-grounding tags and, if you want to keep the block during the transition, record a temporary preference in Bing Webmaster Tools from December 15, 2026.

### Is this an official standard?

Not yet. It is an individual draft submitted to the IETF on October 30, 2026, expiring May 3, 2027, under discussion in the AI Preferences working group. It matters because the four largest providers said they will honor it, not because it has been approved.