How do you block AI crawlers from training on your website content?

To block AI crawlers from training on your content today, add a Disallow rule to your robots.txt file for each company's training token: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, Applebot-Extended for Apple and CCBot for Common Crawl. A new shared token, usage_ai-training written with two underscores in front, carries the same message to several companies at once: Anthropic honours it already, Microsoft starts on February 1, 2027, Google plans support in 2027 and OpenAI has said it will follow.

The traffic behind the question keeps growing. Cloudflare reported that 52% of crawler requests on its network were for AI training as of June 2026, up from 22% in spring 2025. The hard part is saying no to training without saying no to search, because OpenAI and Anthropic run separate bots for each job, and blocking the wrong one takes you out of their AI search answers.

What robots.txt can and cannot do

Robots.txt is a plain text file at the root of your domain, for example yoursite.com/robots.txt, that tells crawlers which paths they may fetch. Rules sit in groups, and each group starts with the name of the bot it applies to. The standard behind it, RFC 9309 from September 2022, says plainly that "these rules are not a form of access authorization." It is a posted request, and it only works on bots whose owners agree to read it.

The large AI companies publish the bot names they honour, so a rule against those names works for those companies. A scraper that ignores the file, or borrows another bot's name, walks straight past it. Common Crawl itself warns that some crawlers falsely identify themselves as CCBot.

The second limit is timing. A new rule affects what gets collected from the day you publish it. Anthropic's own wording is that disallowing ClaudeBot signals that a site's "future materials" should be left out of training, and nothing in robots.txt pulls pages back out of datasets that already exist. Treat the file as the first layer rather than the only one, with a firewall or CDN rule as the backstop for bots that ignore it.

The AI training tokens to block today

Each company uses its own name, and the token that controls training is not always the bot that does the crawling. Start with these five.

To keep all five away from your whole site, add these groups below your existing rules:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

You don't have to block everything. Change Disallow: / to Disallow: /blog/ and only the blog is withheld. Product and service pages stay available to every bot. Leave your Googlebot and Bingbot groups exactly as they are, since those control ordinary search.

Keep the AI search bots open

Blocking training and blocking AI search are two separate decisions, and the bot names make them easy to mix up. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers", while a GPTBot block only signals that content should stay out of training. Anthropic splits the same way: ClaudeBot for training, Claude-SearchBot for search and Claude-User for pages fetched when someone asks Claude a question.

Visits that a person triggers follow different rules. OpenAI notes that robots.txt rules "may not apply" to ChatGPT-User, because the visit starts with a user's question rather than an automated crawl.

Google draws its line in another place. AI Overviews and AI Mode sit inside Search, so Google points site owners to Googlebot rules and to the nosnippet, data-nosnippet, max-snippet and noindex controls to limit what those features show. Google-Extended handles Gemini training and grounding in Google's other systems.

Cloudflare's numbers show how site owners treat the two jobs: fewer than 1% of sites on its network block search bots, while 17% switch on some way of blocking training. For a business that wants to be found, that split is the right default. Stay open to search, and close training where the content is what you sell.

How the shared AI training opt-out works

On October 30, 2026, Kevin Kelley of Anthropic published an IETF Internet-Draft that defines one robots.txt name for training preferences: two underscores followed by usage_ai-training. It adds no new directives. You write an ordinary group that starts with User-Agent and that name, and each participating company reads it.

The draft's own example has three lines: the User-Agent line with the shared token, then Disallow: / and then Allow: /press-kit/. That file says no to AI training across the site except the press kit folder. The token describes how content may be used rather than whether a bot may fetch it, so crawling is unchanged. If no group names it, no preference has been expressed.

Support arrives on different dates:

• Anthropic honours the token now, and a rule that names ClaudeBot directly takes priority over it.
• Google said in its Search Central announcement that it will read a training opt-out as if you had disallowed Google-Extended, unless you name Google-Extended yourself. Support is planned for 2027.
• Microsoft begins honouring it as Bing crawls from February 1, 2027, and Bing's guidance asks sites that use NoArchive or NoCache today to publish their training preference by March 1, 2027.
• OpenAI aims to add support in the coming months. GPTBot rules still apply until then.

Two caveats matter. The draft is an individual submission that the IETF has not approved, and it expires on May 3, 2027 unless it is updated. And specific rules beat general ones: if the shared token disallows training but your file explicitly allows Google-Extended, Google follows the Google-Extended rule. Use that on purpose or not at all.

Pros and cons of blocking AI training

What you gain. Your preference is on record with every company that publishes a training token, and it costs nothing to publish. Search rankings stay where they are: Google says Google-Extended is not used as a ranking signal, and Microsoft says the new preferences do not change eligibility for traditional Bing results. You can also work folder by folder, so a paid guide stays protected and the press page stays public.

What you give up, or don't get. Four limits apply:

• Pages already collected stay in existing datasets.
• Scrapers that ignore robots.txt are not stopped, so those need firewall or CDN rules.
• Blocking Google-Extended also takes you out of grounding in Gemini Apps, where Google says sites that allow it can appear in responses and receive clicks.
• No money follows. Microsoft states that the preferences "do not establish licensing or payment arrangements."

Who should block AI crawlers from training

Block training if your content is the product. Publishers, course creators, researchers, photographers, designers and anyone selling paid guides have the most to lose when a model can reproduce their work without sending a visitor. For them, the five tokens plus the shared token are worth adding this week.

A ten-page site for a plumbing firm or an accountancy practice is a different case. It has little original material to protect and more to gain from being visible wherever people ask questions. In our view, sites like these should keep every search bot open and block training only on original work, such as a /guides/ folder.

A common mixed case is an agency, software company or online shop with service pages it wants seen and a blog or resource library it paid to produce. Block training on that library, keep the rest open, and check the result once Google and Microsoft switch on the shared token in 2027.

A six-step robots.txt setup

1. Open yoursite.com/robots.txt in a browser and save a copy of what is live now.
2. Find out what writes the file. If a CMS plugin or your CDN manages it, make your changes there so they are not overwritten.
3. Add the five training groups, for the whole site or only the folders you want to protect.
4. Add a group for the shared token, two underscores then usage_ai-training, with the same paths. Anthropic reads it today, and Google and Microsoft pick it up in 2027.
5. Confirm that Googlebot, Bingbot, OAI-SearchBot and Claude-SearchBot are not blocked anywhere in the file.
6. If you use NoArchive or NoCache for Bing, finish step 4 before March 1, 2027.

Each edit takes minutes, and one wrong Disallow line can quietly drop a site from search. If you would like the Techno Mantra team to review your robots.txt before the 2027 changes land, email info@technomantraa.com.