Google's Gary Illyes stated in a Search Central video published Wednesday that the robots.txt standard governs which URLs a crawler may request rather than what happens to content afterward. Search Engine Journal reports that to keep one AI crawler out while allowing others, site owners must explicitly target that crawler's user-agent token in the file.
Key takeaways
- Gary Illyes explained in a Google video that robots.txt is strictly an access preference protocol defined by RFC 9309.
- Google-Extended is a specific token used to manage whether crawled content trains future Gemini models.
- An IETF Internet-Draft co-authored by Illyes proposes adding a standard Content-Usage rule to robots.txt.
- Cherry Prommawin recommended using percent-encoded form for non-English rule paths to avoid crawler errors.
What does robots.txt control about AI training?
According to Search Engine Journal, the video featured a question about whether robots.txt can allow AI crawling while keeping data out of generative model training. Gary Illyes stated that the protocol does not dictate what a system does with data after crawling it. Blocking a specific AI crawler entirely requires targeting specific user agent tokens explicitly in the file.
The episode demonstrated a fictional directive that parsers would ignore. Google's specification supports User-Agent, Allow, Disallow, and Sitemap elements, meaning invalid lines are ignored. Section 2.2.4 of RFC 9309 lets crawlers decide how to process records outside the protocol scope.
How do google-Extended and draft standards affect webmasters?
Google-Extended acts as a token with no user agent of its own to manage content use for future Gemini models, app grounding, and Google Search on Vertex AI. Search Engine Journal notes this control stems from Google product documentation rather than RFC 9309.
Meanwhile, the IETF draft co-authored by Illyes and Mozilla's Martin Thomson proposes a Content-Usage rule and matching HTTP header to update RFC 9309. This draft is still a Working Group Document and is scheduled to expire on February 20, 2027, unless a new revision replaces it.
What it means for site owners
Webmasters must use percent encoding for paths containing non-English characters or emojis in their rules. Cherry Prommawin advised that although Google handles raw characters by encoding them, older or other web crawlers might get confused. Site owners aiming to restrict AI data use while maintaining search visibility must rely on specific user-agent tokens or wait for standardized usage rules to pass through the IETF approval process.
Frequently asked questions
No, according to Gary Illyes, robots.txt governs which URLs a crawler may request rather than what happens to the content afterward. Blocking a specific AI crawler requires targeting its user-agent token explicitly.
Google-Extended is a robots.txt token that manages whether crawled content can be used to train future Gemini models and for grounding in Gemini Apps and Google Search on Vertex AI.
Google's Gary Illyes co-authored the IETF Internet-Draft with Mozilla's Martin Thomson to give robots.txt a standard way of stating usage preferences.
Cherry Prommawin recommended writing non-English rule paths and emojis in percent-encoded form so that crawlers other than Google do not get confused.
Gary Illyes made these statements in the second episode of The Non-Canonical Cabinet, a Search Central video published on Wednesday by Google Search Relations.
Based on reporting by Search Engine Journal. Read the original article
