resulthack

All AI crawlers /

Google-Extended

Google · Training AI search · robots.txt: Yes

A robots.txt control token, not a separate crawler: it decides whether content Google has already crawled may be used to train Gemini models and to ground Gemini's answers.

Block Google-Extended

User-agent: Google-Extended
Disallow: /

Blocking Google-Extended opts your content out of Gemini training and Gemini grounding. Google says it does not affect your inclusion or ranking in Google Search.

Allow Google-Extended

User-agent: Google-Extended
Allow: /

An allow group only changes anything if a broader rule would block Google-Extended. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.

Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.

Facts

Operator
Google
Purpose
Model training + AI search index

Google says the token controls both training of future Gemini models and grounding: giving Gemini content from the Google Search index at prompt time (in Gemini Apps and Grounding with Google Search on Vertex AI).

Obeys robots.txt
Yes. Google-Extended exists only as a robots.txt token; Google says its common crawlers always obey robots.txt rules when crawling automatically.
In your server logs
None of its own: Google says there is no separate HTTP user-agent string for Google-Extended; crawling is done with Google's existing user agents.
How to verify it
There is no Google-Extended traffic to verify. For the Google crawlers that do the fetching, Google says to run a reverse DNS lookup on the IP, check the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP; or match the IP against the published common-crawlers list.

Published list: developers.google.com/static/crawling/ipranges/common-crawlers.json — 170 IPv4 and 147 IPv6 ranges when we fetched it, list dated 22 Sep 2026. Always match against the live list; operators update them.

What Google says

Quoted word for word from the operator's documentation.

“is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.”

“Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.”

“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

“They always obey robots.txt rules when crawling automatically.”

“The common crawlers generally crawl from the IP ranges published in the common-crawlers.json object, and the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.”

“Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”

“Verify that it's the same as the original accessing IP address from your logs.”

Sources

← All 19 AI crawlers