resulthack

All AI crawlers /

Google-Extended

Google · Training AI search · robots.txt: Yes

A robots.txt control token, not a separate crawler: it decides whether content Google has already crawled may be used to train Gemini models and to ground Gemini's answers.

Block Google-Extended

User-agent: Google-Extended
Disallow: /

Blocking Google-Extended opts your content out of Gemini training and Gemini grounding. Google says it does not affect your inclusion or ranking in Google Search.

Allow Google-Extended

User-agent: Google-Extended
Allow: /

An allow group only changes anything if a broader rule would block Google-Extended. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.

Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.

Block or allow it at the server

No server rule for Google-Extended. Google says: Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity. (Google: List of Google's common crawlers, checked 2026-09-22). A server rule acts on requests, and Google documents Google-Extended as a robots.txt control with no requests of its own to match, so the robots.txt lines above are the way to use it.

The fetching is done by Google's common crawlers.

Who blocks it

Of the 1,000 most-visited domains on Tranco list PY35J (generated 2026-10-08), we read 508 robots.txt files on 2026-10-09 and ran each through the same RFC 9309 matcher as the checker, for the path /. 68 block Google-Extended from the whole site by name, 22 block it through their User-agent: * group, and 25 name it with partial rules (some paths closed, the home page open). 17 more name it only to let it in.

68

block it by name 13.4%

22

block it through * 4.3%

25

name it, partial rules 4.9%

Percentages are of the 508 files we could read, not of all 1,000 sites.

Highest-ranked sites that block Google-Extended by name

  1. fbcdn.net rank 12
  2. twitter.com rank 16
  3. amazon.com rank 27
  4. whatsapp.net rank 32
  5. x.com rank 48
  6. whatsapp.com rank 57
  7. tiktok.com rank 58
  8. yahoo.com rank 61
  9. msn.com rank 72
  10. chatgpt.com rank 87

Each links to the site's live robots.txt, which may have changed since 2026-10-09.

robots.txt is only what a site asks for. A site can also turn a crawler away at its firewall or CDN, or let an operator in under a licensing deal, and neither shows in its robots.txt. A block through * stops every crawler the file does not name, so it says nothing about this one in particular. The files we could not read are not counted: 222 of the 1,000 domains are infrastructure names with no website at the bare domain (CDN and DNS zones, for example), 59 answered our request with 401 or 403, 93 failed (timeout, TLS or server error), 59 have no robots.txt (404 and similar) and 59 answered with an HTML page instead of a robots.txt file. Every agent's count, side by side.

Facts

Operator
Google
Purpose
Model training + AI search index

Google says the token controls both training of future Gemini models and grounding: giving Gemini content from the Google Search index at prompt time (in Gemini Apps and Grounding with Google Search on Vertex AI).

Obeys robots.txt
Yes. Google-Extended exists only as a robots.txt token; Google says its common crawlers always obey robots.txt rules when crawling automatically.
In your server logs
None of its own: Google says there is no separate HTTP user-agent string for Google-Extended; crawling is done with Google's existing user agents.
How to verify it
There is no Google-Extended traffic to verify. For the Google crawlers that do the fetching, Google says to run a reverse DNS lookup on the IP, check the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP; or match the IP against the published common-crawlers list.

Published list: developers.google.com/static/crawling/ipranges/common-crawlers.json — 170 IPv4 and 147 IPv6 ranges when we fetched it, list dated 23 Sep 2026. Always match against the live list; operators update them. To count real and fake Google-Extended requests in your own server log against this list, drop the log into the log reader.

What Google says

Quoted word for word from the operator's documentation.

“is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.”

“Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.”

“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

“They always obey robots.txt rules when crawling automatically.”

“The common crawlers generally crawl from the IP ranges published in the common-crawlers.json object, and the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.”

“Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”

“Verify that it's the same as the original accessing IP address from your logs.”

Sources

Other crawlers from Google

← All 49 AI crawlers

Made with AI tools. · Privacy