resulthack

All AI crawlers /

Google-Agent

Google · User fetch · robots.txt: No

Used by AI agents running on Google infrastructure to navigate the web and take actions when a user asks them to.

Block Google-Agent

User-agent: Google-Agent
Disallow: /

Unconfirmed by the operator Google's documentation we checked does not say what blocking Google-Agent changes beyond the crawler itself.

Allow Google-Agent

User-agent: Google-Agent
Allow: /

An allow group only changes anything if a broader rule would block Google-Agent. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.

Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.

Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks Google-Agent on a given path, and which line decides.

Facts

Operator
Google
Purpose
User-requested fetch
Obeys robots.txt
No. Google says its user-triggered fetchers generally ignore robots.txt, because a user requested the fetch, and it lists no robots.txt token for Google-Agent. A "User-agent: Google-Agent" group is therefore unlikely to have any effect.
In your server logs
The user-agent string contains "compatible; Google-Agent;", in mobile and desktop variants.
How to verify it
Match the IP against Google’s published user-triggered-agents list. Google says these fetchers’ reverse DNS names end in gae.googleusercontent.com or google-proxy-…google.com, and that Google is also experimenting with Web Bot Auth for this agent.

Published list: developers.google.com/static/crawling/ipranges/user-triggered-agents.json — 12 IPv4 and 8 IPv6 ranges when we fetched it, list dated 23 Sep 2026. Always match against the live list; operators update them. To count real and fake Google-Agent requests in your own server log against this list, drop the log into the log reader.

What Google says

Quoted word for word from the operator's documentation.

“Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.”

“Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”

“(KHTML, like Gecko; compatible; Google-Agent;”

“It uses IP ranges from user-triggered-agents.json.”

“The user-triggered fetchers' reverse DNS mask, depending on whether the fetcher is Google or user owned, matches ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com respectively.”

“Google is also experimenting with the Web Bot Auth protocol”

Sources

Other crawlers from Google

Same purpose, other operators

← All 39 AI crawlers