resulthack

All AI crawlers /

Google-CloudVertexBot

Google · Unconfirmed · robots.txt: Yes

Crawls a site at that site owner's request, for building Vertex AI Agents on Google Cloud.

Block Google-CloudVertexBot

User-agent: Google-CloudVertexBot
Disallow: /

Google says rules for Google-CloudVertexBot affect only crawls a site owner requested for building Vertex AI Agents, and have no effect on Google Search or other products.

Allow Google-CloudVertexBot

User-agent: Google-CloudVertexBot
Allow: /

An allow group only changes anything if a broader rule would block Google-CloudVertexBot. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.

Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.

Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks Google-CloudVertexBot on a given path, and which line decides.

Facts

Operator
Google
Purpose
Unconfirmed by the operator

Google says it crawls when a site's owner asks, to build Vertex AI Agents; it is not a general crawl for training or search.

Obeys robots.txt
Yes. It is one of Google's common crawlers, which Google says always obey robots.txt when crawling automatically. Google lists two robots.txt tokens for it, Google-CloudVertexBot and Googlebot, and says a crawler needs to match only one of its tokens for a rule to apply — so a Googlebot group can govern it too. This site's checker applies RFC 9309 and only reads the Google-CloudVertexBot token.
In your server logs
Google gives only a substring: the user-agent contains "Google-CloudVertexBot".
How to verify it
Google says to run a reverse DNS lookup on the IP, check the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP; or match the IP against the published common-crawlers list.

Published list: developers.google.com/static/crawling/ipranges/common-crawlers.json — 170 IPv4 and 147 IPv6 ranges when we fetched it, list dated 23 Sep 2026. Always match against the live list; operators update them. To count real and fake Google-CloudVertexBot requests in your own server log against this list, drop the log into the log reader.

What Google says

Quoted word for word from the operator's documentation.

“Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents.”

“It has no effect on Google Search or other products.”

“Some crawlers have more than one user agent token; you need to match only one crawler token for a rule to apply.”

“They always obey robots.txt rules when crawling automatically.”

“User-Agent substring in HTTP requests Google-CloudVertexBot”

“The common crawlers generally crawl from the IP ranges published in the common-crawlers.json object, and the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.”

“Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”

“Verify that it's the same as the original accessing IP address from your logs.”

Sources

Other crawlers from Google

Same purpose, other operators

← All 39 AI crawlers