AI crawlers: what each one is for, and how to block or allow it

Search engines whose index also feeds AI answers

Googlebot, Bingbot and PetalBot are search-engine crawlers first, and each one's operator says the same crawl serves its AI answers. Google says AI is built into Search, so there is no separate robots.txt token for AI Overviews or AI Mode: a Disallow for Googlebot stops Google crawling those pages for Google Search itself, and Google-Extended does not affect Search. Microsoft says Bing search, Copilot and grounding results share one crawl and index, and points to NOINDEX, NOARCHIVE and NOCACHE page directives, not robots.txt, to limit what Copilot uses. To stay in these search engines but out of their AI answers, the controls are page-level, and each crawler's page quotes them. The generator never names these 3 tokens unless you choose them one by one.

Tokens that may do nothing

anthropic-ai, Claude-Web, Google-NotebookLM, FacebookBot and cohere-ai still turn up in robots.txt files, but the current crawler documentation of Anthropic, Google, Meta and Cohere does not list them as a current token, so a line naming one may do nothing. Each page says when we checked, quotes the operator where it calls a token former or replaced, and names the tokens that operator documents now. The checker badges any group in your file that names one.

How to read this directory

• Obeys robots.txt is what the operator says. "Not always" means the operator documents an exception, usually for fetches a user asked for.
• Verify: anyone can send a user-agent string. An operator's published IP list or reverse-DNS check is how you tell the real crawler from an impostor. Where an operator publishes neither, the entry says so.
• Unconfirmed by the operator means the operator's own documentation does not state the fact, or its page refused our fetch. We do not fill gaps from third-party lists.
• Tokens such as Google-Extended, Applebot-Extended and YandexAdditional never visit your site; they are rules the operator reads to decide how its other crawlers' data may be used.
• A Not in … docs badge marks a token the operator's current crawler documentation does not list, as of the date on its page. "Retired" is used only where the operator itself says so.

Use the data

The whole directory is one JSON file: resulthack.com/crawlers.json. Each record has the token, operator, purpose, what it does, robots.txt behaviour, verification method and IP list, block effect, the operator's sentences quoted word for word, and every source with the date we checked it.