anthropic-ai
Operator unconfirmed · Unconfirmed Not in Anthropic’s docs · robots.txt: Unconfirmed
Not in Anthropic’s current crawler documentation (checked 2026-09-24). A robots.txt line naming anthropic-ai may do nothing: robots.txt only works through the names a crawler answers to, and Anthropic does not document this one.
Anthropic's current tokens: ClaudeBot, Claude-SearchBot and Claude-User. Each one’s page says what it does and whether it follows robots.txt. Keeping the old line next to them costs nothing.
Every fact below is unconfirmed by the operator: Anthropic's documentation does not mention this token, so this page states nothing about what it does. The robots.txt lines below are still correct syntax for it.
Block anthropic-ai
User-agent: anthropic-ai
Disallow: /Unconfirmed by the operator No operator documentation says whether anthropic-ai reads robots.txt at all, so these lines are a request it may or may not honour.
Allow anthropic-ai
User-agent: anthropic-ai
Allow: /An allow group only changes anything if a broader rule would block anthropic-ai. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks anthropic-ai on a given path, and which line decides.
Facts
- Operator
- Unconfirmed by the operator Anthropic's current crawler documentation does not list this token.
- Purpose
- Unconfirmed by the operator
- Obeys robots.txt
- Unconfirmed by the operator
- In your server logs
- Unconfirmed by the operator The documentation we checked does not give its user-agent string.
- How to verify it
- Unconfirmed by the operator The documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be anthropic-ai cannot be verified against the operator.
What Anthropic says
Quoted word for word from the operator's documentation.
“Anthropic uses the Bots described above to access web content.”
Sources
- Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? — fetched and checked 2026-09-24
Where we looked
- support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler — 2026-09-24: Loaded; lists ClaudeBot, Claude-User and Claude-SearchBot. No mention of anthropic-ai.