Claude-User
Anthropic · User fetch · robots.txt: Yes
Fetches a web page when a Claude user's question needs it.
Block Claude-User
User-agent: Claude-User
Disallow: /Blocking Claude-User stops Claude retrieving your pages in answer to a user's question, which Anthropic says may reduce your visibility for user-directed web search.
Allow Claude-User
User-agent: Claude-User
Allow: /An allow group only changes anything if a broader rule would block Claude-User. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Facts
- Operator
- Anthropic
- Purpose
- User-requested fetch
- Obeys robots.txt
- Yes. Unlike some user-triggered fetchers, Anthropic says Claude-User lets site owners control which sites these user-initiated requests can reach, and that its bots honour robots.txt directives.
- In your server logs
- Unconfirmed by the operator Anthropic's documentation we checked does not give its user-agent string.
- How to verify it
- Anthropic says a crawler whose source IP address is on its published list is coming from Anthropic.
Published list: claude.com/crawling/bots.json — 26 IPv4 ranges when we fetched it, list dated 18 Aug 2026. Always match against the live list; operators update them.
What Anthropic says
Quoted word for word from the operator's documentation.
“Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.”
“Claude-User allows site owners to control which sites can be accessed through these user-initiated requests. Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.”
“Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.”
“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
Sources
- Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? — fetched and checked 2026-09-22
- Anthropic: crawler IP ranges (bots.json) — fetched and checked 2026-09-22