Diffbot-User
Diffbot · User fetch · robots.txt: Unconfirmed
Requests made on behalf of a person browsing a URL with Diffbot software, in response to their input.
Block Diffbot-User
User-agent: Diffbot-User
Disallow: /Unconfirmed by the operator Diffbot's documentation we checked does not say what blocking Diffbot-User changes beyond the crawler itself.
Allow Diffbot-User
User-agent: Diffbot-User
Allow: /An allow group only changes anything if a broader rule would block Diffbot-User. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks Diffbot-User on a given path, and which line decides.
Facts
- Operator
- Diffbot
- Purpose
- User-requested fetch
- Obeys robots.txt
- Unconfirmed by the operator
- In your server logs
- Unconfirmed by the operator Diffbot's documentation we checked does not give its user-agent string.
- How to verify it
- Unconfirmed by the operator Diffbot's documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be Diffbot-User cannot be verified against the operator.
What Diffbot says
Quoted word for word from the operator's documentation.
“This is used by requests originating on behalf of a human user browsing a URL using Diffbot software, in response to their input.”
“To whitelist Diffbot for a site, specify the appropriate Diffbot user-agents in the site’s robots.txt.”
Sources
- Diffbot Docs: Does Diffbot respect robots.txt? — fetched and checked 2026-09-24