Meta-ExternalFetcher
Meta · User fetch · robots.txt: Not always
Fetches individual links at a user's request, including helping Meta's AI navigate websites to complete tasks for users.
Block Meta-ExternalFetcher
User-agent: Meta-ExternalFetcher
Disallow: /A Disallow line may not stop Meta-ExternalFetcher, because Meta says it may bypass robots.txt for user-requested fetches.
Allow Meta-ExternalFetcher
User-agent: Meta-ExternalFetcher
Allow: /An allow group only changes anything if a broader rule would block Meta-ExternalFetcher. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Facts
- Operator
- Meta
- Purpose
- User-requested fetch
- Obeys robots.txt
- Not always. Meta says this crawler may bypass robots.txt because it performs fetches that were requested by the user.
- In your server logs
- The user-agent string contains "meta-externalfetcher/1.1".
- How to verify it
- Unconfirmed by the operator Meta's documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be Meta-ExternalFetcher cannot be verified against the operator.
What Meta says
Quoted word for word from the operator's documentation.
“The Meta-ExternalFetcher crawler fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities—including helping AI navigate websites to complete tasks for users. Accordingly, this crawler may bypass robots.txt rules.”
“The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user.”
“`meta-externalfetcher/1.1`”
Sources
- Meta for Developers: Meta Web Crawlers — fetched and checked 2026-09-22