Meta-ExternalAgent
Meta · Training · robots.txt: Yes
Crawls the web for uses such as training Meta's foundation AI models or improving Meta products by indexing content directly.
Block Meta-ExternalAgent
User-agent: Meta-ExternalAgent
Disallow: /Unconfirmed by the operator Meta's documentation we checked does not say what blocking Meta-ExternalAgent changes beyond the crawler itself.
Allow Meta-ExternalAgent
User-agent: Meta-ExternalAgent
Allow: /An allow group only changes anything if a broader rule would block Meta-ExternalAgent. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Facts
- Operator
- Meta
- Purpose
- Model training
Meta's wording is a mix: "training foundation AI models or improving products by indexing content directly".
- Obeys robots.txt
- Yes. Meta says to block its crawlers with a Disallow for the relevant crawler in robots.txt, and that robots.txt may be cached for up to 24 hours. Meta's own example writes the token in lower case (meta-externalagent).
- In your server logs
- The user-agent string contains "meta-externalagent/1.1".
- How to verify it
- Unconfirmed by the operator Meta's documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be Meta-ExternalAgent cannot be verified against the operator.
What Meta says
Quoted word for word from the operator's documentation.
“The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”
“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.”
“Please allow up to 24 hours for changes to `robots.txt` to take effect because crawlers may cache the contents of `robots.txt` for up to 24 hours.”
“User-agent: meta-externalagent”
“`meta-externalagent/1.1`”
Sources
- Meta for Developers: Meta Web Crawlers — fetched and checked 2026-09-22