Meta-WebIndexer
Meta · AI search · robots.txt: Yes
Crawls the web to improve Meta AI's search results so Meta AI can cite and link to pages in its answers.
Block Meta-WebIndexer
User-agent: Meta-WebIndexer
Disallow: /Meta says allowing Meta-WebIndexer helps it cite and link to your content in Meta AI's responses, so blocking it gives that up.
Allow Meta-WebIndexer
User-agent: Meta-WebIndexer
Allow: /An allow group only changes anything if a broader rule would block Meta-WebIndexer. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Facts
- Operator
- Meta
- Purpose
- AI search index
- Obeys robots.txt
- Yes. Meta says to block its crawlers with a Disallow for the relevant crawler in robots.txt; the documented exceptions are Meta-ExternalFetcher and FacebookExternalHit, not this crawler.
- In your server logs
- The user-agent string contains "meta-webindexer/1.1".
- How to verify it
- Unconfirmed by the operator Meta's documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be Meta-WebIndexer cannot be verified against the operator.
What Meta says
Quoted word for word from the operator's documentation.
“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.”
“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt. The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user. Also, the FacebookExternalHit crawler might bypass robots.txt when performing security or integrity checks.”
“`meta-webindexer/1.1`”
Sources
- Meta for Developers: Meta Web Crawlers — fetched and checked 2026-09-22