Bytespider
Operator unconfirmed · Unconfirmed · robots.txt: Unconfirmed
Unconfirmed by the operator. We looked for documentation of the Bytespider token from its operator on 22 Sep 2026 and found none, so this page states nothing about what it does. The robots.txt lines below are still correct syntax for the token.
The token Bytespider is listed because site owners ask about it, not because an operator documents it. Every fact below is unconfirmed by the operator.
Block Bytespider
User-agent: Bytespider
Disallow: /Unconfirmed by the operator No operator documentation says whether Bytespider reads robots.txt at all, so these lines are a request it may or may not honour.
Allow Bytespider
User-agent: Bytespider
Allow: /An allow group only changes anything if a broader rule would block Bytespider. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Facts
- Operator
- Unconfirmed by the operator We found no page from any organisation that documents this token.
- Purpose
- Unconfirmed by the operator
- Obeys robots.txt
- Unconfirmed by the operator
- In your server logs
- Unconfirmed by the operator The documentation we checked does not give its user-agent string.
- How to verify it
- Unconfirmed by the operator The documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be Bytespider cannot be verified against the operator.
Sources
Unconfirmed by the operator We found no documentation from an operator for this token.
Where we looked
- www.bytedance.com/en/ — 2026-09-22: Page loaded; no mention of Bytespider.
- www.bytedance.com/robots.txt — 2026-09-22: Loaded; no mention of Bytespider.
- www.bytedance.com/en/bytespider — 2026-09-22: Page not found (the site's 404 page).