bedrockbot
Amazon · Unconfirmed · robots.txt: Yes
Crawls the URLs an AWS customer selects for an Amazon Bedrock knowledge base, and on each run fetches every page reachable from them within the customer's scope and filters.
Block bedrockbot
User-agent: bedrockbot
Disallow: /Unconfirmed by the operator Amazon's documentation we checked does not say what blocking bedrockbot changes beyond the crawler itself.
Allow bedrockbot
User-agent: bedrockbot
Allow: /An allow group only changes anything if a broader rule would block bedrockbot. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks bedrockbot on a given path, and which line decides.
Who blocks it
bedrockbot is not in the 2026-10-09 survey: it was added to this directory after we last read the most-visited sites' robots.txt files, so we have no count for it yet.
Facts
- Operator
- Amazon
- Purpose
- Unconfirmed by the operator
Amazon documents it as the Web Crawler of Amazon Bedrock knowledge bases: an AWS customer selects URLs and the crawler fetches them into that customer's knowledge base. That is none of training, a public AI search index or a user fetch as Amazon states it.
- Obeys robots.txt
- Yes. Amazon says the Bedrock Web Crawler respects robots.txt in accordance with RFC 9309. It looks first for rules naming bedrockbot-UUID and then for generic bedrockbot rules.
- In your server logs
- Unconfirmed by the operator Amazon's documentation we checked does not give its user-agent string.
- How to verify it
- Unconfirmed by the operator Amazon's documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be bedrockbot cannot be verified against the operator.
What Amazon says
Quoted word for word from the operator's documentation.
“The crawler will first look for bedrockbot-UUID rules and then for generic bedrockbot rules in the robots.txt file.”
“The Web Crawler respects robots.txt in accordance with the RFC 9309.”
“The Amazon Bedrock provided Web Crawler connects to and crawls URLs you have selected for use in your Amazon Bedrock knowledge base.”
“Each time the the Web Crawler runs, it retrieves content for all URLs that are reachable from the source URLs and which match the scope and filters.”
Sources
- Amazon Bedrock User Guide: Crawl web pages for your knowledge base — fetched and checked 2026-10-09