amazon-kendra
Amazon · Unconfirmed · robots.txt: Yes
Crawls the web pages an Amazon Kendra customer points it at, to index them in that customer's own Kendra search index.
Block amazon-kendra
User-agent: amazon-kendra
Disallow: /Amazon says a Disallow for amazon-kendra stops the Amazon Kendra Web Crawler from indexing your website.
Allow amazon-kendra
User-agent: amazon-kendra
Allow: /An allow group only changes anything if a broader rule would block amazon-kendra. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks amazon-kendra on a given path, and which line decides.
Who blocks it
amazon-kendra is not in the 2026-10-09 survey: it was added to this directory after we last read the most-visited sites' robots.txt files, so we have no count for it yet.
Facts
- Operator
- Amazon
- Purpose
- Unconfirmed by the operator
Amazon documents it as the web crawler of Amazon Kendra, a search service AWS customers use to index and search documents of their choice: a customer points it at URLs for their own search index. That is none of training, a public AI search index or a user fetch as Amazon states it. Amazon also says Kendra is no longer open to new customers.
- Obeys robots.txt
- Yes. Amazon says the Kendra Web Crawler respects the standard robots.txt Allow and Disallow directives, and gives User-agent: amazon-kendra with Disallow: / as the way to stop it crawling a site.
- In your server logs
- Unconfirmed by the operator Amazon's documentation we checked does not give its user-agent string.
- How to verify it
- Unconfirmed by the operator Amazon's documentation we checked gives no IP list or DNS check for this crawler, so a request claiming to be amazon-kendra cannot be verified against the operator.
What Amazon says
Quoted word for word from the operator's documentation.
“To stop Amazon Kendra Web Crawler from crawling the website, use the following directive: User-agent: amazon-kendra # Amazon Kendra Web Crawler Disallow: / # disallow access to any pages”
“Amazon Kendra is an intelligent search service that AWS customers use to index and search documents of their choice.”
“In order to index documents on the web, customers may use Amazon Kendra Web Crawler, indicating which URL(s) should be indexed and other operational parameters.”
“Amazon Kendra Web Crawler respects standard robots.txt directives like Allow and Disallow.”
“You can stop Amazon Kendra Web Crawler from indexing your website using the Disallow directive.”
“Amazon Kendra is no longer open to new customers.”
Sources
- Amazon Kendra Developer Guide: Configuring the robots.txt file for Amazon Kendra Web Crawler — fetched and checked 2026-10-09