robots.txt checker: which AI agents may read which pages
Paste your robots.txt and choose a path. For each of the 19 AI crawler tokens in the directory you get allowed or blocked, and the exact line that decided it. It runs in this tab: nothing you paste leaves the page.
A path such as /blog/post?id=3, or a full URL. The host part is ignored.
| Agent | Operator | Purpose | Result | Decided by |
|---|
What the checker read in your file
How a crawler decides, and where answers go wrong
The checker applies RFC 9309, the 2022 standard for robots.txt. These are the rules that most often surprise people, each with the result the checker gives.
1. A group that names the agent replaces the * group entirely
User-agent: *
Disallow: /private/
User-agent: GPTBot
Disallow: /drafts/GPTBot may fetch /private/report. It follows only the group that names it, and that group says nothing about /private/. The * rules are not added on top. If you want GPTBot kept out of both, repeat Disallow: /private/ in its group.
2. The longest matching rule wins, not the first one
User-agent: *
Disallow: /
Allow: /blog//blog/post is allowed and /about is blocked, whatever order the two lines are in. Allow: /blog/ matches 6 octets and Disallow: / matches 1. The 1996 draft that came before the standard let the first matching line win, and some older tools still work that way.
3. When an allow rule and a disallow rule are the same length, allow wins
User-agent: *
Allow: /docs/
Disallow: /*.pdf$/docs/guide.pdf is blocked: /*.pdf$ is 7 characters and /docs/ is 6. Remove the $ and the two tie at 6, so allow wins and the PDF can be fetched. The RFC says the match "that has the most octets" wins but does not say whether * and $ count. This checker counts the pattern as written, wildcards included.
4. A blank line does not end a group
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /Both agents are blocked. Under RFC 9309, a group ends only when a User-agent line comes after a rule. Blank lines, comments and Sitemap lines in between change nothing. A parser that treats blank lines as separators would give GPTBot an empty group and let it in.
5. * matches anything, including /; $ pins the end
User-agent: *
Disallow: /*.pdf$
Disallow: /shop/*/checkout/files/a.pdf is blocked but /files/a.pdf?v=2 is not, because the query string comes after .pdf. /shop/shoes/checkout and /shop/a/b/checkout are blocked. /shop/checkout is not, because the pattern needs a second /.
6. Agent names ignore case; paths do not
user-agent: gptbot applies to GPTBot. Disallow: /Private does not block /private. All groups that name the same agent are merged into one, even if they are far apart in the file.
7. Percent-encoding is normalised before comparing
/%7Efoo and /~foo are the same path, because ~ is an unreserved character. %7e and %7E are the same too. But /a%2Fb is not /a/b, because an encoded reserved character stays encoded. Non-ASCII characters are compared as UTF-8 bytes, so /café and /caf%C3%A9 match each other.
8. Comma-separated agents are not in the standard
User-agent: GPTBot, ClaudeBot is not valid RFC 9309. A reported erratum (8895, April 2026) proposes allowing it, but it has not been adopted. Until then, parsers differ. This checker reads only the leading name, GPTBot, and flags the line. A stricter parser may match neither agent. Give each agent its own User-agent line.
What this check cannot tell you
- Whether your server returns the file. If
/robots.txtanswers with a 4xx status, crawlers may fetch everything. If it answers with a 5xx or cannot be reached, RFC 9309 says crawlers must assume everything is disallowed (sections 2.3.1.3 and 2.3.1.4). - Other hosts. Each host and subdomain has its own file.
www.example.com/robots.txtsays nothing aboutshop.example.com. - When a change takes effect. Crawlers may cache your file. The RFC says they should not use a cached copy for more than 24 hours.
- Whether the agent listens. Rows marked "Not always obeys robots.txt" are user-triggered fetchers whose operators say they may not follow it. Blocking them here is a request, not a guarantee. robots.txt is not access control, and every path you list in it is public.
- Control tokens.
Google-ExtendedandApplebot-Extendednever fetch pages. A "blocked" result for them means Google or Apple should not use pages its other crawlers fetched for the purposes those tokens control. Their pages in the directory say exactly which purposes. - Operator-specific fallbacks. Amazon says that if your file does not name
Amzn-SearchBotbut allows other search bots, it follows the rules you gave those bots. The checker applies RFC 9309 only, so it shows the*group for Amzn-SearchBot. - Page-level rules.
noindex,nosnippetandX-Robots-Taglive in the page or its headers, not in robots.txt, and are not checked here.
The matcher is tested against every example in RFC 9309 sections 5.1, 5.2 and 2.2.2, copied from the RFC text. Two published errata affect those examples: 7128, a filename typo in 5.2, and 7124, a code point in 2.2.2. The tests apply the corrected readings.