robots.txt and llms.txt generator for AI agents
Choose what AI agents may do. Your robots.txt uses the exact tokens of the 19 agents in the directory, keeps your existing rules, and is checked line by line before you copy it. It all runs in this tab: nothing you type or paste leaves the page.
robots.txt
Groups for other crawlers are kept exactly as written. A group that names an agent from the directory is replaced, and the removed lines are listed under the output.
One per line, starting with /. Applies to the agents you allow; blocked agents are already kept out of everything.
An agent named in its own group ignores the * group, so the generator repeats your * rules for it.
Lines taken out of your file
Verification: the file above, read back by an RFC 9309 matcher
Each directory agent is decided against the generated file by the checker’s own matcher, on / and on every path your choices block. ✓ means the file does what you chose. A red ✗ row would be a bug in this generator.
| Agent | You chose | Result by path |
|---|
Each agent: what blocking it does
In the operator’s own terms, from the directory. Change one agent without touching the rest; the preset switches to Custom.
llms.txt
llms.txt is a community proposal, not a standard. It is described at llmstxt.org: a Markdown file at /llms.txt that gives a language model a short summary of a site and links to the pages worth reading. No standards body has adopted it. Whether the agents in the directory read it is unconfirmed: none of the operator crawler pages the directory cites says its agent fetches or uses llms.txt (checked 23 September 2026). Some of those operators’ documentation sites, Perplexity’s for one, publish an llms.txt of their own for their docs. That is not the same as their crawlers reading yours.
Two mistakes this generator is built to avoid
1. Adding an AI group can reopen paths you had closed
User-agent: *
Disallow: /admin/
User-agent: OAI-SearchBot
Allow: /OAI-SearchBot may now fetch /admin/. Under RFC 9309, section 2.2.1, a crawler follows only the group that names it; the * group stops applying to it. So every group this generator writes for an agent you allow repeats your * Allow and Disallow lines, unless you choose “May fetch everything”. The verification table then checks /admin/ for every agent.
2. Tokens that are not documented
Every User-agent line here is copied from the directory, where each token is spelled as its operator documents it. Snippets passed around the web still use names such as anthropic-ai and Claude-Web. Anthropic’s crawler article, checked on 23 September 2026, names ClaudeBot, Claude-SearchBot and Claude-User and does not mention either of those. If your pasted file has groups for tokens that are not in the directory, they are kept as they are and listed under the output, so you can decide.
Where the file goes, and when crawlers see it
- At
/robots.txton each host. RFC 9309, section 2.3 says the file must be named/robots.txt, all lower case, at the top level of the host, UTF-8 encoded and served astext/plain.www.and every other subdomain need their own copy. - Size. Crawlers must read at least the first 500 KiB (section 2.5) and may ignore the rest. The generated section adds about 2 KB.
- Caching. Crawlers may cache the file, and should not use a cached copy for more than 24 hours unless the file cannot be reached (section 2.4). Operators describe their own delays: OpenAI says a change can take about 24 hours to reach its search systems, Perplexity says up to 24 hours, and Amazon says Amazonbot may use a copy cached for up to 30 days. Each is quoted on its crawler page.
- Test before you rely on it. Once it is live, paste the served file into the checker and try the paths you care about.
robots.txt is a request, not enforcement
A crawler that honours robots.txt stops; nothing forces the rest. Four user-triggered fetchers in the directory come with their operator’s own warning: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it, and Amazon says Amzn-User may not follow all of it. The generator marks these agents rather than promise a block. If you need to enforce, block at your server. Each crawler page’s “How to verify it” entry gives the operator’s reverse-DNS check or published IP list, so you can tell the real crawler from something borrowing its name. Anthropic says blocking ClaudeBot by IP address may not work reliably, because it stops the crawler reading your robots.txt. Every path you list in robots.txt is public, so it is not a place to hide anything.
What an llms.txt contains
The proposal fixes a small Markdown shape, and the builder above writes exactly that. First comes an H1 with the site name, the only required part. Then a blockquote with a one-line summary, and any plain paragraphs you need. After that come H2 sections, each a list of links written - [title](url): note. Titles are escaped, so a ] or * in a page name cannot break the link. Only full http:// or https:// URLs are accepted. The file goes at /llms.txt at the root of the site. It changes nothing about who may crawl you: that is robots.txt’s job.