robots.txt and llms.txt generator for AI agents
Choose what AI agents may do. Your robots.txt uses the exact tokens of the 49 agents in the directory, keeps your existing rules, and is checked line by line before you copy it. It all runs in this tab: nothing you type or paste leaves the page.
robots.txt
Groups for other crawlers are kept exactly as written. A group that names an agent from the directory is replaced, and the removed lines are listed under the output.
One per line, starting with /. Applies to the agents you allow; blocked agents are already kept out of everything.
An agent named in its own group ignores the * group, so the generator repeats your * rules for it.
Lines taken out of your file
Verification: the file above, read back by an RFC 9309 matcher
Each directory agent is decided against the generated file by the checker’s own matcher, on / and on every path your choices block. ✓ means the file does what you chose. A red ✗ row would be a bug in this generator.
| Agent | You chose | Result by path |
|---|
Each agent: what blocking it does
In the operator’s own terms, from the directory. Change one agent without touching the rest; the preset switches to Custom.
Enforce it at the server
robots.txt asks; a server rule refuses. For each agent you block, a rule that refuses its User-Agent (the server answers 403). For each agent you allow whose operator publishes an IP list, a rule that refuses requests claiming to be it from an address outside that list, so only the real one gets in. The same rules are on each crawler page. Built in this tab from data shipped with the page.
What a User-Agent rule cannot do: any client can send any User-Agent. Apache’s own documentation: Access control by User-Agent is an unreliable technique, since the User-Agent header can be set to anything at all, at the whim of the end user.
The IP rules compare the address your server sees; behind a CDN or load balancer, restore the client address first or put the rule at the CDN.
Syntax from each server’s own documentation, with every line form and its quote in server-rules-sources.json: nginx (rewrite, map, geo); Apache (Access Control, expressions, Require ip); Caddy (matchers, respond); Cloudflare (custom rule example, operators, lists, expression limit).
llms.txt
llms.txt is a community proposal, not a standard. It is described at llmstxt.org: a Markdown file at /llms.txt that gives a language model a short summary of a site and links to the pages worth reading. No standards body has adopted it. Whether the agents in the directory read it is unconfirmed: none of the operator crawler pages the directory cites says its agent fetches or uses llms.txt (checked 23 September 2026). Some of those operators’ documentation sites, Perplexity’s for one, publish an llms.txt of their own for their docs. That is not the same as their crawlers reading yours.
Already have an llms.txt, hand-written or made by a CMS plugin? Check a file you already have against the format, line by line.
Three mistakes this generator is built to avoid
1. Adding an AI group can reopen paths you had closed
User-agent: *
Disallow: /admin/
User-agent: OAI-SearchBot
Allow: /
OAI-SearchBot may now fetch /admin/. Under RFC 9309, section 2.2.1, a crawler follows only the group that names it; the * group stops applying to it. So every group this generator writes for an agent you allow repeats your * Allow and Disallow lines, unless you choose “May fetch everything”. The verification table then checks /admin/ for every agent.
2. Tokens the operator no longer documents
Snippets passed around the web still use names such as anthropic-ai and Claude-Web. Anthropic’s crawler article, checked on 24 September 2026, names ClaudeBot, Claude-SearchBot and Claude-User and mentions neither of the old two. The directory keeps such tokens, badged with the date we checked, so the generated file still names them (an extra line costs nothing) with a comment that the line may do nothing — the operator’s current tokens, which are in the same file, do the work. Groups in your pasted file for tokens the directory does not list are kept as they are and listed under the output, so you can decide.
3. Blocking a search engine when you meant its AI
Googlebot, Bingbot and PetalBot also decide Google Search, Bing and Petal Search. Google says AI is built into Search and has no separate robots.txt token for AI Overviews or AI Mode; Microsoft says Bing search and Copilot share one crawl. A Disallow for one of them is a request to leave that search engine, not just its AI answers. So no purpose choice here names them, “Block all AI agents” included: they follow your * group, or a group you already have for them, which is kept as it is. Choose Block or Allow for one by name if that is what you mean. Each crawler’s page quotes the page-level controls (such as nosnippet or NOARCHIVE) its operator offers instead.
Where the file goes, and when crawlers see it
- At
/robots.txton each host. RFC 9309, section 2.3 says the file must be named/robots.txt, all lower case, at the top level of the host, UTF-8 encoded and served astext/plain.www.and every other subdomain need their own copy. - Size. Crawlers must read at least the first 500 KiB (section 2.5) and may ignore the rest. The generated section adds about 2 KB.
- Caching. Crawlers may cache the file, and should not use a cached copy for more than 24 hours unless the file cannot be reached (section 2.4). Operators describe their own delays: OpenAI says a change can take about 24 hours to reach its search systems, Perplexity says up to 24 hours, and Amazon says Amazonbot may use a copy cached for up to 30 days. Each is quoted on its crawler page.
- Test before you rely on it. Once it is live, paste the served file into the checker and try the paths you care about.
robots.txt is a request, not enforcement
A crawler that honours robots.txt stops; nothing forces the rest. Four user-triggered fetchers in the directory come with their operator’s own warning: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it, and Amazon says Amzn-User may not follow all of it. The generator marks these agents rather than promise a block. If you need to enforce, block at your server: “Enforce it at the server” above writes the rules for nginx, Apache, Caddy or Cloudflare. Each crawler page’s “How to verify it” entry gives the operator’s reverse-DNS check or published IP list, so you can tell the real crawler from something borrowing its name. Anthropic says blocking ClaudeBot by IP address may not work reliably, because it stops the crawler reading your robots.txt. Every path you list in robots.txt is public, so it is not a place to hide anything.
What an llms.txt contains
The proposal fixes a small Markdown shape, and the builder above writes exactly that. First comes an H1 with the site name, the only required part. Then a blockquote with a one-line summary, and any plain paragraphs you need. After that come H2 sections, each a list of links written - [title](url): note. Titles are escaped, so a ] or * in a page name cannot break the link. Only full http:// or https:// URLs are accepted. The file goes at /llms.txt at the root of the site. It changes nothing about who may crawl you: that is robots.txt’s job.