resulthack

robots.txt and llms.txt generator for AI agents

Choose what AI agents may do. Your robots.txt uses the exact tokens of the 49 agents in the directory, keeps your existing rules, and is checked line by line before you copy it. It all runs in this tab: nothing you type or paste leaves the page.

Three mistakes this generator is built to avoid

1. Adding an AI group can reopen paths you had closed

User-agent: *
Disallow: /admin/

User-agent: OAI-SearchBot
Allow: /

OAI-SearchBot may now fetch /admin/. Under RFC 9309, section 2.2.1, a crawler follows only the group that names it; the * group stops applying to it. So every group this generator writes for an agent you allow repeats your * Allow and Disallow lines, unless you choose “May fetch everything”. The verification table then checks /admin/ for every agent.

2. Tokens the operator no longer documents

Snippets passed around the web still use names such as anthropic-ai and Claude-Web. Anthropic’s crawler article, checked on 24 September 2026, names ClaudeBot, Claude-SearchBot and Claude-User and mentions neither of the old two. The directory keeps such tokens, badged with the date we checked, so the generated file still names them (an extra line costs nothing) with a comment that the line may do nothing — the operator’s current tokens, which are in the same file, do the work. Groups in your pasted file for tokens the directory does not list are kept as they are and listed under the output, so you can decide.

3. Blocking a search engine when you meant its AI

Googlebot, Bingbot and PetalBot also decide Google Search, Bing and Petal Search. Google says AI is built into Search and has no separate robots.txt token for AI Overviews or AI Mode; Microsoft says Bing search and Copilot share one crawl. A Disallow for one of them is a request to leave that search engine, not just its AI answers. So no purpose choice here names them, “Block all AI agents” included: they follow your * group, or a group you already have for them, which is kept as it is. Choose Block or Allow for one by name if that is what you mean. Each crawler’s page quotes the page-level controls (such as nosnippet or NOARCHIVE) its operator offers instead.

Where the file goes, and when crawlers see it

robots.txt is a request, not enforcement

A crawler that honours robots.txt stops; nothing forces the rest. Four user-triggered fetchers in the directory come with their operator’s own warning: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it, and Amazon says Amzn-User may not follow all of it. The generator marks these agents rather than promise a block. If you need to enforce, block at your server: “Enforce it at the server” above writes the rules for nginx, Apache, Caddy or Cloudflare. Each crawler page’s “How to verify it” entry gives the operator’s reverse-DNS check or published IP list, so you can tell the real crawler from something borrowing its name. Anthropic says blocking ClaudeBot by IP address may not work reliably, because it stops the crawler reading your robots.txt. Every path you list in robots.txt is public, so it is not a place to hide anything.

What an llms.txt contains

The proposal fixes a small Markdown shape, and the builder above writes exactly that. First comes an H1 with the site name, the only required part. Then a blockquote with a one-line summary, and any plain paragraphs you need. After that come H2 sections, each a list of links written - [title](url): note. Titles are escaped, so a ] or * in a page name cannot break the link. Only full http:// or https:// URLs are accepted. The file goes at /llms.txt at the root of the site. It changes nothing about who may crawl you: that is robots.txt’s job.

Made with AI tools. · Privacy