resulthack

Which AI-training opt-out signals does your site send?

There are at least five ways to tell AI companies not to train on your site, each defined by someone different. Paste your files and see which signals you send, whether each is written the way its definition says, and which AI operators say in their own documentation that they read it. Everything is read in this tab and nothing is fetched or sent.

Crawlers whose data is used for AI training by others, or whose operator documents no purpose

The checker above counts a token as an AI-training opt-out only when its operator writes that blocking it keeps your content out of training. The six below do not meet that bar, but many sites block them for the same reason, and nytimes.com's robots.txt blocks all six. CCBot fills Common Crawl's free web archive, which its operator describes as being for research and analysis and which third parties document as a major source of AI training data. Four have no purpose their operator documents: anthropic-ai, FacebookBot and cohere-ai are not listed on the crawler pages of Anthropic, Meta and Cohere, the companies they are attributed to, and ByteDance publishes no page about Bytespider. ImagesiftBot's operator, Hive, documents an image-search index, not training; it is here because Cloudflare counts it among AI bots and block lists carry it.

Each card quotes the operator first, then named third parties, labelled as such. A third party's claim is theirs, not the operator's and not ours. Whether your robots.txt blocks each one is decided with the same RFC 9309 matching as the table above, for the page path you give. The generator adds a crawler only when you tick it. The same data is in opt-out-others.json.

CCBot Common Crawl

Paste a robots.txt into the checker above to see whether it blocks CCBot.

Common Crawl documents its archive as a free copy of the web for research and analysis, not as AI training. Its own Terms of Use name training AI systems as one use others make of the archive, and third parties document that many large language models are trained on it.

What the operator says

“Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the Internet to Internet researchers, companies and individuals at no cost for the purpose of research and analysis.”

FAQ, answer to “What is Common Crawl?” commoncrawl.org, checked 26 Sep 2026.

“People have used the data to improve language translation software, predict trends, track disease propagation, and much more.”

FAQ, answer to “What can you do with Common Crawl data?” commoncrawl.org, checked 26 Sep 2026.

“use of Crawled Content in connection with artificial intelligence, machine learning, or other similar technologies, including, without limitation, large language models and neural networks (collectively, "AI Systems"), including the use of Crawled Content for the purpose of developing, training, or deploying AI Systems”

Terms of Use (last updated 7 March 2024), indemnification clause: whoever uses the archive covers Common Crawl's costs from claims arising from, among other things, this use. It is the only place on the pages checked where Common Crawl mentions AI training. commoncrawl.org, checked 26 Sep 2026.

“Add these lines to your robots.txt file and our crawler will stop crawling your website: User-agent: CCBot Disallow: /”

FAQ, answer to “How can I block the Common Crawl CCBot?”. The pages checked do not say whether blocking removes your pages from crawls already published; the same answer points to a separate opt-out registry. commoncrawl.org, checked 26 Sep 2026.

What others say third party, not the operator

Third party: Mozilla Foundation

“This data has been valuable for many researchers, but since 2020 when OpenAI published GPT-3, the large language model (LLM) that still powers the free version of ChatGPT, Common Crawl has become one of the most important sources of training data for generative AI.”

Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)

“Provides open crawl dataset, used for many purposes, including Machine Learning/AI.”

Third party: Cloudflare

“In the graph below, which looks at disallowed crawlers for these sites, we see that customers most often reference GPTBot, CCBot, and Google in robots.txt, but do not specifically disallow popular AI crawlers like Bytespider and ClaudeBot.”

Bytespider ByteDance (attributed by third parties; ByteDance documents no crawler of this name)

Paste a robots.txt into the checker above to see whether it blocks Bytespider.

No page of ByteDance's that we could find names Bytespider. Third parties attribute it to ByteDance and report it collects training data for ByteDance's language models.

What the operator says

The operator documents no purpose for Bytespider (checked 26 Sep 2026).

What others say third party, not the operator

Third party: Cloudflare

“Operated by ByteDance, the Chinese company that owns TikTok, Bytespider is reportedly used to gather training data for its large language models (LLMs), including those that support its ChatGPT rival, Doubao.”

Third party: Cloudflare

“Among the top AI bots that we see, Bytespider not only leads in terms of number of requests but also in both the extent of its Internet property crawling and the frequency with which it is blocked.”

Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)

“Downloads data to train LLMS, including ChatGPT competitors.”

cohere-ai Cohere (attributed by third parties; Cohere's crawler page does not name this token)

Paste a robots.txt into the checker above to see whether it blocks cohere-ai.

Cohere's crawler page does not name cohere-ai and says Cohere does not crawl the web to train its models. A third-party list describes cohere-ai as fetching pages for user prompts.

What the operator says

The operator documents no purpose for cohere-ai (checked 26 Sep 2026).

“We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to train generative AI foundation models at this time.”

Cohere Web Crawlers, first paragraph. About Cohere's bots in general; it does not name cohere-ai. docs.cohere.com, checked 26 Sep 2026.

“Should Cohere make use of these tools to crawl or scrape web content to train AI models in the future, Cohere will identify those bots and/or user agents in the table below alongside information about how to disallow their access to your site.”

Cohere Web Crawlers, first paragraph. The table below it reads “N/A”. docs.cohere.com, checked 26 Sep 2026.

What others say third party, not the operator

Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)

“Retrieves data to provide responses to user-initiated prompts.”

anthropic-ai Anthropic (attributed by third parties; Anthropic's crawler page does not name this token)

Paste a robots.txt into the checker above to see whether it blocks anthropic-ai.

Anthropic's crawler page lists three bots and not anthropic-ai; the one it documents for training is ClaudeBot, which is in the documented group above. A third-party list still describes anthropic-ai as a training scraper.

What the operator says

The operator documents no purpose for anthropic-ai (checked 26 Sep 2026).

“Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.”

The three are ClaudeBot, Claude-User and Claude-SearchBot; anthropic-ai is not among them. support.claude.com, checked 26 Sep 2026.

What others say third party, not the operator

Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)

“Scrapes data to train LLMs and AI products offered by Anthropic.”

FacebookBot Meta (attributed by third parties; Meta's crawler page does not name this token)

Paste a robots.txt into the checker above to see whether it blocks FacebookBot.

Meta's crawler page lists five crawlers and not FacebookBot; the one it documents for AI training is Meta-ExternalAgent, which is in the documented group above. A third-party list says Meta once described FacebookBot as training speech recognition.

What the operator says

The operator documents no purpose for FacebookBot (checked 24 Sep 2026).

What others say third party, not the operator

Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)

“Officially used for training Meta "speech recognition technology," unknown if used to train Meta AI specifically.”

ImagesiftBot Hive (ImageSift)

Paste a robots.txt into the checker above to see whether it blocks ImagesiftBot.

Hive documents ImagesiftBot as collecting images, page text and alt text for an image index its “web intelligence products” search. It does not call this AI training, and no third party we checked says it is.

What the operator says

“ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products”

ImageSift Bot page, first sentence. imagesift.com, checked 26 Sep 2026.

“Once images and text are downloaded from a webpage, ImageSift analyzes this data from the page and stores the information in an index. Our web intelligence products use this index to enable search and retrieval of similar images.”

FAQ, “How do we use this information?” imagesift.com, checked 26 Sep 2026.

How it reads robots.txt: The verdict here uses RFC 9309 (its own group, else User-agent: *). If your robots.txt has no ImagesiftBot group but has a Googlebot group, Hive says ImagesiftBot follows the Googlebot group instead. In its words: “If there is no rule targeting ImagesiftBot, but there is a rule targeting Googlebot, then ImagesiftBot will follow the Googlebot directives.” (imagesift.com, checked 26 Sep 2026)

What others say third party, not the operator

Who says they honour which signal

Each cell comes from the operator's own documentation, quoted, with the page and the date we read it. "Not documented" means that page does not mention the signal: it is the absence of a statement, not proof the signal is ignored, and an operator may read a signal it does not write about. It also cannot tell you whether a crawler actually complies. The same table is in opt-out-signals.json.

OperatorPer-agent robots.txt tokensContent SignalsTDMRepnoai and noimageaiai.txt
OpenAI“Not documented”: not mentioned on OpenAI’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented GPTBotDisallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.OpenAI documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiNot documentedai.txtNot documented
Anthropic“Not documented”: not mentioned on Anthropic’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented ClaudeBotWhen a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.Anthropic documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiNot documentedai.txtNot documented
Google“Not documented”: not mentioned on Google’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented Google-ExtendedGoogle-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.Google-Extended also controls grounding in Gemini Apps, not only training. Google says it does not affect inclusion in Google Search.Google documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiNot documentedai.txtNot documented
Apple“Not documented”: not mentioned on Apple’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented Applebot-ExtendedWith Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s general purpose foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.Apple documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiNot documentedai.txtNot documented
Meta“Not documented”: not mentioned on Meta’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented Meta-ExternalAgentIn order to block these crawlers, add a disallow for the relevant crawler to robots.txt.Meta documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiPrefers robots.txtWe make it easy for site managers and content owners to indicate their preferences by using industry-standard practices like robots.txt rather than non-standard formats like NoAI tags.Meta documentation, read 25 Sep 2026ai.txtNot documented
Amazon“Not documented”: not mentioned on Amazon’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented AmazonbotAutomated crawling from these listed user agents respects the Robots Exclusion Protocol , honoring the user-agent and the allow/disallow directives.Amazon documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiNot documentedai.txtNot documented
Mistral AI“Not documented”: not mentioned on Mistral AI’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented MistralAI-TrainingMistralAI-Training crawls web content to help build datasets for training Mistral generative AI models. Webmasters can disallow this user agent in their robots.txt file.Mistral AI documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiNot documentedai.txtNot documented
Common Crawl“Not documented”: not mentioned on Common Crawl’s crawler page, read 25 Sep 2026Per-agent robots.txt tokensDocumented CCBotAdd these lines to your robots.txt file and our crawler will stop crawling your website:This stops CCBot crawling at all. Common Crawl’s own pages do not describe CCBot as an AI-training crawler; its dataset is widely used for training by others.Common Crawl documentation, read 25 Sep 2026Content SignalsNot documentedTDMRepNot documentednoai and noimageaiNot documentedai.txtNot documented

Only one kind of signal has written support from all eight: a robots.txt line naming the operator's own token. Meta is the only one of the eight that addresses noai at all, and it points to robots.txt instead. None of the eight crawler pages mentions Content-Signal, TDMRep or ai.txt.

The five signals, and who defined each

Per-agent robots.txt tokens

Where: robots.txt at the site root.

Defined by: robots.txt is RFC 9309 (IETF, 2022). Each training token is defined by the operator that reads it: GPTBot by OpenAI, Google-Extended by Google, and so on.

Specification: RFC 9309: Robots Exclusion Protocol (read 25 Sep 2026).

Content Signals (Content-Signal line in robots.txt)

Where: robots.txt, inside a User-agent group.

Defined by: Cloudflare, the Content Signals Policy, announced 24 Sep 2025. It defines search, ai-input and ai-train (yes or no); Cloudflare's robots.txt documentation (last updated 3 Aug 2026) adds a fourth field, use, with the values immediate, reference or full.

Specification: Content Signals (contentsignals.org, the policy text and examples) (read 25 Sep 2026); Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy (24 Sep 2025) (read 25 Sep 2026); Cloudflare docs: robots.txt setting (use= field; last updated 3 Aug 2026) (read 25 Sep 2026).

TDMRep (TDM Reservation Protocol)

Where: /.well-known/tdmrep.json, the tdm-reservation and tdm-policy HTTP headers, and meta tags of the same names.

Defined by: The W3C Text and Data Mining Reservation Protocol Community Group; Final Community Group Report of 10 May 2024, editor Laurent Le Meur (EDRLab). The report itself says it is not a W3C Standard nor on the W3C Standards Track.

Specification: TDM Reservation Protocol (TDMRep), Final Community Group Report, 10 May 2024 (read 25 Sep 2026).

noai and noimageai

Where: a robots meta tag, or an X-Robots-Tag response header.

Defined by: DeviantArt, November 2022, for its own artists' pages; other sites copied it. It is not in Google's or Bing's robots meta documentation and has no published specification.

Specification: DeviantArt: UPDATE All Deviations Are Opted Out of AI Datasets (Nov 2022) (read 25 Sep 2026; the page answered our fetch with HTTP 403; the origin and date are as the page is indexed by search engines and reported by the press at the time).

ai.txt (Spawning)

Where: /ai.txt at the site root.

Defined by: Spawning, the company behind Have I Been Trained and the Do Not Train registry. It uses robots.txt syntax (User-Agent, Allow, Disallow with file-type patterns). There is no published specification beyond Spawning's generator and FAQ; Spawning says its API passes ai.txt permissions to "a growing list of AI researchers and partners" without naming them.

Specification: Spawning: ai.txt generator and FAQ (read 25 Sep 2026).

How TDMRep's three places combine

TDMRep section 6.7 sets the order: a TDM agent reads /.well-known/tdmrep.json first, then the tdm-reservation header on each response, then the meta tag in each HTML page. Each later value replaces the earlier one, and a missing value never resets it. So a site-wide file saying 1 and a page header saying 0 means 0 for that page. In tdmrep.json, the first rule whose location matches the path wins, and patterns use robots.txt syntax (* and a final $). tdm-reservation is 1 (reserved) or 0 (not reserved); the report calls any other value a protocol error, to be treated as unset.

Content-Signal, line by line

A Content-Signal line sits inside a User-agent group of robots.txt, so it applies to the agents that group names. Its value is comma-separated key=value pairs: search, ai-input and ai-train take yes or no, and use (in Cloudflare's robots.txt documentation, last updated 3 Aug 2026) takes immediate, reference or full. A path before the pairs, as in Content-Signal: /blog/ ai-train=no, scopes the line to that path, following contentsignals.org's own examples. Leaving a key out expresses no preference for that use, in the policy's words it "neither grants nor restricts permission".

The law these signals point to

TDMRep, Content Signals and Spawning's ai.txt all say they answer Article 4 of the EU's 2019 copyright directive. Its paragraph 3 reads:

“The exception or limitation provided for in paragraph 1 shall apply on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online.”

Directive (EU) 2019/790, Article 4(3), EUR-Lex. Read 25 Sep 2026 from the Publications Office copy of the official text, because EUR-Lex answered our plain request with a browser challenge.

This page is not legal advice. It does not say whether any signal, or any combination, is a reservation "in an appropriate manner" under that article, in your country or anywhere else, and it does not cover the law outside the EU. The directive does not name any file or header. If it matters to you, ask a lawyer who practises in your jurisdiction.

What this checker cannot do

Made with AI tools. · Privacy