Which AI-training opt-out signals does your site send?
There are at least five ways to tell AI companies not to train on your site, each defined by someone different. Paste your files and see which signals you send, whether each is written the way its definition says, and which AI operators say in their own documentation that they read it. Everything is read in this tab and nothing is fetched or sent.
View source (Ctrl+U, or Cmd+Option+U on a Mac) and copy from <head> to </head>.
In a terminal: curl -sI followed by your page's address. Or the browser's Network panel: click the page request, then Response Headers.
The robots.txt rules and tdmrep.json locations are matched against this path.
Paste at least one of the five first.
Problems found
None: every signal found is written the way its definition says.
Per-agent robots.txt training tokens
Token
Operator
Purpose
Result
Decided by
Content Signals
TDMRep
noai and noimageai
ai.txt
The signals you are missing
Written for the files you pasted. A box is left out when you already send that signal correctly. Add only the ones you want: each is a separate request, read by whoever chooses to read it.
Crawlers whose data is used for AI training by others, or whose operator documents no purpose
The checker above counts a token as an AI-training opt-out only when its operator writes that blocking it keeps your content out of training. The six below do not meet that bar, but many sites block them for the same reason, and nytimes.com's robots.txt blocks all six. CCBot fills Common Crawl's free web archive, which its operator describes as being for research and analysis and which third parties document as a major source of AI training data. Four have no purpose their operator documents: anthropic-ai, FacebookBot and cohere-ai are not listed on the crawler pages of Anthropic, Meta and Cohere, the companies they are attributed to, and ByteDance publishes no page about Bytespider. ImagesiftBot's operator, Hive, documents an image-search index, not training; it is here because Cloudflare counts it among AI bots and block lists carry it.
Each card quotes the operator first, then named third parties, labelled as such. A third party's claim is theirs, not the operator's and not ours. Whether your robots.txt blocks each one is decided with the same RFC 9309 matching as the table above, for the page path you give. The generator adds a crawler only when you tick it. The same data is in opt-out-others.json.
Paste a robots.txt into the checker above to see whether it blocks CCBot.
Common Crawl documents its archive as a free copy of the web for research and analysis, not as AI training. Its own Terms of Use name training AI systems as one use others make of the archive, and third parties document that many large language models are trained on it.
What the operator says
“Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the Internet to Internet researchers, companies and individuals at no cost for the purpose of research and analysis.”
“People have used the data to improve language translation software, predict trends, track disease propagation, and much more.”
“use of Crawled Content in connection with artificial intelligence, machine learning, or other similar technologies, including, without limitation, large language models and neural networks (collectively, "AI Systems"), including the use of Crawled Content for the purpose of developing, training, or deploying AI Systems”
“Add these lines to your robots.txt file and our crawler will stop crawling your website: User-agent: CCBot Disallow: /”
What others say third party, not the operator
Third party: Mozilla Foundation
“This data has been valuable for many researchers, but since 2020 when OpenAI published GPT-3, the large language model (LLM) that still powers the free version of ChatGPT, Common Crawl has become one of the most important sources of training data for generative AI.”
Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)
“Provides open crawl dataset, used for many purposes, including Machine Learning/AI.”
Third party: Cloudflare
“In the graph below, which looks at disallowed crawlers for these sites, we see that customers most often reference GPTBot, CCBot, and Google in robots.txt, but do not specifically disallow popular AI crawlers like Bytespider and ClaudeBot.”
BytespiderByteDance (attributed by third parties; ByteDance documents no crawler of this name)
Paste a robots.txt into the checker above to see whether it blocks Bytespider.
No page of ByteDance's that we could find names Bytespider. Third parties attribute it to ByteDance and report it collects training data for ByteDance's language models.
What the operator says
The operator documents no purpose for Bytespider (checked 26 Sep 2026).
“Operated by ByteDance, the Chinese company that owns TikTok, Bytespider is reportedly used to gather training data for its large language models (LLMs), including those that support its ChatGPT rival, Doubao.”
Third party: Cloudflare
“Among the top AI bots that we see, Bytespider not only leads in terms of number of requests but also in both the extent of its Internet property crawling and the frequency with which it is blocked.”
Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)
“Downloads data to train LLMS, including ChatGPT competitors.”
cohere-aiCohere (attributed by third parties; Cohere's crawler page does not name this token)
Paste a robots.txt into the checker above to see whether it blocks cohere-ai.
Cohere's crawler page does not name cohere-ai and says Cohere does not crawl the web to train its models. A third-party list describes cohere-ai as fetching pages for user prompts.
What the operator says
The operator documents no purpose for cohere-ai (checked 26 Sep 2026).
“We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to train generative AI foundation models at this time.”
“Should Cohere make use of these tools to crawl or scrape web content to train AI models in the future, Cohere will identify those bots and/or user agents in the table below alongside information about how to disallow their access to your site.”
What others say third party, not the operator
Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)
“Retrieves data to provide responses to user-initiated prompts.”
anthropic-aiAnthropic (attributed by third parties; Anthropic's crawler page does not name this token)
Paste a robots.txt into the checker above to see whether it blocks anthropic-ai.
Anthropic's crawler page lists three bots and not anthropic-ai; the one it documents for training is ClaudeBot, which is in the documented group above. A third-party list still describes anthropic-ai as a training scraper.
What the operator says
The operator documents no purpose for anthropic-ai (checked 26 Sep 2026).
“Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.”
What others say third party, not the operator
Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)
“Scrapes data to train LLMs and AI products offered by Anthropic.”
FacebookBotMeta (attributed by third parties; Meta's crawler page does not name this token)
Paste a robots.txt into the checker above to see whether it blocks FacebookBot.
Meta's crawler page lists five crawlers and not FacebookBot; the one it documents for AI training is Meta-ExternalAgent, which is in the documented group above. A third-party list says Meta once described FacebookBot as training speech recognition.
What the operator says
The operator documents no purpose for FacebookBot (checked 24 Sep 2026).
developers.facebook.com/documentation/sharing/webmasters/web-crawlers, checked 24 Sep 2026: Loaded; lists FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. No mention of FacebookBot. (On 2026-09-26 the page returned no readable content to a plain GET, so the 24 Sep reading stands.)
What others say third party, not the operator
Third party: ai.robots.txt (community-maintained list of AI crawlers, MIT licence)
“Officially used for training Meta "speech recognition technology," unknown if used to train Meta AI specifically.”
Paste a robots.txt into the checker above to see whether it blocks ImagesiftBot.
Hive documents ImagesiftBot as collecting images, page text and alt text for an image index its “web intelligence products” search. It does not call this AI training, and no third party we checked says it is.
What the operator says
“ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products”
“Once images and text are downloaded from a webpage, ImageSift analyzes this data from the page and stores the information in an index. Our web intelligence products use this index to enable search and retrieval of similar images.”
How it reads robots.txt: The verdict here uses RFC 9309 (its own group, else User-agent: *). If your robots.txt has no ImagesiftBot group but has a Googlebot group, Hive says ImagesiftBot follows the Googlebot group instead. In its words: “If there is no rule targeting ImagesiftBot, but there is a rule targeting Googlebot, then ImagesiftBot will follow the Googlebot directives.” (imagesift.com, checked 26 Sep 2026)
What others say third party, not the operator
Third party: Cloudflare
“ImagesiftBot 8.75%”
Who says they honour which signal
Each cell comes from the operator's own documentation, quoted, with the page and the date we read it. "Not documented" means that page does not mention the signal: it is the absence of a statement, not proof the signal is ignored, and an operator may read a signal it does not write about. It also cannot tell you whether a crawler actually complies. The same table is in opt-out-signals.json.
Per-agent robots.txt tokensDocumentedGPTBotDisallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.OpenAI documentation, read 25 Sep 2026
Per-agent robots.txt tokensDocumentedClaudeBotWhen a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.Anthropic documentation, read 25 Sep 2026
Per-agent robots.txt tokensDocumentedGoogle-ExtendedGoogle-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.Google-Extended also controls grounding in Gemini Apps, not only training. Google says it does not affect inclusion in Google Search.Google documentation, read 25 Sep 2026
Per-agent robots.txt tokensDocumentedApplebot-ExtendedWith Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s general purpose foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.Apple documentation, read 25 Sep 2026
Content SignalsNot documented
TDMRepNot documented
noai and noimageaiNot documented
ai.txtNot documented
Meta“Not documented”: not mentioned on Meta’s crawler page, read 25 Sep 2026
Per-agent robots.txt tokensDocumentedMeta-ExternalAgentIn order to block these crawlers, add a disallow for the relevant crawler to robots.txt.Meta documentation, read 25 Sep 2026
Content SignalsNot documented
TDMRepNot documented
noai and noimageaiPrefers robots.txtWe make it easy for site managers and content owners to indicate their preferences by using industry-standard practices like robots.txt rather than non-standard formats like NoAI tags.Meta documentation, read 25 Sep 2026
Per-agent robots.txt tokensDocumentedAmazonbotAutomated crawling from these listed user agents respects the Robots Exclusion Protocol , honoring the user-agent and the allow/disallow directives.Amazon documentation, read 25 Sep 2026
Per-agent robots.txt tokensDocumentedMistralAI-TrainingMistralAI-Training crawls web content to help build datasets for training Mistral generative AI models. Webmasters can disallow this user agent in their robots.txt file.Mistral AI documentation, read 25 Sep 2026
Per-agent robots.txt tokensDocumentedCCBotAdd these lines to your robots.txt file and our crawler will stop crawling your website:This stops CCBot crawling at all. Common Crawl’s own pages do not describe CCBot as an AI-training crawler; its dataset is widely used for training by others.Common Crawl documentation, read 25 Sep 2026
Content SignalsNot documented
TDMRepNot documented
noai and noimageaiNot documented
ai.txtNot documented
Only one kind of signal has written support from all eight: a robots.txt line naming the operator's own token. Meta is the only one of the eight that addresses noai at all, and it points to robots.txt instead. None of the eight crawler pages mentions Content-Signal, TDMRep or ai.txt.
The five signals, and who defined each
Per-agent robots.txt tokens
Where: robots.txt at the site root.
Defined by: robots.txt is RFC 9309 (IETF, 2022). Each training token is defined by the operator that reads it: GPTBot by OpenAI, Google-Extended by Google, and so on.
Content Signals (Content-Signal line in robots.txt)
Where: robots.txt, inside a User-agent group.
Defined by: Cloudflare, the Content Signals Policy, announced 24 Sep 2025. It defines search, ai-input and ai-train (yes or no); Cloudflare's robots.txt documentation (last updated 3 Aug 2026) adds a fourth field, use, with the values immediate, reference or full.
Where: /.well-known/tdmrep.json, the tdm-reservation and tdm-policy HTTP headers, and meta tags of the same names.
Defined by: The W3C Text and Data Mining Reservation Protocol Community Group; Final Community Group Report of 10 May 2024, editor Laurent Le Meur (EDRLab). The report itself says it is not a W3C Standard nor on the W3C Standards Track.
Where: a robots meta tag, or an X-Robots-Tag response header.
Defined by: DeviantArt, November 2022, for its own artists' pages; other sites copied it. It is not in Google's or Bing's robots meta documentation and has no published specification.
Defined by: Spawning, the company behind Have I Been Trained and the Do Not Train registry. It uses robots.txt syntax (User-Agent, Allow, Disallow with file-type patterns). There is no published specification beyond Spawning's generator and FAQ; Spawning says its API passes ai.txt permissions to "a growing list of AI researchers and partners" without naming them.
TDMRep section 6.7 sets the order: a TDM agent reads /.well-known/tdmrep.json first, then the tdm-reservation header on each response, then the meta tag in each HTML page. Each later value replaces the earlier one, and a missing value never resets it. So a site-wide file saying 1 and a page header saying 0 means 0 for that page. In tdmrep.json, the first rule whose location matches the path wins, and patterns use robots.txt syntax (* and a final $). tdm-reservation is 1 (reserved) or 0 (not reserved); the report calls any other value a protocol error, to be treated as unset.
Content-Signal, line by line
A Content-Signal line sits inside a User-agent group of robots.txt, so it applies to the agents that group names. Its value is comma-separated key=value pairs: search, ai-input and ai-train take yes or no, and use (in Cloudflare's robots.txt documentation, last updated 3 Aug 2026) takes immediate, reference or full. A path before the pairs, as in Content-Signal: /blog/ ai-train=no, scopes the line to that path, following contentsignals.org's own examples. Leaving a key out expresses no preference for that use, in the policy's words it "neither grants nor restricts permission".
The law these signals point to
TDMRep, Content Signals and Spawning's ai.txt all say they answer Article 4 of the EU's 2019 copyright directive. Its paragraph 3 reads:
“The exception or limitation provided for in paragraph 1 shall apply on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online.”
This page is not legal advice. It does not say whether any signal, or any combination, is a reservation "in an appropriate manner" under that article, in your country or anywhere else, and it does not cover the law outside the EU. The directive does not name any file or header. If it matters to you, ask a lawyer who practises in your jurisdiction.
What this checker cannot do
It cannot fetch your site. You paste the files. A site that serves a different robots.txt or different headers to crawlers, or a CDN that adds its own (Cloudflare's managed robots.txt prepends lines to yours), is only checked as far as what you paste shows.
It cannot tell whether a crawler complies. The table says what operators write, not what their crawlers do. To see what actually visits, read your access log on the log reader.
It cannot see signals in files. TDMRep also defines metadata inside EPUB and PDF files, and IPTC defines a data-mining field inside image metadata. Neither is read here.
It cannot read a TDM policy. It checks that tdm-policy is a URL, not what the policy at that URL says.
It cannot make a signal binding. Every one of these is a request. The generator writes them correctly; whether anyone reads them is up to the reader.