CCBot
Common Crawl · Unconfirmed · robots.txt: Yes
Crawls a sample of the web for Common Crawl's free, openly downloadable archive of web pages.
Block CCBot
User-agent: CCBot
Disallow: /Common Crawl says CCBot stops crawling your site once robots.txt disallows it. Its pages we checked do not say whether pages already in the archive are removed; it separately runs an opt-out registry.
Allow CCBot
User-agent: CCBot
Allow: /An allow group only changes anything if a broader rule would block CCBot. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Facts
- Operator
- Common Crawl
- Purpose
- Unconfirmed by the operator
Common Crawl describes its purpose as a free, open copy of the web for research and analysis by anyone. Its pages we checked do not say whether the archive is used to train AI models, so we do not file it under training, search or user fetch.
- Obeys robots.txt
- Yes. Common Crawl says CCBot checks robots.txt before fetching, obeys Crawl-delay, honours link-level nofollow and keeps re-checking robots.txt for changes.
- In your server logs
- The user-agent string contains "CCBot/2.0". Common Crawl warns that other crawlers falsely identify themselves as CCBot.
- How to verify it
- CCBot runs on dedicated IP ranges with reverse DNS in crawl.commoncrawl.org (not yet over IPv6); check with a reverse and forward lookup, or match the IP against the published list.
Published list: index.commoncrawl.org/ccbot.json — 4 IPv4 and 1 IPv6 ranges when we fetched it, list dated 11 Aug 2026. Always match against the live list; operators update them.
What Common Crawl says
Quoted word for word from the operator's documentation.
“To prevent Common Crawl from crawling your website, include the following in your robots.txt : User-agent: CCBot Disallow: /”
“Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the Internet to Internet researchers, companies and individuals at no cost for the purpose of research and analysis.”
“Common Crawl's dataset is a sample of the web, and we do not generally archive any entire website but a randomly selected subset of it.”
“CCBot is an automated crawler, checking first the robots.txt , and if crawling a page is allowed, fetches pages using HTTP GET requests.”
“We obey the Crawl-delay parameter for robots.txt .”
“We currently honor the nofollow attribute as it applies to links embedded on your site.”
“Add these lines to your robots.txt file and our crawler will stop crawling your website:”
“We will periodically continue to check if the robots.txt file has been updated.”
“You may also wish to be added to our opt-out registry.”
“Please note that we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity.”
“CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)”
Sources
- Common Crawl: CCBot — fetched and checked 2026-09-22
- Common Crawl: FAQ — fetched and checked 2026-09-22
- Common Crawl: CCBot IP ranges (ccbot.json) — fetched and checked 2026-09-22