CloudflareBrowserRenderingCrawler
Cloudflare · Unconfirmed · robots.txt: Yes
Crawls a site from a starting URL on behalf of a Cloudflare customer using Browser Run's /crawl endpoint, following links up to a set depth or page limit and returning the pages as HTML, Markdown or JSON.
Block CloudflareBrowserRenderingCrawler
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /Unconfirmed by the operator Cloudflare's documentation we checked does not say what blocking CloudflareBrowserRenderingCrawler changes beyond the crawler itself.
Allow CloudflareBrowserRenderingCrawler
User-agent: CloudflareBrowserRenderingCrawler
Allow: /An allow group only changes anything if a broader rule would block CloudflareBrowserRenderingCrawler. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks CloudflareBrowserRenderingCrawler on a given path, and which line decides.
Who blocks it
CloudflareBrowserRenderingCrawler is not in the 2026-10-09 survey: it was added to this directory after we last read the most-visited sites' robots.txt files, so we have no count for it yet.
Facts
- Operator
- Cloudflare
- Purpose
- Unconfirmed by the operator
Cloudflare documents it as the crawler behind the /crawl endpoint of Browser Run, which Cloudflare's customers call to scrape a site from a starting URL and get its pages back as HTML, Markdown or JSON. The customer, not Cloudflare, decides the use, and declares it as search, ai-input or ai-train for sites that publish Content Signals.
- Obeys robots.txt
- Yes. Cloudflare says the /crawl endpoint respects robots.txt, including Crawl-delay (0.5 seconds between requests to a domain when none is set), and lists every URL robots.txt keeps it from as "disallowed". It also refuses a crawl when the site's Content-Signal line says no to a use the customer declared.
- In your server logs
- The user-agent is "CloudflareBrowserRenderingCrawler/1.0"; Cloudflare says it cannot be changed on the /crawl endpoint.
- How to verify it
- Cloudflare says the user-agent is not a reliable way to identify Browser Run requests and points to its Web Bot Auth signatures (Signature, Signature-Input and Signature-Agent headers), which can be checked against Cloudflare's published keys.
What Cloudflare says
Quoted word for word from the operator's documentation.
“The /crawl endpoint uses CloudflareBrowserRenderingCrawler/1.0 as its User-Agent, which is different from other Quick Actions endpoints.”
“The /crawl endpoint identifies itself as CloudflareBrowserRenderingCrawler/1.0.”
“The /crawl endpoint scrapes content from a starting URL and follows links across the site, up to a configurable depth or page limit.”
“Responses can be returned as HTML, Markdown, or JSON.”
“The /crawl endpoint respects the directives of robots.txt files, including crawl-delay.”
“If a site does not specify a crawl-delay in its robots.txt, the crawler uses a default delay of 0.5 seconds between requests to the same domain to avoid overwhelming the origin server.”
“All URLs that /crawl is directed not to crawl are listed in the response with "status": "disallowed".”
“Allowed values: search, ai-input, ai-train.”
“This User-Agent is not customizable.”
“If a target site sets any of those content signals to no, the crawl request will be rejected at initiation with a 400 Bad Request error unless you explicitly narrow your declared purposes using the crawlPurposes parameter to exclude the disallowed use.”
“The User-Agent header is not a reliable way to identify Browser Run requests.”
“To verify a request originated from Cloudflare Browser Run, use the keys found on this directory”
“Destination servers should use the non-configurable headers and Web Bot Auth signatures below, which provide cryptographic proof that a request originated from Cloudflare Browser Run.”
Sources
- Cloudflare Browser Run docs: /crawl - Crawl web content — fetched and checked 2026-10-09
- Cloudflare Browser Run docs: Automatic request headers — fetched and checked 2026-10-09