What an AI crawler that runs no JavaScript can read in your page
Paste your page's HTML source and see the text, headings, meta tags and structured data a crawler gets from the HTML alone, before any script runs. It works on pages that are not public yet. The HTML is parsed in this tab and nothing is fetched or sent.
Use view source (Ctrl+U, or Cmd+Option+U on a Mac), not the browser's Elements panel: the Elements panel shows the page after JavaScript has run.
Run curl -sI YOUR-URL (add -A "GPTBot" to be sent what a crawler is sent) and paste all of it. X-Robots-Tag and Link: rel="canonical" are read with the meta tags; this page still fetches nothing.
Paste some HTML first.
The text a crawler reads
Headings outline
Title, description, canonical, language
Robots and per-crawler meta tags
Any <meta name> named robots, googlebot, bingbot or one of the 49 tokens in the directory (case does not matter), with each directive decoded.
No robots or per-crawler meta tags. Crawlers treat that as index, follow.
Headers and meta tags together, per crawler
Where a header and a meta tag disagree, the stricter rule wins, wherever it sits: Google, “In the case of conflicting robots rules, the more restrictive rule applies.” Each row names the source that decided it.
Structured data (JSON-LD)
No <script type="application/ld+json"> blocks.
Links
Why the HTML alone matters
A crawler that does not run JavaScript reads the HTML your server sends and nothing else. If your page is built in the browser (a React, Vue or Svelte app without server rendering), that HTML is an empty container and a list of scripts. The check above counts what is actually there.
What operators and measurements say about running JavaScript:
- Googlebot renders JavaScript, later. Google: "Once Google's resources allow, a headless Chromium renders the page and executes the JavaScript." Pages wait in a render queue that "may stay on this queue for a few seconds, but it can take longer than that." (Google Search Central, JavaScript SEO basics, last updated 4 Mar 2026, checked 24 Sep 2026.)
- Applebot may render. Apple: "Applebot may render the content of your website within a browser", and if JavaScript and CSS are blocked by robots.txt "it may not be able to render the content properly." (Apple, About Applebot, checked 24 Sep 2026.)
- Bingbot renders. Microsoft's guidelines ask sites to "Allow Bingbot to crawl and render content efficiently." (Bing Webmaster Guidelines, checked 24 Sep 2026.)
- OpenAI, Anthropic, Meta, ByteDance and Perplexity crawlers: measured not to render. Vercel's study of its own network, published 17 Dec 2024, found "none of the major AI crawlers currently render JavaScript", naming OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. It adds that ChatGPT's and Claude's crawlers "do fetch JavaScript files (ChatGPT: 11.50%, Claude: 23.84% of requests), they don't execute them." (Vercel, The rise of the AI crawler, read 24 Sep 2026.) That is a third party's measurement from 2024, not a statement by the operators, and crawlers change.
- Everyone else in the directory (Amazonbot, DuckAssistBot, the MistralAI agents, PetalBot, CCBot and the rest): whether they run JavaScript is not documented by the operator, as far as we have found, and we have read no published measurement. Assume they get the raw HTML.
The safe reading: if the words are not in the HTML, most AI crawlers do not get them, and Google gets them only after rendering.
What counts as readable text
Every text node inside <body>, in source order, with runs of spaces collapsed and each block element on its own line. Left out: <script>, <style>, <template>, <iframe>, <svg>, <canvas> and the <head>. <noscript> is shown on its own, because a client without JavaScript sees it and a browser with JavaScript never does. Text hidden with CSS is counted: it is in the HTML, and a crawler that runs no CSS reads it too.
Words are counted with the browser's Intl.Segmenter, which uses Unicode word boundaries and dictionaries for Chinese, Japanese and Thai, so 我们今天去公园散步 counts as 5 words, not 1. "Client-side" counts as 2 words and "don't" as 1.
The signs of a client-rendered shell
- An empty
#root,#app,#__nextor#__nuxtelement, the mount points React, Vue, Next.js and Nuxt fill in. - A
<body>holding only scripts and empty elements. - A
<noscript>that says to enable JavaScript. Create React App ships "You need to enable JavaScript to run this app." - Text that exists only inside a
__NEXT_DATA__,__NUXT_DATA__orwindow.__NUXT__script blob. A server-rendered Next.js page has the blob and the text in its HTML, which is fine; the check flags it only when the blob holds far more words than the page.
The fix is to send the content in the HTML: server-side rendering or static generation (Next.js, Nuxt, Astro, SvelteKit and Remix all do it), or prerendering for a plain single-page app.
How the X-Robots-Tag header combines with meta tags
Every rule a robots meta tag can carry can also be sent as a response header, which is how PDFs, images and whole sites behind a CDN get them. Google's Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 24 Mar 2026, checked 30 Sep 2026) set the rules this check applies:
- Several headers add up. “Multiple X-Robots-Tag headers can be combined within the HTTP response, or you can specify a comma-separated list of rules.”
- A header can name one crawler.
X-Robots-Tag: googlebot: noindex, nofollowbinds Googlebot only; “Rules specified without a user agent are valid for all crawlers.” Names and values are not case sensitive. - The stricter rule wins. “If a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply.” A
noindexin the header beats anindexin the HTML, and the other way round. unavailable_after: “Do not show this page in search results after the specified date/time”, written in RFC 822, RFC 850 or ISO 8601. An RFC 822 date such asFri, 25 Jun 2010 15:00:00 GMTcontains a comma; the check keeps it whole.Link: <https://…>; rel="canonical"names the canonical URL the way the<link>element does. Google: “use absolute URLs in the rel="canonical" HTTP header”, and do not give one URL by one method and a different one by another (Google, consolidate duplicate URLs, last updated 10 Jul 2026, checked 30 Sep 2026).
Who says they read the header, in their own documentation, checked 30 Sep 2026:
- Google (Googlebot): yes, in the specification above.
- Microsoft (Bingbot): yes. The directives “can be present as Meta Tags in the page HTML or X-Robots-Tag Tags in the HTTP Header” (Bing Webmaster Blog, 3 Jun 2008). That post lists
noindex,nofollow,nosnippet,noarchive/nocacheandnoodp; Bing's current help pages did not render for us, so a rule outside that list (such asunavailable_after) gets “operator does not say” for Bingbot. - Apple (Applebot): yes. “Applebot also supports indexing directives delivered via the X-Robots-Tag HTTP response header” (About Applebot), and it lists
noindex,nosnippet,nofollow,noneandall. - Amazon (Amazonbot): meta tags yes, header not stated. Amazon lists the meta tags
noarchive,noindexandnoneand does not mention X-Robots-Tag (Amazonbot). A rule only the header carries gets “operator does not say”. - Everyone else. The crawler pages of OpenAI, Anthropic, Perplexity, Meta, Mistral, DuckDuckGo and Common Crawl describe robots.txt only; none mentions robots meta tags or X-Robots-Tag. For them, and for any crawler not listed above, the check says “operator does not say”, not a guess. To keep them out, use robots.txt.
What a status line means, from Google's How HTTP status codes affect Google's crawlers (last updated 4 Feb 2026, checked 30 Sep 2026): a redirect is followed for up to 10 hops, and the target is what gets indexed; a 4xx page is not indexed and is removed if it was; a 5xx or 429 makes the crawlers slow down, and pages already indexed are eventually dropped.
What this check cannot see
- HTTP headers you do not paste. An
X-Robots-Tag: noindexheader has the same effect as the meta tag but is not in the HTML. Paste yourcurl -sIoutput in the second box and it is read with the meta tags. This page never requests your URL, so it knows only the headers you paste, and a server or CDN may send a crawler different headers than it sends curl. - robots.txt. If robots.txt blocks a page, a crawler never fetches it and never sees these meta tags. Google says such rules "will not be found and will therefore be ignored" (robots meta tag specification, last updated 24 Mar 2026, checked 24 Sep 2026). Check your robots.txt for that.
- What a rendering crawler adds. Googlebot, and by their operators' own words Applebot and Bingbot, may run your scripts later and see more than this page shows. This check shows the floor, not the ceiling.
- What your server sends a bot. View-source shows what your browser was sent. A bot-protection layer may send a crawler a challenge page instead. To see that, fetch the page with the crawler's user-agent, for example
curl -s -A "GPTBot" YOUR-URL, and paste the result. Each agent's full user-agent string is on its directory page. - Anything loaded later. Iframes, images, and content fetched by scripts are not followed. Nothing on this page makes a request for your markup.
- Whether a crawler honours a tag.
noaiandnoimageaiare conventions, not standards, and a meta tag named after a crawler works only if its operator reads it. Both are shown decoded, not promised.