resulthack

What an AI crawler that runs no JavaScript can read in your page

Paste your page's HTML source and see the text, headings, meta tags and structured data a crawler gets from the HTML alone, before any script runs. It works on pages that are not public yet. The HTML is parsed in this tab and nothing is fetched or sent.

Why the HTML alone matters

A crawler that does not run JavaScript reads the HTML your server sends and nothing else. If your page is built in the browser (a React, Vue or Svelte app without server rendering), that HTML is an empty container and a list of scripts. The check above counts what is actually there.

What operators and measurements say about running JavaScript:

The safe reading: if the words are not in the HTML, most AI crawlers do not get them, and Google gets them only after rendering.

What counts as readable text

Every text node inside <body>, in source order, with runs of spaces collapsed and each block element on its own line. Left out: <script>, <style>, <template>, <iframe>, <svg>, <canvas> and the <head>. <noscript> is shown on its own, because a client without JavaScript sees it and a browser with JavaScript never does. Text hidden with CSS is counted: it is in the HTML, and a crawler that runs no CSS reads it too.

Words are counted with the browser's Intl.Segmenter, which uses Unicode word boundaries and dictionaries for Chinese, Japanese and Thai, so 我们今天去公园散步 counts as 5 words, not 1. "Client-side" counts as 2 words and "don't" as 1.

The signs of a client-rendered shell

The fix is to send the content in the HTML: server-side rendering or static generation (Next.js, Nuxt, Astro, SvelteKit and Remix all do it), or prerendering for a plain single-page app.

How the X-Robots-Tag header combines with meta tags

Every rule a robots meta tag can carry can also be sent as a response header, which is how PDFs, images and whole sites behind a CDN get them. Google's Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 24 Mar 2026, checked 30 Sep 2026) set the rules this check applies:

Who says they read the header, in their own documentation, checked 30 Sep 2026:

What a status line means, from Google's How HTTP status codes affect Google's crawlers (last updated 4 Feb 2026, checked 30 Sep 2026): a redirect is followed for up to 10 hops, and the target is what gets indexed; a 4xx page is not indexed and is removed if it was; a 5xx or 429 makes the crawlers slow down, and pages already indexed are eventually dropped.

What this check cannot see

Made with AI tools. · Privacy