resulthack

Which AI crawlers really visited your site, and which were fakes

Drop your server's access log. For each of the 49 AI agents in the directory you get hits, first and last visit, top paths and status codes, and how many requests came from the operator's published IP ranges. Anyone can put GPTBot in a user-agent; only OpenAI can send it from OpenAI's addresses. It then counts the people ChatGPT, Perplexity, Microsoft Copilot, Gemini and eight other assistants sent you, per landing page and per day, beside how often their crawlers fetched the same page. The log is read in this tab and never uploaded.

How each request is judged

The check is by IP range only. Reverse-DNS checks cannot be done from a browser, so the DNS method that Google, Microsoft, Apple and Common Crawl also describe is not used here; those operators also publish the list that is used. Huawei (for PetalBot) and Yandex describe only the DNS method, so their agents read “cannot be verified” on this page; their pages in the directory give the DNS steps.

A request belongs to an agent when its user-agent contains that agent's token, ignoring case. When two tokens match, the longer wins, so Applebot-Extended is not counted as Applebot. A token that appears only inside a link in the user-agent does not count: Bing’s ads crawler, AdIdxBot, ends its user-agent with a link to Bing’s bingbot.htm page and is not counted as Bingbot. Then the address that made the request is looked up in the IP list that agent's operator publishes, as snapshotted by this site:

Lists used: OpenAI (one each for GPTBot, OAI-SearchBot and ChatGPT-User), Anthropic (one list covering ClaudeBot, Claude-SearchBot and Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google's common-crawlers list (Googlebot, GoogleOther, Google-CloudVertexBot) and its user-triggered-agents list (Google-Agent), Microsoft's Bingbot list, DuckDuckGo's DuckAssistBot list, Mistral's lists for MistralAI-Index and MistralAI-User, Apple's Applebot list, Common Crawl's CCBot list, and Amazon's three lists for Amazonbot, Amzn-SearchBot and Amzn-User. Each is a copy fetched from the operator's own URL, listed with its date under the results. The copies are in /data/ipranges/ if you want to check them.

A worked example

74.7.175.130 - - [20/Sep/2026:10:04:11 +0200] "GET /blog/post-1 HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4;"
203.0.113.9 - - [20/Sep/2026:10:05:02 +0200] "GET /wp-login.php HTTP/1.1" 404 162 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4;"

Both lines say GPTBot. The first address is inside 74.7.175.128/25, a range in OpenAI's GPTBot list, so it is verified. The second comes from 203.0.113.9, which is in no OpenAI range, so it is spoofed. It also asked for /wp-login.php, which a training crawler has no reason to want. The table lists spoofing addresses under each agent's details, ready for a firewall rule.

How people sent by an AI assistant are counted

Each line that is a page view by a person is checked against the assistant list: ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, DeepSeek, Meta AI, Mistral Le Chat, You.com, Phind, Grok and Poe. A line counts for an assistant when its Referer is on one of the assistant's hosts, subdomains included (www.perplexity.ai is Perplexity), or when the requested URL carries one of its utm_source values. OpenAI documents that ChatGPT adds utm_source=chatgpt.com to its links. For the others the hosts are simply where their chat runs, which is observed, not documented; the list says which is which.

203.0.113.20 - - [20/Sep/2026:09:12:40 +0000] "GET /pricing/ HTTP/1.1" 200 8120 "https://chatgpt.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) … Safari/604.1"
203.0.113.21 - - [20/Sep/2026:11:02:05 +0000] "GET /pricing/?utm_source=chatgpt.com HTTP/1.1" 200 8120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) … Chrome/140.0.0.0 Safari/537.36"
203.0.113.20 - - [20/Sep/2026:09:12:41 +0000] "GET /img/plans.webp HTTP/1.1" 200 40960 "https://chatgpt.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) … Safari/604.1"

That is two people ChatGPT sent to /pricing/: one known by the Referer, one by utm_source after the app dropped the Referer. The third line is the first visitor's browser loading an image, which is not a page view. A line is a page view when the user-agent names no directory agent and does not say bot, crawler or spider; the status is 2xx or 304; the method is GET; and the path is not a stylesheet, script, image, font, .json, .xml, .txt, favicon or web manifest. Clicks from Google and Bing search are counted apart, for comparison, and never as an assistant. The CSV has one row per day, assistant and landing page, with the header date,assistant,page,visits.

Behind a proxy or CDN

If Cloudflare, CloudFront, a load balancer or a reverse proxy sits in front of your server, the origin server's own log records the proxy's address for every request unless it is told to log the client's. Every GPTBot request would then look spoofed. The page warns when it sees the signs: verifiable agents that never verify, most traffic from private addresses such as 10.x or 172.16–31.x, or a busy log with only one to three addresses. Either read the CDN's or load balancer's own log instead (Cloudflare's ClientIP, CloudFront's c-ip and the load balancer's client:port are the visitor, see the formats), or have the server log the client address the proxy passes on:

Where to find the log

On Debian and Ubuntu, Nginx writes /var/log/nginx/access.log and Apache writes /var/log/apache2/access.log. On RHEL-family systems Apache uses /var/log/httpd/access_log. Older days are rotated to access.log.1, access.log.2.gz and so on, and you can drop them all at once. Caddy writes JSON wherever the log directive's output points. A CDN or hosting export in CSV works if its header names the client IP, user-agent, path, status and time columns. IIS writes daily files such as u_ex260920.log under %SystemDrive%\inetpub\logs\LogFiles\W3SVC1; CloudFront, Cloudflare Logpush and the AWS load balancer deliver gzipped files to the bucket you configured, and those can be dropped as they are.

Formats read, and what each cannot show

The format is decided per file from its first 20 lines. Fields are read by name wherever the format names them, so a header with extra or reordered fields still works. Each format below was checked against the vendor's own field reference on 2026-09-26.

A file in none of these shapes is not guessed at. The page says what it saw instead, for example JSON lines with keys host, path, agent or a #Fields header without c-ip or cs(User-Agent), so you can tell whether a field is missing or the format is simply another one.

What this page cannot do

Made with AI tools. · Privacy