Which AI crawlers really visited your site, and which were fakes

Drop your server's access log. For each of the 19 AI agents in the directory you get hits, first and last visit, top paths and status codes, and how many requests came from the operator's published IP ranges. Anyone can put GPTBot in a user-agent; only OpenAI can send it from OpenAI's addresses. The log is read in this tab and never uploaded.

Drop an access log here or choose files

Nginx or Apache combined / common, Caddy JSON, or a CSV with a header row. .gz is fine; rotated files can be dropped together. The format is detected per file.

Agent	Hits	Verified	Spoofed	Cannot be verified	Seen	Details

Verdicts are by IP range only: the address in each log line is matched against the operator's published list. Reverse-DNS checks cannot be done from a browser, so they are not attempted.

Files read

IP lists used

How each request is judged

The check is by IP range only. Reverse-DNS checks cannot be done from a browser, so the DNS method that Google, Apple and Common Crawl also describe is not used; every operator on this page with a DNS method also publishes the list that is used.

A request belongs to an agent when its user-agent contains that agent's token, ignoring case. When two tokens match, the longer wins, so Applebot-Extended is not counted as Applebot. Then the address that made the request is looked up in the IP list that agent's operator publishes, as snapshotted by this site:

Verified The address is inside one of the operator's published ranges. The operator publishes these ranges as the addresses its crawler uses.
Spoofed The operator publishes ranges and the address is outside all of them. Something else is using the name: a user-agent is text the client chooses, and copying one costs nothing. Common Crawl's own page warns that other crawlers falsely claim to be CCBot.
Cannot be verified There is nothing to check against. We found no Bytespider documentation from ByteDance at all, and Meta's crawler documentation gives no list for its three agents, so their requests are counted and never guessed at. The same verdict appears when a list could not be loaded, or when the log holds a host name instead of an address.

Lists used: OpenAI (one each for GPTBot, OAI-SearchBot and ChatGPT-User), Anthropic (one list covering ClaudeBot, Claude-SearchBot and Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google's common-crawlers list, Apple's Applebot list, Common Crawl's CCBot list, and Amazon's three lists for Amazonbot, Amzn-SearchBot and Amzn-User. Each is a copy fetched from the operator's own URL, listed with its date under the results. The copies are in /data/ipranges/ if you want to check them.

A worked example

74.7.175.130 - - [20/Sep/2026:10:04:11 +0200] "GET /blog/post-1 HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4;"
203.0.113.9 - - [20/Sep/2026:10:05:02 +0200] "GET /wp-login.php HTTP/1.1" 404 162 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4;"

Both lines say GPTBot. The first address is inside 74.7.175.128/25, a range in OpenAI's GPTBot list, so it is verified. The second comes from 203.0.113.9, which is in no OpenAI range, so it is spoofed. It also asked for /wp-login.php, which a training crawler has no reason to want. The table lists spoofing addresses under each agent's details, ready for a firewall rule.

Behind a proxy or CDN

If Cloudflare, a load balancer or a reverse proxy sits in front of your server, the log may record the proxy's address for every request. Every GPTBot request would then look spoofed. The page warns when it sees the signs: verifiable agents that never verify, most traffic from private addresses such as 10.x or 172.16–31.x, or a busy log with only one to three addresses. To fix it, have the server log the client address the proxy passes on:

Nginx: set_real_ip_from with your proxy's ranges and real_ip_header X-Forwarded-For; (behind Cloudflare, CF-Connecting-IP). The realip module then puts the client's address in $remote_addr, which is the first field of the combined format.
Apache: mod_remoteip with RemoteIPHeader and RemoteIPTrustedProxy, and %a instead of %h in the LogFormat.
Caddy: set trusted_proxies. Its JSON log then fills request.client_ip, which this page reads before remote_ip.

Where to find the log

On Debian and Ubuntu, Nginx writes /var/log/nginx/access.log and Apache writes /var/log/apache2/access.log. On RHEL-family systems Apache uses /var/log/httpd/access_log. Older days are rotated to access.log.1, access.log.2.gz and so on, and you can drop them all at once. Caddy writes JSON wherever the log directive's output points. A CDN export works if it is a CSV whose header names the client IP, user-agent, path, status and time columns. Cloudflare Logpush names such as ClientIP, ClientRequestUserAgent, ClientRequestURI, EdgeResponseStatus and EdgeStartTimestamp are recognised.

What this page cannot do

No live DNS. Google, Apple and Common Crawl also describe a reverse-then-forward DNS check. A web page cannot make DNS lookups, so only the IP-range check is done. For those operators the published list is the other method they give.
Only the listed formats. Nginx or Apache combined and common, Caddy JSON, and CSV with a header row. A custom log_format that moves the address or the quoted fields, W3C/IIS logs, CloudFront's tab-separated logs and other JSON layouts are not read. A file in another shape is reported as not recognised, and nothing is guessed.
No logs larger than the tab can stream. The file is read in chunks by a background worker, so memory stays small and the page stays responsive. In our tests it reads roughly 25–30 MB of log per second on a desktop, so 1 GB takes well over half a minute, and longer on a phone. Per agent, the first 50,000 distinct paths are kept for the top-10 list.
Snapshots age. The IP lists are copies with the dates shown. If an operator adds a range after that date, its newest crawlers will show as spoofed here until the copy is refreshed. The date of each copy is shown with the results.
Only agents that say who they are. A crawler that sends a browser user-agent is counted under "everything else". Query strings are dropped from paths, so /a?page=2 counts as /a.
Control tokens. Google-Extended and Applebot-Extended never fetch pages. Google's and Apple's regular crawlers do the fetching, so a request that names either token is not documented behaviour, whatever its address.

Seen an agent you want to keep out? Generate a robots.txt for it, or check the one you have. Remember that robots.txt only asks. A spoofer ignores it, and blocking its addresses is the only thing that stops it.