Which AI crawlers really visited your site, and which were fakes
Drop your server's access log. For each of the 49 AI agents in the directory you get hits, first and last visit, top paths and status codes, and how many requests came from the operator's published IP ranges. Anyone can put GPTBot in a user-agent; only OpenAI can send it from OpenAI's addresses. It then counts the people ChatGPT, Perplexity, Microsoft Copilot, Gemini and eight other assistants sent you, per landing page and per day, beside how often their crawlers fetched the same page. The log is read in this tab and never uploaded.
This browser lacks the streaming or worker support the log reader needs. A current Chrome, Edge, Firefox or Safari has it.
Drop an access log here or
Nginx or Apache combined / common, Caddy JSON, Cloudflare Logpush JSON, CloudFront, AWS load balancer, IIS, or a CSV with a header row (details). .gz is fine; rotated files can be dropped together. The format is detected per file.
This log may record a proxy's address, not the crawler's. If so, the verified and spoofed counts below are wrong: they judge your proxy or CDN, not the visitor.
Log the original client address instead (see behind a proxy or CDN below) and read the log again.
| Agent | Hits | Verified | Spoofed | Cannot be verified | Seen | Details |
|---|
Verdicts are by IP range only: the address in each log line is matched against the operator's published list. Reverse-DNS checks cannot be done from a browser, so they are not attempted.
People sent by AI assistants
| Assistant | Visits | Landing pages | By day (UTC) |
|---|
Per page: crawler fetches beside people sent
| Page | People sent | By assistant | AI crawler fetches |
|---|
A visit is a page view by a browser that does not call itself a bot, answered 2xx or 304, for a page rather than an image, script, stylesheet or font. It counts for an assistant when its Referer is the assistant's site or its link carries the assistant's utm_source. The count is a floor: clicks that arrive without either are not counted.
Files read
IP lists used
How each request is judged
The check is by IP range only. Reverse-DNS checks cannot be done from a browser, so the DNS method that Google, Microsoft, Apple and Common Crawl also describe is not used here; those operators also publish the list that is used. Huawei (for PetalBot) and Yandex describe only the DNS method, so their agents read “cannot be verified” on this page; their pages in the directory give the DNS steps.
A request belongs to an agent when its user-agent contains that agent's token, ignoring case. When two tokens match, the longer wins, so Applebot-Extended is not counted as Applebot. A token that appears only inside a link in the user-agent does not count: Bing’s ads crawler, AdIdxBot, ends its user-agent with a link to Bing’s bingbot.htm page and is not counted as Bingbot. Then the address that made the request is looked up in the IP list that agent's operator publishes, as snapshotted by this site:
- Verified The address is inside one of the operator's published ranges. The operator publishes these ranges as the addresses its crawler uses.
- Spoofed The operator publishes ranges and the address is outside all of them. Something else is using the name: a user-agent is text the client chooses, and copying one costs nothing. Common Crawl's own page warns that other crawlers falsely claim to be
CCBot. This word is used for every list except one whose copy is older than the operator's own update rhythm (next). - Not in OpenAI’s list of date Used instead of Spoofed only for OpenAI's
ChatGPT-Userlist, and only when this site's copy is more than one day older than your clock. OpenAI regenerates that list daily: its owncreationTimeread 2026-09-22 18:05 in the copy taken on 2026-09-23 and 2026-10-07 16:05 in the copy taken on 2026-10-08, and it grew from 229 to 234 ranges in between. So an address OpenAI added after the copy's date would read as outside it. The cell says which dated list the address is not in and adds one line, for example “OpenAI regenerates this list daily; this copy is 3 days old, so a new address can read as outside it”. The age is worked out in your browser from the list's date and your clock. If an address keeps showing up, look it up in the currentchatgpt-user.jsonon openai.com before blocking it. While the copy is less than a day old, an address outside it reads Spoofed as usual. The other lists either keep the samecreationTimefor weeks or months (GPTBot's still read 2026-09-22 on 2026-10-08, Bingbot's 2024-01-03) or publish no update rhythm, so outside them means Spoofed. - Cannot be verified There is nothing to check against. We found no Bytespider documentation from ByteDance at all, Meta's crawler documentation gives no list for its three agents, and Mistral (for
MistralAI-Training), Diffbot, Hive (ImagesiftBot), Huawei and Yandex publish none either, so their requests are counted and never guessed at. Tokens the operator no longer documents, such asanthropic-ai, get the same verdict. The same verdict appears when a list could not be loaded, or when the log holds a host name instead of an address.
Lists used: OpenAI (one each for GPTBot, OAI-SearchBot and ChatGPT-User), Anthropic (one list covering ClaudeBot, Claude-SearchBot and Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google's common-crawlers list (Googlebot, GoogleOther, Google-CloudVertexBot) and its user-triggered-agents list (Google-Agent), Microsoft's Bingbot list, DuckDuckGo's DuckAssistBot list, Mistral's lists for MistralAI-Index and MistralAI-User, Apple's Applebot list, Common Crawl's CCBot list, and Amazon's three lists for Amazonbot, Amzn-SearchBot and Amzn-User. Each is a copy fetched from the operator's own URL, listed with its date under the results. The copies are in /data/ipranges/ if you want to check them.
A worked example
74.7.175.130 - - [20/Sep/2026:10:04:11 +0200] "GET /blog/post-1 HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4;"
203.0.113.9 - - [20/Sep/2026:10:05:02 +0200] "GET /wp-login.php HTTP/1.1" 404 162 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4;"
Both lines say GPTBot. The first address is inside 74.7.175.128/25, a range in OpenAI's GPTBot list, so it is verified. The second comes from 203.0.113.9, which is in no OpenAI range, so it is spoofed. It also asked for /wp-login.php, which a training crawler has no reason to want. The table lists spoofing addresses under each agent's details, ready for a firewall rule.
How people sent by an AI assistant are counted
Each line that is a page view by a person is checked against the assistant list: ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, DeepSeek, Meta AI, Mistral Le Chat, You.com, Phind, Grok and Poe. A line counts for an assistant when its Referer is on one of the assistant's hosts, subdomains included (www.perplexity.ai is Perplexity), or when the requested URL carries one of its utm_source values. OpenAI documents that ChatGPT adds utm_source=chatgpt.com to its links. For the others the hosts are simply where their chat runs, which is observed, not documented; the list says which is which.
203.0.113.20 - - [20/Sep/2026:09:12:40 +0000] "GET /pricing/ HTTP/1.1" 200 8120 "https://chatgpt.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) … Safari/604.1"
203.0.113.21 - - [20/Sep/2026:11:02:05 +0000] "GET /pricing/?utm_source=chatgpt.com HTTP/1.1" 200 8120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) … Chrome/140.0.0.0 Safari/537.36"
203.0.113.20 - - [20/Sep/2026:09:12:41 +0000] "GET /img/plans.webp HTTP/1.1" 200 40960 "https://chatgpt.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) … Safari/604.1"
That is two people ChatGPT sent to /pricing/: one known by the Referer, one by utm_source after the app dropped the Referer. The third line is the first visitor's browser loading an image, which is not a page view. A line is a page view when the user-agent names no directory agent and does not say bot, crawler or spider; the status is 2xx or 304; the method is GET; and the path is not a stylesheet, script, image, font, .json, .xml, .txt, favicon or web manifest. Clicks from Google and Bing search are counted apart, for comparison, and never as an assistant. The CSV has one row per day, assistant and landing page, with the header date,assistant,page,visits.
Behind a proxy or CDN
If Cloudflare, CloudFront, a load balancer or a reverse proxy sits in front of your server, the origin server's own log records the proxy's address for every request unless it is told to log the client's. Every GPTBot request would then look spoofed. The page warns when it sees the signs: verifiable agents that never verify, most traffic from private addresses such as 10.x or 172.16–31.x, or a busy log with only one to three addresses. Either read the CDN's or load balancer's own log instead (Cloudflare's ClientIP, CloudFront's c-ip and the load balancer's client:port are the visitor, see the formats), or have the server log the client address the proxy passes on:
- Nginx:
set_real_ip_fromwith your proxy's ranges andreal_ip_header X-Forwarded-For;(behind Cloudflare,CF-Connecting-IP). Therealipmodule then puts the client's address in$remote_addr, which is the first field of the combined format. - Apache:
mod_remoteipwithRemoteIPHeaderandRemoteIPTrustedProxy, and%ainstead of%hin theLogFormat. - Caddy: set
trusted_proxies. Its JSON log then fillsrequest.client_ip, which this page reads beforeremote_ip.
Where to find the log
On Debian and Ubuntu, Nginx writes /var/log/nginx/access.log and Apache writes /var/log/apache2/access.log. On RHEL-family systems Apache uses /var/log/httpd/access_log. Older days are rotated to access.log.1, access.log.2.gz and so on, and you can drop them all at once. Caddy writes JSON wherever the log directive's output points. A CDN or hosting export in CSV works if its header names the client IP, user-agent, path, status and time columns. IIS writes daily files such as u_ex260920.log under %SystemDrive%\; CloudFront, Cloudflare Logpush and the AWS load balancer deliver gzipped files to the bucket you configured, and those can be dropped as they are.
Formats read, and what each cannot show
The format is decided per file from its first 20 lines. Fields are read by name wherever the format names them, so a header with extra or reordered fields still works. Each format below was checked against the vendor's own field reference on 2026-09-26.
- Cloudflare Logpush,
http_requestsdataset, JSON lines (Cloudflare's field list). Read:ClientIP,ClientRequestURI(orClientRequestPath),ClientRequestUserAgent,ClientRequestReferer,ClientRequestMethod,EdgeResponseStatusandEdgeStartTimestampin any of the job's timestamp formats (unixnano,unix,rfc3339,rfc3339ms,rfc3339ns). Crawlers are verified byClientIP, the visitor's address;EdgeServerIPandOriginIPare Cloudflare's and your server's and are never used. Limit: Logpush sends only the fields you picked for the job; withoutClientRequestUserAgentno crawler can be named, and withoutClientRequestRefererassistant visits are counted fromutm_sourcetags only (the file report says so). Logpush's CSV output, when it has a header row, is read by the CSV reader. - AWS CloudFront standard logs (AWS's field reference). A
#Versionline, a#Fields:header, tab-separated values. Read:c-ip,cs-method,cs-uri-stemwithcs-uri-query,sc-status,cs(Referer)andcs(User-Agent)(both URL-encoded in the file,%20for a space, and decoded here), anddateplustime, which are UTC. Limit:c-ipis whoever connected to CloudFront, so a visitor behind their own proxy shows the proxy's address (AWS logs the original inx-forwarded-for, which is not used because the visitor can write anything there). - AWS Application Load Balancer access logs (AWS's field reference). Space-separated with quoted fields. Read:
time(ISO 8601),client:port(the port is dropped, IPv6 included),elb_status_code, the quotedrequestline anduser_agent. Limit: the load balancer logs no Referer at all, so for these files people sent by an assistant are counted only when the link carried autm_sourcetag (ChatGPT addsutm_source=chatgpt.com); an untagged click from Perplexity or Copilot is invisible. The page says this beside the results. Classic Load Balancer logs have a different layout and are not read. - IIS W3C extended logs (Microsoft's W3C logging page). A
#Fields:header, space-separated values, a space inside a value written as+(soMozilla/5.0+(Windows+NT+10.0...is read with spaces again). Read:c-ip,cs-method,cs-uri-stemwithcs-uri-query,sc-status,cs(User-Agent),cs(Referer),dateandtime(UTC). A new header written after a restart mid-file is followed. Limit:cs(Referer)is there only if the Referer field is switched on. Microsoft's logFile reference says it joined the defaultlogExtFileFlagsin IIS 8.0, but IIS 7.5 and earlier, hand-picked field sets and the example on Microsoft's HTTP Server API page leave it out. Without it the file report says so and the assistant count usesutm_sourceonly; turn it on under IIS Manager, Logging, Select Fields, Referer. An IIS server behind a load balancer or CDN logs that device's address inc-ip, as any origin does.
A file in none of these shapes is not guessed at. The page says what it saw instead, for example JSON lines with keys host, path, agent
or a #Fields header without c-ip or cs(User-Agent)
, so you can tell whether a field is missing or the format is simply another one.
What this page cannot do
- No live DNS. Google, Apple and Common Crawl also describe a reverse-then-forward DNS check. A web page cannot make DNS lookups, so only the IP-range check is done. For those operators the published list is the other method they give.
- Only the listed formats. Nginx or Apache
combinedandcommon, Caddy JSON, Cloudflare Logpush JSON, CloudFront standard logs, Application Load Balancer logs, IIS W3C extended logs, and CSV with a header row. A customlog_formatthat moves the address or the quoted fields, Classic Load Balancer logs, CloudFront real-time logs and other JSON layouts are not read. A file in another shape is reported with a description of what was in it, and nothing is guessed. - No logs larger than the tab can stream. The file is read in chunks by a background worker, so memory stays small and the page stays responsive. In our tests it reads roughly 25–30 MB of log per second on a desktop, so 1 GB takes well over half a minute, and longer on a phone. Per agent, the first 50,000 distinct paths are kept for the top-10 list.
- Snapshots age. The IP lists are copies with the dates shown. If an operator adds a range after that date, its newest crawlers will show as spoofed here until the copy is refreshed, except on the ChatGPT-User list, which reads “Not in OpenAI’s list of date” once the copy is more than a day old. The date of each copy is shown with the results, with its age for the daily list.
- Only agents that say who they are. A crawler that sends a browser user-agent is counted under "everything else". Query strings are dropped from paths, so
/a?page=2counts as/a. - Referrer-stripped assistant clicks go uncounted, so the assistant numbers are a floor. A click from an assistant's phone or desktop app, from some mobile browsers, or from a page with a strict referrer policy arrives with no Referer. It is still counted when the link carries the assistant's
utm_source(ChatGPT addsutm_source=chatgpt.com), and otherwise it lands among the unattributed visits. Answers shown inside Google Search (AI Overviews, AI Mode) or inside Bing arrive from the search engine's own address and are counted as search, not as an assistant. - Control tokens.
Google-ExtendedandApplebot-Extendednever fetch pages. Google's and Apple's regular crawlers do the fetching, so a request that names either token is not documented behaviour, whatever its address.
Seen an agent you want to keep out? Generate a robots.txt for it, or check the one you have. Remember that robots.txt only asks. A spoofer ignores it, and blocking its addresses is the only thing that stops it.