GPTBot
OpenAI · Training · robots.txt: Yes
Crawls public web pages whose content may be used to train OpenAI's generative AI foundation models.
Block GPTBot
User-agent: GPTBot
Disallow: /Blocking GPTBot only opts your content out of training. It does not remove you from ChatGPT search, which is controlled separately by OAI-SearchBot. If you allow both, OpenAI may use one crawl for both purposes.
Allow GPTBot
User-agent: GPTBot
Allow: /An allow group only changes anything if a broader rule would block GPTBot. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.
Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.
Facts
- Operator
- OpenAI
- Purpose
- Model training
- Obeys robots.txt
- Yes. OpenAI documents GPTBot as a robots.txt token: disallowing it tells OpenAI the site's content should not be used for training. When it fetches robots.txt it may add a robots.txt marker to its user-agent string.
- In your server logs
- The user-agent string contains "GPTBot/1.4"; OpenAI notes the version number may change.
- How to verify it
- Match the request's IP address against OpenAI's published GPTBot IP list.
Published list: openai.com/gptbot.json — 18 IPv4 ranges when we fetched it, list dated 22 Sep 2026. Always match against the live list; operators update them.
What OpenAI says
Quoted word for word from the operator's documentation.
“GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models.”
“Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”
“Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.”
“If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling.”
“When fetching robots.txt files, we may add a robots.txt marker to the user-agent string to help site owners distinguish those requests from requests for other resources, especially when logs do not include paths.”
“Example user-agent string (the version number may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4;”
“Published IP addresses:”
Sources
- OpenAI: Overview of OpenAI Crawlers — fetched and checked 2026-09-22
- OpenAI: GPTBot IP ranges (gptbot.json) — fetched and checked 2026-09-22