resulthack

All AI crawlers /

OAI-SearchBot

OpenAI · AI search · robots.txt: Yes

Crawls the web so that pages can be shown as results in ChatGPT's search features.

Block OAI-SearchBot

User-agent: OAI-SearchBot
Disallow: /

Blocking OAI-SearchBot keeps your pages out of ChatGPT search answers, though OpenAI says they can still appear as navigational links. It does not decide whether your content is used for training; that is GPTBot.

Allow OAI-SearchBot

User-agent: OAI-SearchBot
Allow: /

An allow group only changes anything if a broader rule would block OAI-SearchBot. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.

Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.

Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks OAI-SearchBot on a given path, and which line decides.

Facts

Operator
OpenAI
Purpose
AI search index
Obeys robots.txt
Yes. OpenAI documents OAI-SearchBot as a robots.txt token and says a robots.txt change can take about 24 hours to reach its search systems.
In your server logs
The user-agent string contains "OAI-SearchBot/1.4" (the version number may change); requests for robots.txt may carry an extra "robots.txt" marker.
How to verify it
Match the request's IP address against OpenAI's published OAI-SearchBot IP list. OpenAI recommends allowing requests from these ranges if you want to appear in ChatGPT search.

Published list: openai.com/searchbot.json — 39 IPv4 ranges when we fetched it, list dated 2 Jan 2026. Always match against the live list; operators update them. To count real and fake OAI-SearchBot requests in your own server log against this list, drop the log into the log reader.

What OpenAI says

Quoted word for word from the operator's documentation.

“OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features.”

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

“To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below.”

“For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust.”

“Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt;”

“Each setting is independent of the others”

Sources

Other crawlers from OpenAI

Same purpose, other operators

← All 39 AI crawlers