OpenAI sends four different bots to your site, and each one answers to a different robots.txt rule. OAI-SearchBot indexes pages so they can appear in ChatGPT search. GPTBot collects content that may be used to train OpenAI’s models. ChatGPT-User fetches a page when a person in ChatGPT asks about it. OAI-AdsBot checks landing pages submitted as ChatGPT ads. If you want to be cited in ChatGPT but keep your content out of training, allow OAI-SearchBot and ChatGPT-User, and make your own call on GPTBot.
This guide explains what each crawler does, what its user agent looks like in your logs, and the robots.txt rules that match the most common goals. Everything below comes from OpenAI’s crawler documentation and its publisher FAQ, checked October 23, 2026.
OpenAI crawlers at a glance
| User agent | What it does | Respects robots.txt? | Affects ChatGPT search visibility? | Used for training? |
|---|---|---|---|---|
| OAI-SearchBot | Indexes pages for ChatGPT’s search features | Yes | Yes. Block it and you won’t appear in search answers | No |
| GPTBot | Crawls content that may be used to train OpenAI’s foundation models | Yes | No | Yes, unless you disallow it |
| ChatGPT-User | Visits a page when a ChatGPT user or custom GPT asks about it | May not apply, because a person triggered it | Not used to decide search inclusion | Not documented as a training crawler |
| OAI-AdsBot | Checks landing pages submitted as ads on ChatGPT | Only visits submitted ad pages | No | No, per OpenAI |
The key point is that the settings are independent. OpenAI says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training.
What is OAI-SearchBot?
OAI-SearchBot is the crawler that decides whether your pages can show up in ChatGPT search. OpenAI describes it as the bot “used to surface websites in search results in ChatGPT’s search features.” Sites that block it won’t be shown in ChatGPT search answers, though OpenAI notes they can still appear as navigational links.
This is the one to protect if AI visibility matters to you. OpenAI’s help center says that to make a site eligible for ChatGPT search you should allow OAI-SearchBot and make sure your host or CDN lets in traffic from OpenAI’s published search bot IP addresses.
Its user agent currently looks like this (OpenAI says the version number may change):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotWhen it fetches robots.txt, OpenAI may add a robots.txt marker to the string so you can tell those requests apart in logs that don’t record paths. Changes to your robots.txt take about 24 hours to reach ChatGPT search, according to OpenAI.
What is GPTBot?
GPTBot is OpenAI’s training crawler. OpenAI says it crawls “content that may be used in training our generative AI foundation models,” and that disallowing it tells OpenAI your content shouldn’t be used that way.
Blocking GPTBot does not remove you from ChatGPT search. It’s a common mix-up. Plenty of sites added User-agent: GPTBot and Disallow: / in 2023, assumed they were “blocking ChatGPT,” and never revisited it. That rule only covers training. Search is controlled by OAI-SearchBot.
GPTBot’s user agent:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotOne detail worth knowing: if you allow both GPTBot and OAI-SearchBot, OpenAI says it may use a single crawl for both purposes to avoid crawling your site twice. So seeing only one of them in your logs doesn’t mean the other is blocked.
What is ChatGPT-User?
ChatGPT-User is the agent ChatGPT uses when a person asks a question and ChatGPT, or a custom GPT, visits a page to answer it. It also covers GPT Actions that call external applications. It isn’t an automatic crawler. It only fetches pages in response to a user.
Because a person triggers these visits, OpenAI says “robots.txt rules may not apply” to ChatGPT-User. OpenAI also says ChatGPT-User is not used to decide whether content may appear in search, and points site owners to OAI-SearchBot for search opt-outs.
This is why “chatgpt-user” has become such a common search. When ChatGPT answers a question by reading your page in real time, that visit shows up in your logs as ChatGPT-User, not as a classic crawler. It’s one of the clearest signs that people are asking ChatGPT about topics you cover.
ChatGPT-User’s user agent:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/botWhat is OAI-AdsBot?
OAI-AdsBot is the newest of the four. OpenAI uses it to check the safety of pages submitted as ads on ChatGPT. It only visits landing pages that advertisers submit, and OpenAI says the data it collects is not used to train foundation models. OpenAI may also use the landing page content to decide when an ad is relevant to show.
If you don’t advertise on ChatGPT, you shouldn’t see it. If you do, don’t block it, or your ads may fail review.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbotWhich OpenAI crawlers should you allow?
For most businesses that want to be found in AI search, the answer is simple:
- Allow OAI-SearchBot. It’s the only way to appear in ChatGPT search answers.
- Allow ChatGPT-User. Blocking it stops ChatGPT from reading your page when a user asks about it, even if that user wants to buy from you.
- Decide on GPTBot. Allowing it may help future models know your brand and category. Blocking it keeps your content out of training. Neither choice affects ChatGPT search inclusion.
- Leave OAI-AdsBot alone unless you have a reason to block it.
robots.txt examples for OpenAI crawlers
Each rule below goes in the robots.txt file at the root of your domain. Repeat it for every subdomain you want covered.
Allow ChatGPT search, block training
This is the most common setup for brands that want AI citations without contributing to training data.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /Allow everything from OpenAI
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
Allow: /Grouping several user agents above one set of rules is part of the robots.txt standard (RFC 9309), so one group covers all three.
Keep a private section out of ChatGPT
User-agent: OAI-SearchBot
Disallow: /members/
User-agent: GPTBot
Disallow: /A disallow alone may not fully hide a URL. OpenAI’s publisher FAQ says that if it learns about a disallowed page from a third-party search provider or other pages, it may still show the link and page title. To keep a page out entirely, use a noindex meta tag, and let the crawler fetch the page so it can see that tag.
Why ChatGPT might still not see your site
robots.txt is only half of access. The other half is your firewall, CDN or bot protection, and that’s where many access problems hide.
- CDN bot rules. Bot protection settings can block or challenge AI crawlers even when robots.txt allows them. Cloudflare, for example, began blocking known AI crawlers by default on new domains in July 2025. Check your dashboard rather than assuming.
- IP allowlists. OpenAI publishes IP ranges for each bot: searchbot.json, gptbot.json, chatgpt-user.json and adsbot.json. If your firewall uses an allowlist, add the ones you want.
- JavaScript-only content. If your main text only appears after scripts run, a crawler that reads raw HTML may find an empty page.
- noindex or nosnippet tags left over from staging or a plugin setting.
The free AEO and GEO checker reads your robots.txt rules for 16 AI crawlers, including OpenAI’s, and makes a live visit as GPTBot to catch firewall blocks that robots.txt won’t show you.
How to verify OpenAI bots in your logs
Anyone can put “GPTBot” in a user agent, so the string alone proves nothing. To confirm a visit is really from OpenAI:
- Find the requests by searching your access logs for
OAI-SearchBot,GPTBotorChatGPT-User. - Download the matching IP list from OpenAI (linked above). Fetch it fresh each time, because the ranges change.
- Check each request’s source IP against that list.
- Treat any request that claims to be an OpenAI bot but comes from outside the list as an impostor. You can block it like any other unwanted bot.
To measure what that access is worth, look at referrals. OpenAI says ChatGPT adds utm_source=chatgpt.com to links it sends from search results, so you can track that traffic in Google Analytics or any other analytics tool.
Beyond crawler access
Letting OpenAI’s bots in makes you eligible to be cited. It doesn’t make you the answer. ChatGPT search rewrites a question into several searches, uses third-party search providers alongside its own index, and picks the passages that answer each part most directly. Pages that open with a clear answer, use question-style headings and back claims with sources tend to get lifted more often. If you want a team to work on the content and mentions side, 95 Projects’ guide to generative engine optimization is a good overview of what that work involves. For the full picture of other AI bots, see our AI crawler list and our ClaudeBot guide.
OpenAI Crawlers FAQ
What is ChatGPT-User?
ChatGPT-User is the user agent ChatGPT uses when a person asks a question and ChatGPT or a custom GPT visits a web page to answer it. It isn’t an automatic crawler. Because a user triggers the request, OpenAI says robots.txt rules may not apply to it.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI’s models. OAI-SearchBot indexes pages for ChatGPT search. They’re controlled separately, so you can block GPTBot and still appear in ChatGPT search by allowing OAI-SearchBot.
Does blocking GPTBot remove my site from ChatGPT?
No. Blocking GPTBot only signals that your content shouldn’t be used for training. To appear in ChatGPT search answers, OAI-SearchBot must be allowed.
How long does a robots.txt change take to reach ChatGPT search?
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
Does ChatGPT use Bing?
ChatGPT search uses OpenAI’s own crawler plus third-party search providers. OpenAI’s help center links Microsoft’s privacy statement among those providers, but it doesn’t publish a full list. Ranking well in Bing can help, but allowing OAI-SearchBot is still required.
How do I check if OpenAI’s crawlers can access my site?
Run your page through the free checker. It reads your robots.txt rules for 16 AI crawlers, including OpenAI’s, then makes a live visit as GPTBot to catch firewall or CDN blocks.
Can AI engines read your site?
Check any page for free: crawler access for ChatGPT, Perplexity, Claude and Google AI Overviews, plus the fixes that matter most.
