Last updated: October 23, 2026. We check every operator’s documentation monthly and log changes in the changelog below.
AI companies don’t send one crawler to your site. Most send three kinds: a training crawler that collects content for future models, a search crawler that indexes pages so the AI can cite them, and a user-triggered fetcher that reads a page live when someone asks about it. Each has its own robots.txt token, so blocking one doesn’t block the others. If you want to be cited in AI answers, allow the search crawlers and user fetchers, and decide separately about training.
This page lists every AI user agent with public documentation from its operator, plus the widely seen ones that have none. For each, you’ll find the robots.txt token, what it does, whether it follows robots.txt and where the official documentation lives.
The three types of AI crawler
| Type | What it does | Block it and… | Examples |
|---|---|---|---|
| Training crawler | Collects public content that may be used to train AI models | Your future content is excluded from that company’s training data. Search visibility isn’t affected | GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot |
| Search crawler | Indexes pages so an AI assistant can find and cite them | You stop appearing in that assistant’s search answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User-triggered fetcher | Visits a page when a user asks a question that needs it | The assistant can’t read your page for that user. Some of these don’t follow robots.txt | ChatGPT-User, Claude-User, Perplexity-User |
There’s also a fourth kind that isn’t a crawler at all: control tokens like Google-Extended and Applebot-Extended. They never visit your site. They’re names you put in robots.txt to tell an existing crawler how its data may be used.
Full AI crawler list
OpenAI
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
GPTBot | Training | Crawls content that may be used to train OpenAI’s foundation models | Yes |
OAI-SearchBot | Search | Indexes pages for ChatGPT search. Blocking it removes you from search answers | Yes |
ChatGPT-User | User-triggered | Visits pages when a ChatGPT user or custom GPT asks about them | OpenAI says robots.txt “may not apply” |
OAI-AdsBot | Ads | Checks landing pages submitted as ChatGPT ads. Not used for training | Only visits submitted ad pages |
Source: OpenAI crawler documentation. OpenAI publishes a separate IP list for each bot. Full breakdown: ChatGPT-User vs OAI-SearchBot vs GPTBot.
Anthropic (Claude)
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
ClaudeBot | Training | Collects content that may be used to train Claude models | Yes, and supports Crawl-delay |
Claude-SearchBot | Search | Indexes pages to improve Claude’s search results | Yes |
Claude-User | User-triggered | Fetches pages when a Claude user asks a question | Yes, per Anthropic |
Source: Anthropic help center. One shared IP list for all three bots at claude.com/crawling/bots.json. Full breakdown: ClaudeBot guide.
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
Googlebot | Search | Crawls for Google Search, including AI Overviews and AI Mode | Yes |
Google-Extended | Control token | Controls whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Doesn’t affect Google Search | Not a crawler |
Google-CloudVertexBot | Site-owner requested | Crawls when a site owner builds a Vertex AI agent. No effect on Search | Yes |
GoogleOther | General | Generic crawler used by Google product teams, for example for research crawls | Yes |
Source: Google’s common crawlers. There’s no separate AI Overviews crawler. Blocking Google-Extended doesn’t remove you from AI Overviews, and blocking Googlebot removes you from Google Search entirely.
Perplexity
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
PerplexityBot | Search | Surfaces and links websites in Perplexity results. Perplexity says it isn’t used to train foundation models | Yes |
Perplexity-User | User-triggered | Visits pages when a Perplexity user asks a question | Perplexity says it “generally ignores robots.txt rules” |
Source: Perplexity crawler documentation. Perplexity says robots.txt changes can take up to 24 hours to apply.
Microsoft (Bing and Copilot)
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
Bingbot | Search | Builds the Bing index, which Copilot uses for grounding | Yes |
Microsoft doesn’t run a separate AI crawler. It controls AI use with meta tags instead: per Bing’s robots meta tag documentation, noarchive keeps a page out of Copilot answers and Microsoft model training, and nocache limits Copilot to the URL, title and snippet. Blocking Bingbot also removes you from Bing.
Apple
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
Applebot | Search | Powers search in Siri, Spotlight and Safari. Its data may also be used to train Apple’s foundation models | Yes. Doesn’t follow Crawl-delay |
Applebot-Extended | Control token | Opts your content out of training Apple’s foundation models. Doesn’t crawl and doesn’t affect search | Not a crawler |
Source: About Applebot. If your robots.txt doesn’t mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules.
Meta
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
meta-externalagent | Training and indexing | Crawls for uses such as training AI models or indexing content | Yes |
meta-webindexer | Search | Improves Meta AI search results. Allowing it helps Meta AI cite and link you | Yes |
meta-externalfetcher | User-triggered | Fetches links at a user’s request, including for agentic tasks | May bypass robots.txt |
meta-externalads | Ads | Crawls to improve advertising and business products | Yes |
facebookexternalhit | Link previews | Builds previews when a link is shared on Facebook, Instagram or Messenger | May bypass it for security checks |
Source: Meta web crawlers. Meta says robots.txt changes can take up to 24 hours.
Amazon
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
Amazonbot | Training and products | Improves Amazon products and services. May be used to train Amazon AI models | Yes. Doesn’t follow Crawl-delay |
Amzn-SearchBot | Search | Powers search experiences such as Alexa. Not used for model training | Yes |
Amzn-User | User-triggered | Fetches live information for user requests, such as Alexa questions | “May not follow all robots.txt directives” |
Source: About Amazonbot.
Mistral
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
MistralAI-Training | Training | Builds datasets for training Mistral models | Yes |
MistralAI-Index | Search | Indexes pages for Mistral search. Not used for training | Yes |
MistralAI-User | User-triggered | Visits pages when a user asks Mistral’s assistant a question | Governs user requests |
Source: Mistral crawlers.
DuckDuckGo
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
DuckAssistBot | Search (real time) | Crawls pages in real time for DuckDuckGo’s AI-assisted answers. Not used to train AI models | Yes. Changes take effect after 72 hours |
Source: DuckDuckGo help pages. Blocking it doesn’t affect your regular DuckDuckGo rankings.
Common Crawl
| robots.txt token | Type | What it does | Follows robots.txt |
|---|---|---|---|
CCBot | Training (indirect) | Builds Common Crawl’s open web dataset, which many AI models have been trained on | Yes |
Source: Common Crawl CCBot. Common Crawl says it’s aware of crawlers falsely identifying as CCBot, and publishes IP ranges with reverse DNS so you can verify.
Crawlers without official documentation
| Token | Operator | What we know |
|---|---|---|
Bytespider | ByteDance | Widely reported as a training crawler, but ByteDance publishes no crawler page, purpose, robots.txt policy or IP list |
cohere-ai | Cohere | Cohere says it doesn’t currently crawl to train models, and that a future bot would be named Coherebot. cohere-ai is a token many sites still list |
GrokBot, xAI-SearchBot | xAI | xAI publishes no crawler documentation. These names come from third-party directories |
For undocumented bots, robots.txt is a request, not a guarantee. If you need them gone, block them at your CDN or firewall.
How to block AI crawlers with robots.txt
These examples go in the robots.txt file at the root of each domain and subdomain.
Block training, stay visible in AI search
This is the setup most businesses want.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /Note that Applebot and Amazonbot may also use crawled data for training, but blocking them would remove you from Apple and Amazon search features too. Applebot-Extended is the safer lever for Apple.
Block all AI crawlers
Add every training and search token above to a single Disallow: / group. Keep Googlebot and Bingbot out of it unless you also want to leave Google and Bing search. User-triggered fetchers may not follow the rule anyway.
Allow everything
If your robots.txt has no rules for these tokens and no blanket User-agent: * disallow, they’re already allowed. You don’t need to list them.
robots.txt isn’t the whole story
A crawler can be allowed in robots.txt and still never reach your page.
- CDN and firewall rules. Bot protection can block or challenge AI crawlers on its own. Cloudflare, for example, began blocking known AI crawlers by default on new domains in July 2025.
- IP blocking backfires. Anthropic and Cohere both warn that blocking crawler IPs can stop the bot reading your robots.txt, so it can’t see your opt-out. Opt out in robots.txt and use IP lists to verify.
- JavaScript. Vercel’s December 2024 analysis of AI crawler traffic found that the major AI crawlers it studied fetched JavaScript files but didn’t execute them. If your main content only appears after scripts run, AI crawlers may see an empty page. Googlebot and Applebot do render pages.
The free AEO and GEO checker reads your robots.txt for 16 of the AI crawlers on this list, then makes live visits as GPTBot, ClaudeBot and PerplexityBot to catch firewall blocks robots.txt won’t show.
How to verify an AI crawler is real
Anyone can fake a user agent. To confirm a visit:
- Find requests in your access logs by token, for example
GPTBotorPerplexityBot. - Check the source IP against the operator’s published list. OpenAI, Anthropic, Perplexity, Amazon, Mistral, DuckDuckGo and Common Crawl all publish IP ranges. Google, Apple and Common Crawl also support reverse DNS checks, and Common Crawl says it’s aware of crawlers falsely identifying as CCBot.
- Treat a request that uses a known name from an unlisted IP as an impostor.
Beyond crawler access
Being crawlable is the entry ticket to AI search, not the win. Once the right bots can reach you, citations come from clear answers, solid structure and third-party mentions. If you want help with that side, 95 Projects explains how generative engine optimization works, and our AI visibility tools comparison covers how to track the results.
Changelog
- October 23, 2026: First published with 33 user agents and control tokens from 14 operators.
AI Crawler List FAQ
What is an AI crawler?
An AI crawler is a bot run by an AI company that visits websites to collect content for model training, index pages for AI search, or fetch a page live when a user asks about it. Each type uses its own user agent and robots.txt token.
Which AI crawlers should I allow?
Allow the search crawlers and user-triggered fetchers of the assistants you want to be cited in, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. Allowing or blocking training crawlers like GPTBot and ClaudeBot is a separate choice that doesn’t affect AI search visibility.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended doesn’t affect inclusion in Google Search. AI Overviews and AI Mode use Googlebot’s crawl, so the only way to leave them through robots.txt is to block Googlebot, which removes you from Google Search too.
Do AI crawlers follow robots.txt?
The documented training and search crawlers say they do. Several user-triggered fetchers, including ChatGPT-User, Perplexity-User, Meta-ExternalFetcher and Amzn-User, say they may not, because a person requested the page. Undocumented bots like Bytespider make no public commitment.
Do AI crawlers render JavaScript?
Most don’t. A December 2024 Vercel analysis found that the major AI crawlers it studied didn’t execute JavaScript, so content that only loads client-side may be invisible to them. Server-side rendering or static HTML is the safest option.
How often is this list updated?
We review every operator’s documentation monthly and update the tables when a crawler is added, renamed or changes its rules.
Can AI engines read your site?
Check any page for free: crawler access for ChatGPT, Perplexity, Claude and Google AI Overviews, plus the fixes that matter most.
