The robots.txt example most sites need in 2026 allows AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, decides separately about training crawlers such as GPTBot and ClaudeBot, and leaves Googlebot alone. Below are nine copy-ready rule sets, grouped by goal, with a line on what each one does and what it doesn’t.
Every crawler name here comes from the operator’s own documentation. For the full list of AI user agents, see our AI crawler list. Last checked: October 26, 2026.
How robots.txt Rules Work (the 60-Second Version)
A robots.txt file sits at the root of your domain (example.com/robots.txt) and contains groups of rules. Each group starts with one or more User-agent lines and is followed by Allow and Disallow lines. The format is defined in RFC 9309.
Three rules explain most surprises:
- A crawler follows only the most specific group that names it. If GPTBot has its own group, it ignores everything under
User-agent: *. - The longest matching path wins.
Allow: /blog/beatsDisallow: /for/blog/post. If an Allow and a Disallow match equally, RFC 9309 says Allow should be used. - robots.txt is a request, not a lock. Well-behaved crawlers follow it. Your firewall and CDN decide what actually gets through.
Google also supports * as a wildcard and $ to mark the end of a URL, and it reads files up to 500 KiB (source: Google Search Central).
Robots.txt Example 1: Allow Everything
This is the default. An empty file, or no file at all, has the same effect.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlUse it if you want every search engine and AI crawler to read your whole site. Add your real sitemap URL.
Robots.txt Example 2: Allow AI Search, Block AI Training
This is the most common choice for brands that want to be cited by AI assistants without contributing to model training.
# AI search and user-triggered fetchers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# AI training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /
# Everyone else
User-agent: *
Allow: /What it does: OpenAI says sites that allow OAI-SearchBot can appear in ChatGPT search answers, and that GPTBot is the crawler for training data (source: OpenAI). Anthropic and Perplexity document the same split (sources: Anthropic, Perplexity).
What it doesn’t do: Google-Extended controls whether your content is used for Gemini training and grounding, but it doesn’t affect Google Search, including AI Overviews and AI Mode, which use Googlebot (source: Google). OpenAI also notes that robots.txt rules may not apply to ChatGPT-User, because those fetches are initiated by a person.
Robots.txt Example 3: Block All AI Crawlers but Keep Google Search
Use this if you don’t want AI assistants to crawl you at all, but you still need normal search traffic.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /The tradeoff is real: blocking the search crawlers removes you from ChatGPT search, Claude and Perplexity answers. Google’s AI Overviews and AI Mode still draw on Googlebot, so this file doesn’t remove you from those. Opting out of those features means using Google’s snippet controls, such as nosnippet, which also affect your regular search listings.
Robots.txt Example 4: Block Only Google’s AI Training (Google-Extended)
User-agent: Google-Extended
Disallow: /Google-Extended is a control token, not a separate crawler. Googlebot still crawls your site for Search. This line tells Google not to use that content for Gemini models and grounding.
Robots.txt Example 5: Block AI Crawlers From One Folder
Useful for paywalled guides, member areas or a research library you want to keep out of AI tools while leaving the rest of the site open.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /premium/
Disallow: /research/
User-agent: *
Allow: /Remember that robots.txt is public. Anyone can read it and see which folders you consider sensitive, so don’t use it to hide private URLs. Protect those with authentication.
Robots.txt Example 6: Let AI Crawlers See Only Your Marketing Pages
This flips example 5: block everything, then allow the sections you want AI assistants to quote.
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
Allow: /blog/
Allow: /pricing$
Allow: /features/
User-agent: *
Allow: /Because the longest matching rule wins, /blog/ and /features/ are allowed even though / is disallowed. The $ limits the pricing rule to the exact /pricing URL. Swap the user agents for search bots if your goal is citations rather than training.
Robots.txt Example 7: Disallow Rules for Common Cases
These patterns work inside any group above.
# Block a single page
Disallow: /thank-you
# Block a folder and everything in it
Disallow: /admin/
# Block every URL with a query parameter
Disallow: /*?
# Block one parameter, such as sort order
Disallow: /*?sort=
# Block all PDFs
Disallow: /*.pdf$
# Block internal search results
Disallow: /search/Paths are case-sensitive, so /Admin/ and /admin/ are different rules. Wildcards follow Google’s documented syntax. Smaller crawlers may treat them differently, so test.
Robots.txt Example 8: Block Everything (Staging Sites)
User-agent: *
Disallow: /Use this only on staging or development sites. It’s also the most common robots.txt disaster: copied to production during a launch, it tells every crawler to leave. If a live page is already indexed, Disallow stops crawling but doesn’t guarantee removal from search results. Use password protection on staging instead whenever you can.
Robots.txt Example 9: WordPress Default With AI Rules
WordPress serves a virtual robots.txt that blocks /wp-admin/ but allows admin-ajax.php. If you add a physical file, keep those lines.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlNote the gotcha: GPTBot, ClaudeBot and CCBot follow only their own group here, so they never see the /wp-admin/ rule. That’s fine in this file because they’re fully blocked. If you allow a bot in its own group, copy any Disallow lines it should still respect into that group too.
Common robots.txt Mistakes With AI Crawlers
Assuming User-agent: * covers named bots. Once a bot has its own group, the wildcard group no longer applies to it.
Blocking GPTBot and expecting to vanish from ChatGPT. GPTBot is for training. ChatGPT search uses OAI-SearchBot.
Blocking Google-Extended to escape AI Overviews. It doesn’t. AI Overviews and AI Mode are part of Google Search.
Allowing a bot in robots.txt while the firewall blocks it. CDN bot protection or a security plugin can return a 403 to crawlers your file allows. Check both.
Expecting instant changes. Crawlers cache robots.txt. Google generally caches it for up to 24 hours, and other operators note that changes can take time to take effect.
How to Test Your robots.txt
- Open
yourdomain.com/robots.txtin a browser and confirm the file you edited is the one being served, especially behind a CDN or a WordPress SEO plugin that generates its own. - Check the robots.txt report in Google Search Console for fetch errors.
- Search your server logs for the bot names above and confirm allowed bots get a 200 response, not a 403.
- Run an outside check. The free CraftOfGrowth checker tests your page as GPTBot, ClaudeBot and PerplexityBot and reports robots.txt access for each crawler.
For bot-by-bot details, see our guides to OpenAI’s crawlers and ClaudeBot. If you’re choosing a tool to track whether AI assistants actually mention you once they can read your site, see our roundup of AI visibility tools.
Frequently Asked Questions
What is a good robots.txt example for AI crawlers?
For most businesses, allow AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) so assistants can cite you, and decide separately whether to block training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended). Example 2 above does exactly that.
How do I block AI crawlers with robots.txt?
Add a group that lists each AI user agent on its own User-agent line, followed by Disallow: /. Name each bot explicitly, because many AI crawlers have their own tokens and only follow the group that names them.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and user-triggered visits come from ChatGPT-User. To leave ChatGPT search answers, you’d need to block OAI-SearchBot.
Can I block AI Overviews with robots.txt?
Not without affecting Google Search. AI Overviews and AI Mode use Googlebot, and Google-Extended doesn’t control them. Google’s snippet controls, such as nosnippet, limit how your content appears, but they also apply to regular results.
What does Disallow: / mean in robots.txt?
It tells the crawlers named in that group not to crawl any URL on the site. Under User-agent: * it applies to every crawler that doesn’t have its own group.
Do all AI crawlers respect robots.txt?
The major operators say their training and search crawlers do. Some user-triggered fetchers, such as ChatGPT-User and Perplexity-User, are treated differently because a person requested the page. Use your firewall if you need enforcement.
How long do robots.txt changes take to apply?
It depends on the crawler. Google generally caches robots.txt for up to 24 hours. Other operators also cache it, so allow a day or more before checking your logs.
Can AI engines read your site?
Check any page for free: crawler access for ChatGPT, Perplexity, Claude and Google AI Overviews, plus the fixes that matter most.
