ClaudeBot is Anthropic’s web crawler for model training. It collects public web content that may be used to train Claude, and it follows robots.txt. Blocking ClaudeBot keeps your future content out of Anthropic’s training data, but it doesn’t remove you from Claude’s search answers. That’s handled by two other agents, Claude-SearchBot and Claude-User, which you control separately.
If you only remember one thing from this guide, make it that split. Many sites block “Claude” with a single ClaudeBot rule and think they’re done, or worse, block all three agents and then wonder why Claude never cites them.
Below you’ll find what each Anthropic bot does, what ClaudeBot looks like in your logs, how to verify it’s real, and copy-ready robots.txt rules. The facts come from Anthropic’s help center article on its crawlers, checked October 23, 2026.
What is ClaudeBot?
ClaudeBot is the crawler Anthropic uses to gather web content “that could potentially contribute to” the training of its generative AI models. When a site restricts ClaudeBot, Anthropic says it “signals that the site’s future materials should be excluded from our AI model training datasets.”
Two details in that wording matter. First, it’s about training, not search. Second, it’s about future material. A new robots.txt rule doesn’t reach back into data that was already collected.
US searches for “claudebot” have grown roughly 4.5 times over the past year, mostly from site owners who spotted it in their logs and want to know what it is.
ClaudeBot vs Claude-SearchBot vs Claude-User
Anthropic runs three bots, and each one has its own robots.txt token.
| Bot | What it does | What blocking it does | Recommendation for AI visibility |
|---|---|---|---|
| ClaudeBot | Collects web content that may be used to train Anthropic’s models | Excludes your future content from training datasets | Your choice |
| Claude-SearchBot | Crawls the web to improve the quality of Claude’s search results | Stops your content being indexed for search, which “may reduce your site’s visibility” in Claude’s answers | Allow |
| Claude-User | Fetches a page when a Claude user asks a question that needs it | Stops Claude retrieving your page for a user’s request | Allow |
A rule for one bot doesn’t apply to the others. If your robots.txt only mentions ClaudeBot, Claude-SearchBot and Claude-User fall back to your User-agent: * group, if you have one.
What does ClaudeBot crawl?
ClaudeBot crawls publicly available pages, the same way other crawlers do: it requests robots.txt first, reads your rules, and fetches what’s allowed. Anthropic says its bots:
- honor robots.txt “do not crawl” directives
- support the non-standard
Crawl-delaydirective - don’t try to bypass CAPTCHAs or other anti-circumvention measures
- aim to be “thoughtful about how quickly” they crawl the same domain
Anthropic doesn’t publish crawl frequency or volumes. One public log sample from Freshly Cited recorded 62 ClaudeBot requests over about 14 hours on a single site in October 2026, many of them for robots.txt and the sitemap. That’s one small site, so treat it as an example of the pattern, not a benchmark.
Anthropic’s article doesn’t say ClaudeBot logs into anything or reads content behind a paywall or login. If a page needs an account to view, expect ClaudeBot to see only what a logged-out visitor sees.
What is the ClaudeBot user agent?
Anthropic’s help article names the robots.txt token, ClaudeBot, but doesn’t print the full user agent header. The string most commonly observed in server logs, including the Freshly Cited sample above, is:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Bot directories also report longer variants that start with a desktop or mobile Chrome string and end in ClaudeBot/1.0; +claudebot@anthropic.com. Because the prefix varies, match on the token ClaudeBot in your log filters and firewall rules, not on the whole string. The same goes for Claude-SearchBot and Claude-User.
How to block ClaudeBot
To block ClaudeBot from your whole site, add this to the robots.txt file at the root of your domain:
User-agent: ClaudeBot
Disallow: /Anthropic says to do this for every subdomain you want to opt out. A rule on example.com/robots.txt doesn’t cover blog.example.com.
Block training but stay visible in Claude search
This is the setup most brands want: no training, but full eligibility for Claude’s answers.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Slow ClaudeBot down instead of blocking it
If ClaudeBot is putting load on your server but you don’t want to block it, Anthropic supports Crawl-delay:
User-agent: ClaudeBot
Crawl-delay: 1Block only part of your site
User-agent: ClaudeBot
Disallow: /members/
Disallow: /drafts/How to allow ClaudeBot
If your robots.txt has no rule for ClaudeBot and no blanket User-agent: * disallow, ClaudeBot is already allowed. To make it explicit:
User-agent: ClaudeBot
Allow: /Then check the layer most people forget: your firewall. CDN bot protection can block or challenge AI crawlers even when robots.txt allows them. Cloudflare, for example, began blocking known AI crawlers by default on new domains in July 2025. If you want ClaudeBot in, confirm your bot settings allow it.
Should you block ClaudeBot?
It depends on what you sell and what you’re protecting.
- Block it if your content is your product: paid research, original datasets, journalism, course material. Training use gives you nothing back.
- Allow it if you’re a business that wants Claude to know your brand, your category and your point of view. Content in training data can shape how future models describe your space, though Anthropic makes no promise about how any page is used.
- Either way, allow Claude-SearchBot and Claude-User if you want to be cited. Those are the agents behind Claude’s search answers.
There’s no ranking effect either way in Google or Bing. ClaudeBot rules only affect Anthropic.
Why you shouldn’t block ClaudeBot by IP address
It sounds safer to block Anthropic’s IP ranges than to trust robots.txt. Anthropic warns against it: blocking its IP addresses “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” If ClaudeBot can’t read your robots.txt, it can’t see your opt-out.
Use robots.txt to opt out. Use the IP list to verify.
How to verify ClaudeBot in your logs
A user agent is just a header, and anyone can claim to be ClaudeBot. To check a request is genuinely from Anthropic:
- Search your access logs for
ClaudeBot,Claude-SearchBotorClaude-User. - Download Anthropic’s published IP list at claude.com/crawling/bots.json. On October 23, 2026 it held 14 IPv4 prefixes, with a creation date of October 20, 2026. It changes, so fetch it fresh.
- Check each source IP against those prefixes. The list covers all three Anthropic bots and doesn’t label which bot uses which range, so the user agent tells you which bot, and the IP tells you whether it’s really Anthropic.
- Treat a ClaudeBot request from an IP outside the list as an impostor.
If you think a genuine Anthropic bot is misbehaving, Anthropic asks you to email claudebot@anthropic.com from an address on the affected domain.
How Claude decides what to cite
Crawler access makes you eligible for Claude’s answers. It doesn’t decide whether you’re picked. Claude’s web search has been reported to use Brave Search to find results, then reads and quotes the pages that answer each part of the question most directly. 95 Projects’ Claude SEO guide covers that side in detail, from Brave rankings to third-party mentions.
For the other AI companies’ bots, see our AI crawler list and our breakdown of OpenAI’s crawlers.
ClaudeBot FAQ
What is ClaudeBot?
ClaudeBot is Anthropic’s web crawler for collecting content that may be used to train its Claude models. It follows robots.txt, so you can block it with a User-agent: ClaudeBot rule.
Does blocking ClaudeBot stop Claude from citing my site?
No. ClaudeBot is for training. Claude’s search answers rely on Claude-SearchBot and Claude-User. Allow those two if you want Claude to find and cite your pages.
Is ClaudeBot safe?
Genuine ClaudeBot traffic comes from Anthropic, follows robots.txt and doesn’t try to bypass CAPTCHAs. Requests that use the name but come from outside Anthropic’s published IP list aren’t from Anthropic and can be blocked.
Does ClaudeBot respect robots.txt?
Yes. Anthropic says its bots honor standard robots.txt directives and support Crawl-delay. Site owners can also use robots.txt to control the user-triggered Claude-User agent.
What is the ClaudeBot user agent string?
The commonly observed string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). Anthropic doesn’t publish the full string, so match on the ClaudeBot token rather than the whole header.
How do I check whether ClaudeBot can reach my site?
Run your page through the free checker. It reads your robots.txt rules for 16 AI crawlers, including ClaudeBot and Claude-SearchBot, and makes a live visit as ClaudeBot to catch firewall or CDN blocks.
Can AI engines read your site?
Check any page for free: crawler access for ChatGPT, Perplexity, Claude and Google AI Overviews, plus the fixes that matter most.
