NewVitalSentinel, redesigned
// Anthropic crawler

ClaudeBot:what it is and how to block it

Anthropic says ClaudeBot collects public web content that may help train its AI models. Blocking it keeps your future pages out of that training data.

robots.txt token
ClaudeBot
Operator
Anthropic
Purpose
AI training
Follows robots.txt
Yes, per Anthropic
User agent
Not published
IP list
Published
// user agent

ClaudeBot user agent

robots.txt token
ClaudeBot

Anthropic names the ClaudeBot token but does not publish a full user-agent string. Look for the token, which is also what robots.txt matches on.

Source: Anthropic Help Center: Does Anthropic crawl data from the web?. Checked October 30, 2026.

// robots.txt

How to block ClaudeBot

Anthropic says its crawlers honor robots.txt, support the non-standard Crawl-delay rule and do not try to get past CAPTCHAs. Add the rule on every subdomain you want covered.

Blocking it stops future training crawls and leaves Claude-SearchBot and Claude-User unaffected.

Of Anthropic's two other tokens, Claude-SearchBot indexes pages for Claude's search results and Claude-User fetches a page when a person asks Claude about it.

Block it from the whole site
User-agent: ClaudeBot
Disallow: /
Allow it everywhere
User-agent: ClaudeBot
Allow: /

Add the group to the robots.txt at the root of each host. A crawler with a group of its own ignores your User-agent: * rules, so repeat any you still want it to follow.

Write rules for all 20 AI crawlers at once
// verify

How to verify a real ClaudeBot visit

Anthropic publishes its crawlers' IP addresses as one JSON list, not split by bot. A request from an address on it comes from Anthropic. Anthropic warns that blocking those addresses can stop it reading your robots.txt, so opt out with robots.txt.

Published IP list
https://claude.com/crawling/bots.json
// your site

See whether ClaudeBot visits your site

robots.txt testerCheck whether your rules let ClaudeBot reach any path, and which line decides it.Open the toolAI crawler blockerSee what your live robots.txt allows ClaudeBot and the other 19 AI crawlers on our list today.Open the tool
Server Log MonitoringLists ClaudeBot by name with its hits, its 403s and whether it follows your robots.txt. Hits are checked against Anthropic's published addresses, and fakes land in Fake bots.
Start for freeHow it works
// faq

Questions

Does blocking ClaudeBot stop Claude from reading my site?

Not entirely. ClaudeBot is the training crawler. Claude-SearchBot and Claude-User are separate tokens, so they need rules of their own if you want them out too.

Can I slow ClaudeBot down instead of blocking it?

Yes. Anthropic supports Crawl-delay, so a line such as Crawl-delay: 1 in the ClaudeBot group asks it to crawl more slowly.

Does blocking ClaudeBot affect my Google rankings?

No. ClaudeBot is not Googlebot, so a rule for it changes nothing about how Google Search crawls, indexes or ranks your pages.

Other AI crawlers

// free to start

See every AI crawler that visits your site

Server Log Monitoring names every AI crawler in your server logs, with its hits and 403s, and flags hits that fake a crawler's name.

Start for freeSee pricing
1 domain freeNo credit cardAll 12 modules5.0on G2