CCBot:what it is and how to block it
Common Crawl is a nonprofit that keeps an open repository of web crawl data, and CCBot is the crawler that collects it.
- robots.txt token
CCBot- Operator
- Common Crawl
- Purpose
- Data collection
- Follows robots.txt
- Yes, per Common Crawl
- User agent
- Published
- IP list
- Published
CCBot user agent
CCBot
CCBot/2.0 (https://commoncrawl.org/faq/)
The robots.txt token is CCBot.
Sources: Common Crawl: CCBot, Common Crawl: FAQ. Checked October 30, 2026.
How to block CCBot
Common Crawl says CCBot checks robots.txt before it fetches a page, obeys Crawl-delay and honors nofollow on links.
The archive is a common source of training data for many model builders, so this one token affects a lot of downstream use. Existing archive snapshots stay published.
User-agent: CCBot Disallow: /
User-agent: CCBot Allow: /
Add the group to the robots.txt at the root of each host. A crawler with a group of its own ignores your User-agent: * rules, so repeat any you still want it to follow.
How to verify a real CCBot visit
Common Crawl publishes CCBot's IPv4 and IPv6 ranges as JSON. Reverse DNS on an IPv4 address returns a name ending in crawl.commoncrawl.org; confirm it with a forward lookup back to the same IP. IPv6 has no reverse DNS yet.
https://index.commoncrawl.org/ccbot.json
crawl.commoncrawl.org
See whether CCBot visits your site
Questions
Does blocking CCBot remove my pages from the archive?
No. It stops future crawls. Common Crawl does not say what happens to pages already in its archive.
Can I slow CCBot down instead of blocking it?
Yes. Common Crawl says CCBot obeys the Crawl-delay rule in robots.txt.
Does blocking CCBot affect my Google rankings?
No. CCBot is not Googlebot, so a rule for it changes nothing about how Google Search crawls, indexes or ranks your pages.
Other AI crawlers
- ClaudeBotAnthropicCrawls pages for Anthropic's model training.Follows robots.txt, per its vendor
- GPTBotOpenAICrawls pages that may be used to train OpenAI models.Vendor says robots.txt controls access
- OAI-SearchBotOpenAIBuilds the index behind ChatGPT search results.Vendor says robots.txt controls access
- BytespiderByteDanceByteDance's web crawler. ByteDance publishes no documentation we could verify on what it crawls for.robots.txt behavior not stated
- Meta-ExternalAgentMetaCrawls pages Meta may use to train AI models and improve its products.Vendor says robots.txt controls access
- PerplexityBotPerplexityIndexes pages so Perplexity can link to and cite them.Vendor says robots.txt controls access
See every AI crawler that visits your site
Server Log Monitoring names every AI crawler in your server logs, with its hits and 403s, and flags hits that fake a crawler's name.