// ai crawlers
AI crawlers and how to block them
Who runs each major AI crawler, what it is for, the user agent it sends, whether it obeys robots.txt and how to check a visit is real. Every fact links to its source, the vendor's own documentation wherever one exists.
- ClaudeBotAnthropicCrawls pages for Anthropic's model training.Follows robots.txt, per its vendor
- GPTBotOpenAICrawls pages that may be used to train OpenAI models.Vendor says robots.txt controls access
- OAI-SearchBotOpenAIBuilds the index behind ChatGPT search results.Vendor says robots.txt controls access
- BytespiderByteDanceByteDance's web crawler. ByteDance publishes no documentation we could verify on what it crawls for.robots.txt behavior not stated
- Meta-ExternalAgentMetaCrawls pages Meta may use to train AI models and improve its products.Vendor says robots.txt controls access
- CCBotCommon CrawlBuilds the Common Crawl public web archive.Follows robots.txt, per its vendor
- PerplexityBotPerplexityIndexes pages so Perplexity can link to and cite them.Vendor says robots.txt controls access
// free to start
See every AI crawler that visits your site
Server Log Monitoring names every AI crawler in your server logs, with its hits and 403s, and flags hits that fake a crawler's name.