NewVitalSentinel, redesigned
// ByteDance crawler

Bytespider:what it is and how to block it

Bytespider is ByteDance's web crawler. We found no ByteDance documentation we could verify that says what it crawls for, AI training included.

robots.txt token
Bytespider
Operator
ByteDance
Purpose
Data collection
Follows robots.txt
Not stated
User agent
Not published
IP list
Not published
// user agent

Bytespider user agent

robots.txt token
Bytespider

We found no full user-agent string from ByteDance that we could verify. Look for the Bytespider token, which is also what robots.txt matches on.

Source: Cloudflare blog: Declare your AIndependence. Checked October 30, 2026.

// robots.txt

How to block Bytespider

We found no ByteDance statement we could verify on whether Bytespider follows robots.txt. A Disallow rule for the Bytespider token is still the standard way to ask.

A robots.txt rule is a request. If you need certainty about any crawler, the block has to be enforced at your CDN, WAF or origin by user agent and IP range.

Block it from the whole site
User-agent: Bytespider
Disallow: /
Allow it everywhere
User-agent: Bytespider
Allow: /

Add the group to the robots.txt at the root of each host. A crawler with a group of its own ignores your User-agent: * rules, so repeat any you still want it to follow.

Write rules for all 20 AI crawlers at once
// verify

How to verify a real Bytespider visit

We found no IP list or reverse DNS name from ByteDance that we could verify, so a request that uses the name cannot be confirmed either way.

// your site

See whether Bytespider visits your site

robots.txt testerCheck whether your rules let Bytespider reach any path, and which line decides it.Open the toolAI crawler blockerSee what your live robots.txt allows Bytespider and the other 19 AI crawlers on our list today.Open the tool
Server Log MonitoringLists Bytespider by name with its hits, its 403s and whether it follows your robots.txt. ByteDance publishes no machine-readable address list, so its hits land in Cannot be checked.
Start for freeHow it works
// faq

Questions

Does Bytespider obey robots.txt?

ByteDance does not say in any documentation we could verify. A robots.txt rule is still the standard way to ask. To be certain, block the user agent at your CDN or firewall.

Who runs Bytespider?

ByteDance. Beyond the Bytespider token, we found no ByteDance documentation we could verify on what it crawls for.

Does blocking Bytespider affect my Google rankings?

No. Bytespider is not Googlebot, so a rule for it changes nothing about how Google Search crawls, indexes or ranks your pages.

Other AI crawlers

// free to start

See every AI crawler that visits your site

Server Log Monitoring names every AI crawler in your server logs, with its hits and 403s, and flags hits that fake a crawler's name.

Start for freeSee pricing
1 domain freeNo credit cardAll 12 modules5.0on G2