NewVitalSentinel, redesigned
ResearchAI CrawlersServer Logs

AI Crawlers on Your Server: More Than One in Four Hits Was Fake, Half Can't Be Checked

58,422 AI bot requests on five of my sites over 30 days: 22% verified, 27% fake, 51% unverifiable. What GPTBot, ClaudeBot and PerplexityBot got back, and what robots.txt did.

Martin Stepanek
Martin Stepanek
November 13, 2026
7 min read
Share

I pulled 30 days of server logs from five sites I run. Then I filtered for every request that put an AI bot's name in the user agent.

That's 58,422 requests. They came from OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and a long tail of names you've never seen.

They fall into three piles: addresses the vendors publish, addresses that provably aren't theirs, and addresses nobody can check. The real pile was the smallest.

Five sites, 30 days, three piles

The sites and the window, October 5 to November 3, 2026, are the same as in Fake Googlebot: What 100 Sites Return to It. I read the logs of all five in Server Log Monitoring.

A hit counts as verified when the address matches the vendor's published IP ranges or reverse DNS. It counts as fake when the address provably belongs to someone else. Everything else is unverified, because the check couldn't settle it.

I counted every AI bot: training crawlers, search bots, bots that fetch for a user, and agents.

PileHitsShare
Verified13,07522%
Fake15,58827%
Unverified29,75951%
All AI bot hits58,422100%

GPTBot, OAI-SearchBot and ChatGPT-User

Three of OpenAI's documented bots matter here. GPTBot is for training. OAI-SearchBot is for ChatGPT search. ChatGPT-User fetches a page when a person asks ChatGPT to.

OpenAI's bot documentation (opens in a new tab) says opting out of OAI-SearchBot keeps you out of ChatGPT search answers, and a robots.txt change takes about 24 hours to reach OpenAI's systems. For ChatGPT-User it says this:

Because these actions are initiated by a user, robots.txt rules may not apply.

Across the five sites, verified OAI-SearchBot hits far outnumbered the other two. Search outweighs training.

Verified and fake hits for GPTBot, OAI-SearchBot and ChatGPT-User across five sites.OAI-SearchBot is mostly verified, while fakes make up most of the GPTBot and ChatGPT-User hits.

Block GPTBot and you've made a training decision. You haven't touched search, and you haven't touched ChatGPT-User.

ClaudeBot, Claude-SearchBot and Claude-User

Anthropic lists ClaudeBot for training, Claude-User for user requests and Claude-SearchBot for search results. Its help page (opens in a new tab) says it honors robots.txt and the non-standard Crawl-delay. It also says it won't try to bypass CAPTCHAs.

Across the five sites, ClaudeBot was the busiest verified training crawler in my logs.

Verified and fake hits for ClaudeBot, Claude-User and Claude-SearchBot across five sites.ClaudeBot is mostly verified, Claude-User barely, and Claude-SearchBot not at all.

Claude-SearchBot never showed up as itself. It was faked on all five sites. Treat every Claude-SearchBot hit in your logs as unproven.

PerplexityBot and Perplexity-User

Perplexity's documentation (opens in a new tab) says PerplexityBot surfaces sites in Perplexity search and isn't used to train foundation models. Perplexity-User is different. Because a user requested the fetch, Perplexity says it "generally ignores robots.txt rules".

In Andre Alpar's June 2026 test (opens in a new tab), only 3 of 12 AI assistants left a disallowed page alone. They were ChatGPT, Claude and Perplexity.

So OpenAI and Perplexity say their fetchers may ignore the file, and one test saw them comply. Neither is a guarantee.

Across the five sites, Perplexity-User never verified once.

User agentHitsVerifiedFake
PerplexityBot2,1755041,671
Perplexity-User1,71301,713

So my logs can't say what Perplexity-User does with robots.txt.

What the verified crawlers got from my servers

Here is what the 13,075 verified AI hits got back.

ResponseShare of verified AI hits
20093%
Redirect2%
4xx5%
5xx0%

The 4xx didn't spread evenly. 652 of the 657 came from two of the five sites.

A 200 only means something arrived. In Andre Alpar's June 2026 rendering test (opens in a new tab), ChatGPT, Claude and Perplexity answered from the raw HTML and never ran the page's JavaScript. If your content arrives through JavaScript, the crawler gets an empty shell. A bot challenge page returns a 200 too.

What my robots.txt test did and didn't prove

The limit: four of my five robots.txt files say User-agent: * and Allow: /. So there's nothing to obey.

The fifth, on www.vitalsentinel.com, has three Disallow lines. They cover /api/, /tools/inspector/r/ and /api/tools/inspector/media/.

So the real test is one site. Zero of its 1,611 verified AI hits landed in a disallowed path.

A robots.txt compliance card saying no bot visited a blocked page.The card says no bot visited a blocked page, and every verified bot stayed off the pages the rules block.

Now count everything that claimed the name, verified or not.

GPTBot on www.vitalsentinel.comHitsAsked for a blocked path
Verified460
Fake51839
All56439

So most fake GPTBot hits stayed out too, and I won't claim the fakes ignored the file.

The share that stayed out of disallowed paths also drops below 100% for ClaudeBot, PerplexityBot and ChatGPT-User. Every missing point came from traffic that failed or skipped verification.

That's one site, three Disallow lines and 30 days. It isn't a controlled experiment. It does fit what the vendors document for their crawlers.

RFC 9309 (opens in a new tab) is the Robots Exclusion Protocol, the standard behind all of this. It says robots.txt is not a form of access authorization. If the file returns a 4xx, a crawler may access anything. If it returns a 5xx, the crawler must assume a complete disallow.

More than one in four AI bot hits were fake

The fakes barely cared which name they wore. Each of the eight AI bot names drew between 1,617 and 1,925 fake hits, while verified traffic ran from zero to 8,351.

Fake and verified hits for eight AI bot names: fakes sit at roughly the same level for each.Fake hits sit at roughly the same level for every name, while verified traffic ranges from none for Claude-SearchBot and Perplexity-User to 8,351 for OAI-SearchBot.

The Faked identities table for one site, listing each bot name and its fake hits.The Faked identities table for one of the five sites, from an earlier snapshot with smaller totals than above: most bot names are faked a few hundred times, and Googlebot is faked the most, at 649 hits.

Most of the fake traffic was scanners: 89% got a 4xx, and much of it asked for files like /.env, the same network and trick I covered for Googlebot in Fake Googlebot: What 100 Sites Return to It.

Half of AI bot hits can't be checked

Bytespider alone made 11,200 of the unverified hits.

Bytespider was also the largest AI name on www.vitalsentinel.com, with 5,004 hits. 4,423 of those got a 2xx, which is 88%. There's nothing I can check it against. Whether that was the real Bytespider, I can't say.

A check you can run this week

Pull a month of access logs and run them through the free log file analyzer. It lists every bot in the file and flags the fakes, AI crawlers included.

To check the method by hand, open the vendors' own IP lists. OpenAI publishes gptbot.json (opens in a new tab), searchbot.json (opens in a new tab) and chatgpt-user.json (opens in a new tab). Anthropic publishes bots.json (opens in a new tab). Perplexity publishes perplexitybot.json (opens in a new tab) and perplexity-user.json (opens in a new tab). Any address outside the list isn't the vendor.

The second half is compliance. Count the hits on your disallowed paths per user agent, and split them by whether the IP checked out. That's the robots.txt compliance number, and the only version worth quoting.

If you set rules, check a URL against your live file in the free robots.txt tester before you ship it.

Count the three piles apart

Here's what 58,422 AI bot hits on five sites came down to.

Fewer than one in four came from an address the vendor publishes. More than one in four provably came from someone else. The other half can't be checked at all, and Bytespider is the biggest part of it.

The real crawlers behaved. They got a 200 almost every time, and on the one site with Disallow rules, none of them touched a blocked path.

The fakes didn't care which name they wore. Every AI bot name drew roughly the same fake traffic, mostly scanners hunting for files like /.env. Claude-SearchBot and Perplexity-User never once showed up as themselves.

So before you quote any AI crawler number, split it three ways: verified, fake and unverified. Report the unverified pile on its own, never as real traffic and never as fake.

A GPTBot count that skips the address check is a count of a string.

Martin Stepanek

Written by

Martin Stepanek

Enterprise Technical SEO Consultant

15+ years building enterprise websites and designing the infrastructure behind them, and 5+ in technical SEO and web performance. My work is making companies findable – not just on Google, but inside ChatGPT and the other AI engines people now ask first. I write a regular newsletter and blog on technical SEO at TechSEO Vitals, and I built VitalSentinel because the gap between something breaking and someone noticing should be minutes, not days.

// newsletter

The VitalSentinel Newsletter

New features as they ship and practical technical SEO tips. One email a week at most.

Unsubscribe any time

// free to start

Know what changed on your site

Page changes, robots.txt and Core Web Vitals from real users, indexing and the Google and AI crawlers in your server logs, plus uptime and SSL, in one app.

1 domain freeNo credit cardAll 13 modules5.0on G2