Server Log Monitoring
See what search engine and AI crawlers actually do on your site
Search Console shows what Google reports. Your server logs show every request. Server Log Monitoring reads your access logs and tells you which search engine and AI crawlers visited, what your server answered, which bots are fake, and how much of your crawl budget went to errors, redirects and parameterized URLs. It stores search engine and AI crawler traffic only – human visitors' requests never reach your reports.
- Googlebot hits, errors and redirects, straight from your logs
- AI crawlers by intent, with robots.txt compliance
- Fake Googlebots flagged against the IP ranges Google publishes
- Real-time collection through a Cloudflare Worker, on every plan
Googlebot spent the week on redirects and parameter URLs. Nothing told you.
Search Console shows what Google chooses to report, and its URL Inspection data is limited to 2,000 URLs per property per day and can be months old. The real record of what crawlers did on your site sits in your access logs, and almost nobody reads them.
Crawl budget goes to redirects, errors and parameterized URLs, and nothing tells you how much
Any request can claim to be Googlebot in its User-Agent
AI crawlers read your content, and you cannot tell which ones or whether they respect robots.txt
Pages with search impressions quietly stop being re-crawled
Sitemap URLs Googlebot ignores go unnoticed for months
A slow or erroring server answers crawlers badly, and you hear about it from a traffic drop
Every crawler request, classified and verified
Each hit is matched to a known search engine or AI crawler, checked against the IP ranges or reverse DNS its vendor publishes, and grouped into sections you define. Reports fill as requests arrive.
Bot Verification
Every hit is checked against the IP ranges or reverse-DNS names the crawler's vendor publishes and labeled Verified, Unverified or Spoofed. Some AI crawlers publish neither, so their verified share shows as a dash, with the reason, rather than 0%.
Fake Bot Detection
Requests that claim to be Googlebot, Bingbot or an AI crawler from an address that bot does not use are named as fake on the Security report, grouped by the networks they came from.
Googlebot at a Glance
Verified Googlebot hits, the share answered 2xx, redirected or erroring, and response time, plus a crawl capacity chart that sets Googlebot's crawling against how fast your server answered.
AI Crawler Tracking
See which AI crawlers read your site, split by intent: training, live fetches by AI assistants, and search or agents. Each one shows its verified share, robots.txt compliance and 403 blocks.
Crawl Budget & Waste
Crawl waste is the share of bot hits that went to a redirect, an error, a URL with query parameters, or one your robots.txt disallows. A 304 Not Modified is a healthy answer and is not counted.
Status Codes & URLs to Fix
Every exact status code over time, a bot-by-status matrix, and a list of the URLs returning not-found errors, server errors or redirects to crawlers, flagged when they are in your sitemap.
Sections & Crawl Map
Group URLs into sections with prefix or regex rules that apply to past data too. The crawl map shows which folders Googlebot has reached and which it has never touched.
Joined With Your Other Data
Intelligence joins crawl data with Google Search Console, Web Analytics, RUM and your sitemap: stale crawl on pages with impressions, a crawl-to-click funnel, and poor Core Web Vitals on heavily-crawled pages. Site Health and Sitemap Monitoring show the last Googlebot visit to each page from the same logs.
Explore, API & MCP
Find exactly which crawler hit which page with a given status code, and share the answer as a link. The same data is readable through the public API with the read:logs scope and through 27 MCP tools.
See which AI crawlers read your site, and whether they listen
The AI crawlers report separates training crawls from live fetches, where an AI assistant reads a page because someone just asked about it. For each crawler you see its hits, verified share, robots.txt compliance and 403 blocks, plus the pages Googlebot crawled that no AI crawler fetched.
- Training, live fetch and search or agent traffic split by intent
- Robots.txt compliance and Blocked (403) per AI crawler
- AI coverage gap: pages Googlebot crawled that AI crawlers missed
Find the pages Google stopped re-reading
Logs record every verified Googlebot fetch, not a quota-limited sample. Joined with Search Console and your sitemap, they show which pages with impressions have gone stale, which sitemap URLs Googlebot has stopped visiting, and which crawled pages are missing from the sitemap.
- Stale crawl on pages with impressions, with impressions at risk
- Sitemap URLs Googlebot has stopped visiting
- Crawled by Google, missing from your sitemap
Hear about crawl problems before your rankings do
Pick from 15 predefined alerts covering blocked crawlers, server errors, crawl drops and spikes, wasted crawl budget, spoofed bots and AI crawlers. Firing alerts also appear under Open findings on the module's Overview.
- Know when a verified crawler is blocked or hitting server errors
- Catch a Googlebot crawl drop against its own baseline
- Get told about a spoofed-bot surge or a new AI crawler
From access log to answers
Connect a log source
If your site runs through Cloudflare, deploy our Worker into your own Cloudflare account for real-time collection, on every plan. Cloudflare bills it to you at roughly $0.30 per million requests, and your site keeps working if it fails. On paid plans you can instead collect log files over SFTP, FTPS, Amazon S3, Cloudflare R2 or Google Cloud Storage, or upload one. Each domain uses one source at a time.
Only crawler traffic is kept
Search engines, AI training crawlers, AI search crawlers, AI assistants and AI agents are stored. Human visitors' requests are not stored (an uploaded log file is itself kept as-is until you delete the source), and SEO tools, scanners and uptime checkers are dropped on arrival and only counted as excluded.
Every hit is classified
Each request is matched to a known crawler, verified against its vendor's published IP ranges or reverse DNS where they exist, and grouped into your sections.
Reports and alerts fill in
The Worker reports from the moment it is connected. Log-file sources sync every 12 to 24 hours and can import past files. Most reports go back up to 12 months; a few need 30 days of data first.
Real scenarios, real fixes
"Rankings for a key section slipped, and the last-crawl dates in Search Console were weeks old."
Stale crawl on pages with impressions and the crawl-to-click funnel show, from every verified Googlebot fetch, which pages Google stopped re-reading and where they drop off.
"We have no idea which AI crawlers take our articles, or whether our robots.txt rules stop any of them."
The AI crawlers report splits training crawls from live assistant fetches and shows robots.txt compliance and 403 blocks for each crawler, so policy decisions rest on real traffic.
"Googlebot spends its time on filter and sort URLs instead of new products."
Crawl budget shows parameter waste and the top query parameters Googlebot crawls, and duplicate-variant crawl finds the same page crawled under more than one address.
Every module included. You only choose your coverage
All plans include all twelve modules. Start free, upgrade only when you need wider coverage.
Free
Perfect for trying out VitalSentinel
- 1 domain
- 500K bot hits/mo
- Log sources: Cloudflare Worker only
Lite
For consultants and growing businesses
- 5 domains
- 10M bot hits/mo
- Log sources: Cloudflare, SFTP and others
See what crawlers actually do on your site
Server Log Monitoring is on every plan, Free included, with a monthly hit allowance. Connect the Cloudflare Worker and see crawler hits as they arrive.