Introducing Server Log Monitoring: See What Crawlers Actually Do on Your Site

Your server writes a line for every request it answers, whether anyone reads it or not. For most sites, nobody does. Yet those lines are the only first-hand record of what a crawler actually did: which URL it asked for, what your server gave it, and how long that took.

Starting today, VitalSentinel reads them. Server Log Monitoring is live on every plan, including Free. It shows which search engine and AI crawlers hit your server, how your server answered them, which of them are fake, and how much of your crawl budget went somewhere it earned nothing back.

What Google Reports, Versus What Googlebot Did

Most crawl analysis runs on reported data. Search Console tells you what Google chooses to report. URL Inspection is precise, but it covers 2,000 URLs per property per day, and the crawl date it gives you can be up to three months old. Useful, but it is a sample, and it is someone else's summary.

The log is not a sample. If Googlebot fetched a URL, there is a line for it, with the status code your server returned. That makes the uncomfortable questions answerable:

• Is Googlebot spending its visits on redirects, errors and parameter URLs instead of the pages you care about?
• Which pages that still earn impressions has Google stopped re-crawling?
• Which sitemap URLs does Googlebot ignore, and which pages does it crawl that are missing from the sitemap?
• Is your server slow to answer crawlers, even when it feels fine to you?

Why It Matters More Now

A few years ago "crawler" mostly meant a search engine. Today there are crawlers collecting pages to train AI models, crawlers building the indexes behind AI search answers, assistants fetching a page because someone just asked about it, and agents browsing your site on a person's behalf.

None of them report to you in Search Console, and your analytics was built to count people. Your robots.txt says what you asked them to do. The server log is the one place that shows what they actually did.

What Ships Today

The module opens on an Overview: total hits, response time, crawl waste, and the share of traffic that comes from AI crawlers, with a separate Googlebot block showing how many of its requests were answered with a 2xx, redirected, or met an error. The Googlebot crawl capacity chart puts verified Googlebot hits against your server's response time, with a one-line takeaway that names a nearby robots.txt change or downtime when there is one.

Behind it:

• Bots – every bot that hit the site, an AI crawlers view, and a Security view of faked identities.
• Pages – every URL a search or AI crawler fetched, Googlebot first, plus Sections you define by path, a Crawl budget report, and a Crawl map of which parts of the site Googlebot has reached and which it has never touched.
• Responses – every status code over time, a bot-by-status matrix, and URLs to fix, with a flag for whether each one is in your sitemap.
• Explore – find exactly which crawler hit which page with a given status code, and share the answer as a link.

Crawl waste has a strict definition here: hits to a redirect, an error, a URL carrying query parameters, or a URL your robots.txt disallows. A 304 Not Modified is a healthy answer and does not count.

AI Crawlers, Sorted by What They Want

The AI crawlers report splits AI traffic by intent: Training (AI crawler), Live fetch (AI assistant), and Other (search / agent). The distinction matters. A training crawl is a licensing question. A live fetch means a person is asking about your page right now.

For each AI crawler you see its hits, how much of it could be verified, its Robots.txt compliance, how often you blocked it with a 403, and when it was last seen. There is also an AI coverage gap card: pages Googlebot crawled successfully that no AI crawler fetched at all.

Real Googlebot, Fake Googlebot

Anyone can put "Googlebot" in a User-Agent. So every request is checked against the IP ranges or reverse DNS the vendor publishes, and gets one of three verdicts: Verified, Unverified, or Spoofed.

I want to be honest about the limits. Spoofed is only ever assigned when an address check proves the claim false. Unverified means it could not be confirmed either way, and that is not the same thing. Several AI crawlers, including ones Meta runs, publish neither IP ranges nor reverse DNS, so their verified share shows a dash with the reason instead of a misleading 0%. A low verified share on those is not proof of fraud.

The Googlebot figures – the Googlebot tiles, the crawl map, and the stale-crawl and sitemap reports – count verified Googlebot only, so a scraper wearing its name never inflates your crawl numbers. The fakes go to the Security view instead, grouped by the identity they claimed and the networks they came from.

Connecting a Source

Open a domain, open Server Log Monitoring in the sidebar, and choose Connect a log source. A domain collects from one source at a time, and you can change it later. There are two ways in.

Cloudflare Worker (recommended, every plan). If your site runs through Cloudflare, VitalSentinel deploys a Worker into your own Cloudflare account, on that domain's zone, and collection is real time. You create an API token with three permissions and a short wizard does the rest. There is nothing to install on your server, and no other Worker on the account is touched.

The Worker runs on your Cloudflare bill, not ours, at roughly $0.30 per million requests. Where your token can read the zone's analytics, the wizard estimates the cost from the last seven days of traffic and asks you to accept it before anything is deployed. It fails open, so your site keeps working if the Worker fails. On Workers Free, Cloudflare stops the Worker for the rest of the day once its daily request cap is crossed: your site stays up, but collection pauses until the next day.

Your own server (paid plans). Point it at your logs over SFTP or FTPS, or at a bucket in Amazon S3, Cloudflare R2 or Google Cloud Storage. Or upload a file directly, up to 8 GB. These sources sync every 12 to 24 hours rather than in real time. The format is detected for you: Combined (Apache/Nginx), IIS (W3C extended), newline-delimited JSON, Cloudflare log fields, AWS Application Load Balancer, or AWS CloudFront. If yours is a custom Nginx log_format, you paste the directive instead. The one hard requirement is that your logs include the User-Agent.

A log-file source can also import past files with Re-collect historical logs, so your reports can start with history already in them instead of from today.

The setup guide covers where to find your access logs and which fields unlock which reports, and Connecting a Log Source walks through both paths step by step.

Bots Only, on Purpose

This module stores five kinds of traffic: search engines, AI training crawlers, AI search crawlers, AI assistants and AI agents. Nothing else.

SEO tools, social link previews, feed readers, archives, ad verifiers, security scanners, uptime and page-speed checkers, and clients that give no name are dropped and never stored. Neither are your human visitors. There is no setting to widen this, and that is deliberate. Server Log Monitoring is not a second analytics tool, and I did not want it quietly becoming a record of the people who read your site. If you want to understand your human audience, that is what Web Analytics is for.

It also keeps the reports honest. When every chart is crawler traffic and nothing else, a spike means a crawler did something, not that a link went viral or a scanner wandered by.

How that plays out depends on the source:

• With the Cloudflare Worker, human requests never leave your Cloudflare account. Only crawler requests are forwarded.
• With a log file, the file comes to us as your server wrote it, and human requests are dropped while it is parsed, so they never enter the data your reports are built on. An uploaded file itself is kept until you delete the source, so if it was a one-off import, delete it afterwards.
• IP addresses are truncated when they arrive. For unverified and spoofed requests, the reports never show an address at all, only the network it came from.
• Query parameter names are stored, never their values.

The bot traffic it drops is still accounted for. The Settings page shows how many requests were excluded and why, so you can see the filter working rather than take my word for it. The Data & Privacy page has the full detail.

It Joins the Rest of VitalSentinel

A log on its own tells you what crawlers did. Joined with your other data, it tells you whether it mattered. The Intelligence section has four tabs, each one lighting up once the matching data is there:

• Search Console – a crawl-to-click funnel from "not fetched by Googlebot" through to clicks, and stale crawl on pages that still earn impressions.
• Web Analytics – From AI crawl to AI search traffic, which shows whether a page fetched by an AI crawler later got a visit from an AI search or chat product, and the pages people visit that Googlebot hasn't crawled.
• RUM – how much slower a bot sees the same page respond than your real users do, and pages with poor Core Web Vitals that crawlers hit hardest.
• Sitemap – pages Google crawls that are missing from your sitemap, and sitemap URLs Googlebot has stopped visiting.

Two existing modules gain log-based cards too. Indexing Monitoring adds Stale crawl, recomputed from the logs, Crawled but not indexed, against real crawl frequency, Crawl-to-index latency, and Where to spend today's inspections, so your daily URL Inspection quota goes where it is most useful. Robots.txt Monitoring adds Bot Compliance With Disallow Rules, Disallow Rules Bots Are Still Crossing, and AI Bot Policy vs. Reality: what your robots.txt asks of AI bots, next to what they actually do.

There are 15 new alerts, from "A verified crawler is being blocked" and "Googlebot crawling has dropped" to "A new AI crawler discovered your site" and "Money pages are going stale".

And since August's API and MCP server launch, it is all reachable from your assistant as well. The new read:logs scope is read-only, and there are 27 MCP tools for it, so "Is Googlebot getting errors on example.com?" or "Which AI crawlers read my site, and do they respect robots.txt?" is now a question you can just ask.

Where to Start

If your site is on Cloudflare, connect the Worker. The first request is confirmed within about five minutes of arriving, though on a quiet site crawlers come on their own schedule, so give it time.

The Server Log Monitoring introduction is the best place to read first, followed by the reports and Intelligence guides. The feature page has the overview.

VitalSentinel already watched your Core Web Vitals from real users, your indexing and robots.txt, and your uptime and SSL. Now it can tell you what the crawlers deciding your visibility actually do when they arrive. Connect a source, look at what Googlebot really fetched last week, and tell me what surprised you.