Log file analyzer for SEO
What did Googlebot, Bingbot and AI crawlers do on your site? Drop an access log into this free log file analyzer to see every bot, which ones are fake, what your server answered and where crawl budget went. It runs in your browser, so the file is never uploaded.
Drop an access log here, or choose a file
nginx or Apache combined, Cloudflare Logpush, JSON lines, IIS, CloudFront or AWS load balancer logs. Plain or gzipped (.gz). Large files are fine.
Your log stays on your computer. I read it in your browser and nothing is uploaded. Apart from its own code, the only thing this tool downloads is the list of published bot IP ranges.
That was one file. Server Log Monitoring reads every hit.
It keeps reading your logs and alerts you when crawlers hit errors, Googlebot visits less or fake bots rise. Ask Theo, the AI assistant in the app, to explain any finding.
What it checks
- Reads nginx and Apache combined logs, Cloudflare Logpush, JSON lines, IIS, CloudFront and AWS load balancer logs, plain or gzipped
- Names every bot with the same table Server Log Monitoring uses, and checks each hit against the IP ranges Google, Bing, OpenAI, Anthropic and others publish
- Groups AI crawlers by company: training crawls, indexing for AI answers and live fetches for a user
- Shows status codes and top URLs per bot, 404s and 5xx sent to search crawlers, the hours with the most errors, parameter URLs and duplicate addresses
What it does not
- Your log never leaves your computer, so I cannot run reverse DNS. Bots verified that way, such as YandexBot and Baiduspider, read "Cannot be checked"
- Bots that publish no IP ranges, such as Bytespider, can be counted but not confirmed or caught faking
- It sees only what the file holds. Requests a CDN answered from cache, or another server handled, are not in it
- Behind a proxy, load balancer or CDN, the log must record the visitor's real IP. If it records a private proxy address, I cannot check bots. If it records a CDN's public address, real bots read Fake
- It cannot tell which URLs your robots.txt disallows, so crawl waste counts redirects, errors and parameter URLs only
Questions
Is my log file uploaded anywhere?
No. A Web Worker in your own browser tab reads the file, and nothing in it is sent to VitalSentinel or anyone else.
Apart from its own code, the only request the tool makes is for the published bot IP ranges: the same file for every visitor, carrying nothing from your log.
Where do I find my access log?
It depends on the server:
- nginx: /var/log/nginx/access.log
- Apache: /var/log/apache2/access.log on Debian and Ubuntu, /var/log/httpd/access_log on Red Hat
- Shared hosting: usually under Logs or Raw access logs in the control panel
- Cloudflare: a Logpush job of HTTP requests
The default combined format is what you want. The common format has no user agent, so no bot can be named from it.
How does it tell a fake Googlebot from a real one?
Google publishes the IP ranges its crawlers use. A hit that calls itself Googlebot from an address outside them is fake. Bing, OpenAI, Anthropic, Perplexity, Apple and Common Crawl publish ranges too, and I check them the same way.
To check one address by reverse DNS as well, use the Googlebot IP checker.
What counts as wasted crawl budget?
Search and AI crawler hits that earn nothing back: a redirect, an error, or a URL with query parameters. A 304 Not Modified is a healthy answer and is not counted.
Server Log Monitoring also counts URLs your robots.txt disallows, and tracks the share over time by site section.
Stop checking by hand
Core Web Vitals from real users, indexing and robots.txt, uptime and SSL, monitored continuously. Start on the Free plan and add your first domain in under a minute.