VitalSentinel Bot is now a Cloudflare verified bot

Site Health Monitoring and Page Monitoring look at your pages from the outside, the way a search engine does. That only works when your firewall lets our crawler in. On a site behind Cloudflare, it has always come down to one question: is this request really from VitalSentinel?

Starting today, Cloudflare can answer that question on its own. Our crawler, VitalSentinel Bot, now signs its requests with Web Bot Auth, and Cloudflare has approved it as a verified bot in BotBase, the registry behind its bots and agents directory. It is listed under SEO and Monitoring & Operations.

A name anyone can borrow

Until now, there were two ways to recognize our crawler: its User-Agent, VitalSentinel Bot/1.2 (+https://www.vitalsentinel.com/bot), and the list of IP addresses on the bot page.

A User-Agent is only a label. Any script can send it, and fake crawlers borrow well-known names all the time, which is why Server Log Monitoring separates verified bots from fake ones. In Cloudflare's own words, "user agent headers alone are easily spoofed." An IP list is better, but it is one more thing to keep current in your firewall.

A signature instead of a label

Web Bot Auth lets a bot prove who it is. It is built on HTTP Message Signatures (RFC 9421) and described in IETF drafts, and Cloudflare supports it.

We hold a private Ed25519 key and publish the matching public key in a key directory at https://app.vitalsentinel.com/.well-known/http-message-signatures-directory. Each request the crawler sends carries three extra headers: one that points to that directory, one that says what the signature covers, and the signature itself.

Signature-Agent: "https://app.vitalsentinel.com"
Signature-Input: sig1=("@authority" "signature-agent");created=...;keyid="...";alg="ed25519";expires=...;nonce="...";tag="web-bot-auth"
Signature: sig1=:...:

The signature covers the site the request goes to, so it is useless on any other site. It expires after 60 seconds, and every one carries a fresh nonce. The private key lives in a separate service with no network access. The crawler asks it for each signature and never holds the key itself.

Cloudflare reads our key directory, checks the signature and knows the request came from us. There is no User-Agent to trust and no address to look up.

What is signed

Every https request the Site Health Monitoring and Page Monitoring crawler sends: the page itself, each redirect hop, the files a page loads from your site when it renders, image checks and its robots.txt fetches.

A plain http request goes out unsigned, because a signature read off an unencrypted connection could be replayed. Most sites redirect http to https on the first hop, and the request that follows is signed. If the signing service is ever unavailable, the crawler sends the request unsigned rather than skip your check.

Robots.txt Monitoring, Sitemap Monitoring and Uptime Monitoring do not sign their requests. The bot page says how to allow each of them.

What you need to do

Most likely nothing. Wherever your Cloudflare setup lets verified bots through, such as the Verified bots setting in Super Bot Fight Mode, our crawler now counts as one.

If you already allow VitalSentinel Bot with a WAF rule, as described in the Cloudflare guide, keep it. Nothing about it changes, and it still covers the requests that are not signed. And if Site Health Monitoring ever finds your firewall blocking the crawler, its Crawler access settings still walk you through allowing it.

From submission to approval

I submitted VitalSentinel Bot on October 6. After two rounds of review, Cloudflare approved it on October 24.

Cloudflare only verifies bots that are open about who they are and what they do, and that respect robots.txt. Our crawler follows your robots.txt, using a User-agent: VitalSentinel-Bot group if you write one and your User-agent: * group if you do not.

A firewall should not have to take a crawler's word for who it is. Now it does not have to take ours.