◂ ALL NOTES FILE No. 001

I made my scanner a Cloudflare verified bot. Here's what I learned

Setting up Web Bot Auth with HTTP message signatures - and why a scanner should identify itself

Our agentic readiness scanner crawls shops and checks how well they are prepared for AI agents: structured data, WebMCP support, robots.txt rules, that kind of thing. Ironically, the scanner itself is exactly the kind of automated traffic that shops increasingly block. So I did the honest thing: I made it a Cloudflare verified bot. Here is what that involved and what I learned.

The problem: bots lie, so everything gets blocked

The web’s traditional bot identification is a user-agent string, which is just a pinky promise. Anyone can write Googlebot into a header. So the industry response has been aggressive bot protection: IP reputation lists, rate limiting, JS challenges, blocking entire hosting provider ranges. Collateral damage included.

For a scanner like ours this is a real problem. If the scanner gets blocked or challenged, the scan results are wrong - we would report a shop as “not agent-ready” when in fact the shop is perfectly fine and we just never saw it.

Enter Web Bot Auth

Web Bot Auth replaces the pinky promise with cryptography. Instead of claiming an identity, your bot proves it by signing every request. The mechanism builds on two IETF drafts:

  • HTTP Message Signatures (RFC 9421) - sign selected components of an HTTP request (like @authority, the host you’re calling) with a private key, attach the result as Signature and Signature-Input headers.
  • A directory draft that defines how you publish your public keys at /.well-known/http-message-signatures-directory, so anyone can verify your signatures. The directory response itself is signed too, so nobody can mirror your directory and register on your behalf.

Cloudflare only supports Ed25519 keys, which keeps things simple.

The setup, in four steps

1. Generate a key. One openssl command, then convert the public key to a JWK:

openssl genpkey -algorithm ed25519 -out private-key.pem

2. Host the key directory. Serve a JWKS at /.well-known/http-message-signatures-directory over HTTPS, with Content-Type: application/http-message-signatures-directory+json, and sign that response with the same key. This was the fiddliest part - the Signature-Input needs the right tag (http-message-signatures-directory), the keyid as the JWK thumbprint, and @authority with the req parameter. Cloudflare has a CLI tool to validate your directory, which saved me some debugging.

3. Register with Cloudflare. There’s a bot submission form in the dashboard (Manage Account → Configurations → Bot Submission Form). You pick “Request Signature” as the verification method and point them at your directory URL.

4. Sign every request. Once approved, the bot attaches three headers to each request:

Signature-Agent: "https://your-host.example"
Signature-Input: sig1=("@authority" "signature-agent");created=...;expires=...;keyid="...";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:...base64 signature...:

Two things tripped me up here: Signature-Agent is a structured field, so the quotes around the URL are mandatory and part of the signed components (forget it in the component list and verification fails). And expires should be short - a minute or so - but not so short that clock skew or latency kills you.

Cloudflare offers a test endpoint at https://crawltest.com/cdn-cgi/web-bot-auth that tells you whether your signature verifies (200), your key is unknown (401), or your message is malformed (400). Very useful before going live.

Why it’s worth it for the scanner

The point is not vanity. It’s about measurement quality:

  • Fewer false negatives. Shops behind Cloudflare bot protection would previously challenge or block our scanner, and we’d measure the block page instead of the shop. As a verified bot, we get through and measure the real thing.
  • It’s honest. A scanner that evaluates how “agent-friendly” a site is should itself behave like a well-mannered agent: stable identity, reasonable request rates, respects robots.txt. Web Bot Auth formalizes that honesty - you can be removed from the allowlist if you misbehave, which keeps you accountable.
  • It’s the direction the web is heading. As agents become a real share of traffic, site owners need a way to distinguish “an agent acting for a user” from “a scraper pretending to be one”. Cryptographic identity is a much better foundation than user-agent sniffing. If we want shops to be agent-ready, agents have to become identifiable first - so we might as well start with ourselves.

Take away

Getting verified is maybe an afternoon of work: one key, one well-known endpoint, a registration form, and a signing step in your HTTP client. The drafts are still drafts, but Cloudflare’s implementation is real and documented, and the test endpoint makes debugging painless. If you run any kind of automated traffic against other people’s sites - a crawler, a monitor, a scanner - I’d argue this is becoming table stakes.