Web Bot Auth and HTTP Message Signatures

A valid bot signature tells the receiving site which operator stands behind a request. It doesn't decide whether that operator may read an account, consume an expensive endpoint, or submit the same job repeatedly. I want that separation written into the access policy before signed traffic receives different treatment.

Web Bot Auth gives automated clients a way to sign outbound HTTP requests and lets origins or edge providers verify them. The bot operator controls the signing key and publishes the corresponding public directory. A content origin consumes verification results; it doesn't need an operator directory merely because bots visit it.

RFC 9421 supplies the published signature machinery. The current protocol work is draft-ietf-webbotauth-httpsig-protocol-00, dated September 1, 2026. Cloudflare's documented verifier profile must still be checked separately.

Choose the control for the traffic

These controls can coexist on one request. For example, a recognized crawler may read a public article and still receive a rate-limit response. A signed request to a customer's report must also pass that report's authorization check.

The parties and request flow

There are three practical roles:

1. A bot operator controls the automated client and its private signing key.
2. A key directory publisher serves the operator's public keys over HTTPS.
3. A verifier receives a request, locates the directory, validates the signature, and applies local policy.

The first two roles often belong to the same organization. The verifier may be the destination origin or a service such as Cloudflare in front of it.

Authentication answers whether the request is tied to a registered key under the claimed operator's control. It doesn't establish that the requested action is authorized. A valid bot can still exceed rate limits, access the wrong tenant, submit harmful input, or violate a content-use policy. Authorization and application controls remain separate.

RFC 9421 provides the signature machinery

HTTP Message Signatures defines how a signer selects message components, constructs Signature-Input, and carries signature bytes in Signature. Components can include HTTP fields and derived values such as @authority, @method, and @path.

An illustrative shape for the documented Cloudflare profile looks like this; the placeholder signature is not executable:

Signature-Agent: "https://bot.example"
Signature-Input: sig1=("@authority" "signature-agent");created=1784217600;expires=1784217660;keyid="thumbprint";tag="web-bot-auth"
Signature: sig1=:base64-signature-bytes:

The exact signature base construction comes from RFC 9421. Implementers shouldn't concatenate fields by hand from an example. Structured Field parsing, component identifiers, parameter order, byte encoding, and algorithm behavior all affect verification.

Component selection defines what the signature protects. Signing @authority binds the request to its destination authority. Signing the signature-agent field binds the directory reference. Depending on the request, an operator may also bind method, path, query, or a content digest.

Intermediaries complicate the choice. A proxy may normalize or replace fields. If a signed component changes in transit, verification fails even when the request began as legitimate. Test through the complete production route.

The operator key directory

Cloudflare's Web Bot Auth documentation currently tells bot operators to publish a key directory at:

/.well-known/http-message-signatures-directory

The directory is served over HTTPS with a dedicated media type and a JSON Web Key Set containing public keys. Cloudflare requires the directory response itself to be signed. That proof prevents another party from copying a public key set to a different domain and claiming the operator identity during registration.

The private key never belongs in the directory. For an Ed25519 JWK, public members include kty, crv, and x; a private d member would disclose the signing secret. Key-generation and publication tooling should enforce that boundary.

Cloudflare documents draft-meunier-http-message-signatures-directory-03 syntax: Signature-Agent is a quoted structured string, and the field must be covered by Signature-Input. Its verifier rejects the dictionary syntax found in later draft work. Passing one profile's fixtures therefore doesn't establish compatibility with another.

The working-group draft and Cloudflare profile differ

The old architecture and directory drafts are now marked replaced and archived in the IETF tracker. Their successor is the working-group protocol draft 00, still work in progress, with an expiry of March 5, 2027. An archived draft link is useful history but shouldn't be labeled the active specification.

Cloudflare's current instructions still name the older architecture and directory revisions for its integration. They specify Ed25519, signed directories, required request parameters, unsupported component handling, and bot registration. Implement the version the intended verifier accepts and keep its fixtures separate from experiments against the working-group draft.

An origin building its own verifier also owns directory-fetch policy, supported algorithms, caching, and failure handling. Choosing RFC 9421 doesn't supply those decisions.

Replay resistance needs an explicit policy

Signature validity alone doesn't guarantee that a request is fresh. created and expires parameters limit the accepted time window. A nonce can distinguish requests when the verifier tracks it.

Cloudflare recommends a short expires interval, often about a minute. Its current documentation says the nonce field can be included but Cloudflare doesn't yet validate nonce reuse against a database of seen values. That limitation should be stated in a threat model. A short validity period narrows replay exposure but doesn't provide single-use enforcement.

Sensitive actions may require stronger application-level controls. Use idempotency keys for mutation, bind signatures to method and target, validate request bodies where intermediaries won't modify them, and reject replays at the business-operation layer. Keep verifier clocks synchronized and define acceptable skew.

Key rotation also affects availability. Publish overlapping public keys long enough for caches and registrations to update. Remove compromised keys through a documented emergency path. Monitor directory retrieval failures and signature rejection rates.

Receiving signed bots at an origin

A site behind Cloudflare can use verified-bot outcomes in edge policy without implementing a second signature verifier. The origin should understand what identity information the edge forwards and which policy decisions depend on it.

If verification is performed at the origin, directory retrieval becomes a server-side request. Constrain HTTPS, redirects, DNS resolution, response size, media type, caching, and fetch timeouts. Defend against server-side request forgery and cache poisoning. Don't allow a request to choose an arbitrary internal directory URL.

Failure behavior should be predictable. Distinguish malformed signatures, unknown keys, expired requests, directory failures, and valid identities that lack authorization. Avoid logging private request bodies or credentials while still capturing enough metadata for technical assessment.

Why TKOResearch doesn't publish a directory

The current TKOResearch site publishes articles and receives contact submissions. It doesn't send a public crawler or agent whose requests need operator authentication.

An empty directory describes no key. Generating a key that never signs production traffic adds rotation and monitoring obligations without supporting a service. Site TLS keys are also unsuitable because they were issued and managed for a different protocol purpose.

The correct adoption trigger is an outbound bot product with a named operator, documented destinations, request policy, key custody, rotation, registration, observability, and incident handling. The directory and signed-request support should ship with that product.

Receiving authenticated bots is a separate choice. Cloudflare can verify supported requests at the edge. That capability doesn't require the destination site to claim it is a bot operator.

Adoption checklist for a real bot operator

Before registration, confirm:

• the bot's purpose, product token, and operator identity are public and stable;
• the private key is generated and held in an approved key-management system;
• the HTTPS directory is signed, monitored, size-limited, and free of private key members;
• every required request component survives the production proxy path;
• created, expires, nonce handling, and business idempotency match the threat model;
• rotation and emergency revocation have been tested;
• verifier-specific syntax and algorithm limits are covered by integration tests.

Also document what successful authentication permits. Verified identity should feed an explicit allow, rate, or review policy. It shouldn't bypass authorization by default.

Tie verified identity to an explicit policy

For this publishing site, receiving verified bots is the relevant role. Its operator can decide which identities receive access to public content and under what rate limits without publishing a bot directory.

If TKOResearch later operates an outbound client, its team would own keys, signing, registration, and directory maintenance. I would review that client's actual request permissions and retry behavior before treating a successful signature check as release readiness.

Primary references

• RFC 9421: HTTP Message Signatures
• Current Web Bot Auth working-group protocol draft
• Archived architecture draft and replacement history
• Cloudflare Web Bot Auth documentation