Web Bot Auth and HTTP Message Signatures

A valid bot signature identifies the signing operator. The receiving site still decides whether that identity may read an account, use an expensive endpoint or submit another job. Define those permissions before giving signed traffic different treatment.

Web Bot Auth gives automated clients a way to sign outbound HTTP requests and lets origins or edge providers verify them. The bot operator controls the signing key and publishes the corresponding public directory. A content origin consumes verification results; it doesn't need an operator directory merely because bots visit it.

RFC 9421 supplies the published signature machinery. The current protocol work is draft-ietf-webbotauth-httpsig-protocol-00, dated September 1, 2026. Cloudflare's documented verifier profile must still be checked separately.

Choose the control for the traffic

These controls can coexist on one request. For example, a recognized crawler may read a public article and still receive a rate-limit response. A signed request to a customer's report must also pass that report's authorization check.

The parties and request flow

There are three practical roles:

1. A bot operator controls the automated client and its private signing key.
2. A key directory publisher serves the operator's public keys over HTTPS.
3. A verifier receives a request, locates the directory, validates the signature, and applies local policy.

The first two roles often belong to the same organization. The verifier may be the destination origin or a service such as Cloudflare in front of it.

Authentication answers whether the request is tied to a registered key under the claimed operator's control. It doesn't establish that the requested action is authorized. A valid bot can still exceed rate limits, access the wrong tenant, submit harmful input, or violate a content-use policy. Authorization and application controls remain separate.

RFC 9421 provides the signature machinery

HTTP Message Signatures defines how a signer selects message components, constructs Signature-Input, and carries signature bytes in Signature. Components can include HTTP fields and derived values such as @authority, @method, and @path.

An illustrative shape for the documented Cloudflare profile looks like this; the placeholder signature is not executable:

Signature-Agent: "https://bot.example"
Signature-Input: sig1=("@authority" "signature-agent");created=1784217600;expires=1784217660;keyid="thumbprint";tag="web-bot-auth"
Signature: sig1=:base64-signature-bytes:

The exact signature base construction comes from RFC 9421. Implementers shouldn't concatenate fields by hand from an example. Structured Field parsing, component identifiers, parameter order, byte encoding, and algorithm behavior all affect verification.

Component selection defines what the signature protects. Signing @authority binds the request to its destination authority. Signing the signature-agent field binds the directory reference. Depending on the request, an operator may also bind method, path, query, or a content digest.

Intermediaries complicate the choice. A proxy may normalize or replace fields. If a signed component changes in transit, verification fails even when the request began as legitimate. Test through the complete production route.

The operator key directory

Cloudflare's Web Bot Auth documentation currently tells bot operators to publish a key directory at:

/.well-known/http-message-signatures-directory

The directory is served over HTTPS with a dedicated media type and a JSON Web Key Set containing public keys. Cloudflare requires the directory response itself to be signed. That proof prevents another party from copying a public key set to a different domain and claiming the operator identity during registration.

The private key never belongs in the directory. For an Ed25519 JWK, public members include kty, crv, and x; a private d member would disclose the signing secret. Key-generation and publication tooling should enforce that boundary.

Cloudflare documents draft-meunier-http-message-signatures-directory-03 syntax: Signature-Agent is a quoted structured string, and the field must be covered by Signature-Input. Its verifier rejects the dictionary syntax found in later draft work. Passing one profile's fixtures therefore doesn't establish compatibility with another.

The working-group draft and Cloudflare profile differ

The old architecture and directory drafts are now marked replaced and archived in the IETF tracker. Their successor is the working-group protocol draft 00, still work in progress, with an expiry of March 5, 2027. An archived draft link is useful history but shouldn't be labeled the active specification.

Cloudflare's current instructions still name the older architecture and directory revisions for its integration. They specify Ed25519, signed directories, required request parameters, unsupported component handling, and bot registration. Implement the version the intended verifier accepts and keep its fixtures separate from experiments against the working-group draft.

An origin building its own verifier also owns directory-fetch policy, supported algorithms, caching, and failure handling. Choosing RFC 9421 doesn't supply those decisions.

Replay resistance needs an explicit policy

Signature validity alone doesn't guarantee that a request is fresh. created and expires parameters limit the accepted time window. A nonce can distinguish requests when the verifier tracks it.

Cloudflare recommends a short expires interval, often about a minute. Its current documentation says the nonce field can be included but Cloudflare doesn't yet validate nonce reuse against a database of seen values. That limitation should be stated in a threat model. A short validity period narrows replay exposure but doesn't provide single-use enforcement.

Sensitive actions may require stronger application-level controls. Use idempotency keys for mutation, bind signatures to method and target, validate request bodies where intermediaries won't modify them, and reject replays at the business-operation layer. Keep verifier clocks synchronized and define acceptable skew.

Key rotation also affects availability. Publish overlapping public keys long enough for caches and registrations to update. Remove compromised keys through a documented emergency path. Monitor directory retrieval failures and signature rejection rates.

Receiving signed bots at an origin

A site behind Cloudflare can use verified-bot outcomes in edge policy without implementing a second signature verifier. The origin should understand what identity information the edge forwards and which policy decisions depend on it.

If verification is performed at the origin, directory retrieval becomes a server-side request. Constrain HTTPS, redirects, DNS resolution, response size, media type, caching, and fetch timeouts. Defend against server-side request forgery and cache poisoning. Don't allow a request to choose an arbitrary internal directory URL.

Failure behavior should be predictable. Distinguish malformed signatures, unknown keys, expired requests, directory failures, and valid identities that lack authorization. Avoid logging private request bodies or credentials while still capturing enough metadata for technical assessment.

Publish a directory for an outbound bot

The directory belongs to an operator whose bot signs requests. A site that only receives automated traffic can consume supported edge-verification results without publishing its own operator keys.

For an outbound bot, assign ownership of signing, directory updates and emergency key withdrawal. Use keys managed for this purpose rather than reuse site TLS keys. Publish the directory with the service that uses it, then verify the signed request through the production proxy path.

Adoption checklist for a real bot operator

Before registration, confirm:

• the bot's purpose, product token, and operator identity are public and stable;
• the private key is generated and held in an approved key-management system;
• the HTTPS directory is signed, monitored, size-limited, and free of private key members;
• every required request component survives the production proxy path;
• created, expires, nonce handling, and business idempotency match the threat model;
• rotation and emergency revocation have been tested;
• verifier-specific syntax and algorithm limits are covered by integration tests.

Also document what successful authentication permits. Verified identity should feed an explicit allow, rate, or review policy. It shouldn't bypass authorization by default.

Test a valid identity making a refused request

Alongside a successful signed request, send one that exceeds the caller's rate or operation allowance. The signature should remain valid while access policy refuses the operation. Also test expired signatures, an unavailable directory and a withdrawn key.

Record which verifier profile each case exercises. Authentication, freshness and business permission need distinct results so operators can tell a key problem from a correctly denied request.

Primary references

• RFC 9421: HTTP Message Signatures
• Current Web Bot Auth working-group protocol draft
• Archived architecture draft and replacement history
• Cloudflare Web Bot Auth documentation