Web Bot Auth and HTTP Message Signatures

Web Bot Auth is about authenticating outbound automated requests. The bot operator generates a signing key, publishes the public key through a directory, and signs requests. A receiving site or its edge provider verifies the request against that published key.

This direction matters. An ordinary content origin doesn't become “Web Bot Auth ready” by placing an empty key directory on its domain. The directory belongs to an operator that sends signed bot traffic. TKOResearch receives automated traffic but doesn't currently operate a public outbound bot, so it doesn't publish that directory.

The mechanism combines a stable foundation with active draft work. RFC 9421 defines HTTP Message Signatures. The Web Bot Auth architecture and HTTP Message Signatures directory remain Internet-Drafts. Cloudflare's implementation also supports a documented subset and specific draft versions.

The parties and request flow

There are three practical roles:

A bot operator controls the automated client and its private signing key.
A key directory publisher serves the operator's public keys over HTTPS.
A verifier receives a request, locates the directory, validates the signature, and applies local policy.

The first two roles often belong to the same organization. The verifier may be the destination origin or a service such as Cloudflare in front of it.

Authentication answers whether the request is tied to a registered key under the claimed operator's control. It doesn't establish that the requested action is authorized. A valid bot can still exceed rate limits, access the wrong tenant, submit harmful input, or violate a content-use policy. Authorization and application controls remain separate.

RFC 9421 provides the signature machinery

HTTP Message Signatures defines how a signer selects message components, constructs Signature-Input, and carries signature bytes in Signature. Components can include HTTP fields and derived values such as @authority, @method, and @path.

A simplified shape looks like this:

Signature-Agent: "https://bot.example"
Signature-Input: sig1=("@authority" "signature-agent");created=1784217600;expires=1784217660;keyid="thumbprint";tag="web-bot-auth"
Signature: sig1=:base64-signature-bytes:

The exact signature base construction comes from RFC 9421. Implementers shouldn't concatenate fields by hand from an example. Structured Field parsing, component identifiers, parameter order, byte encoding, and algorithm behavior all affect verification.

Component selection defines what the signature protects. Signing @authority binds the request to its destination authority. Signing the signature-agent field binds the directory reference. Depending on the request, an operator may also bind method, path, query, or a content digest.

Intermediaries complicate the choice. A proxy may normalize or replace fields. If a signed component changes in transit, verification fails even when the request began as legitimate. Test through the complete production route.

The operator key directory

Cloudflare's Web Bot Auth documentation currently tells bot operators to publish a key directory at:

/.well-known/http-message-signatures-directory

The directory is served over HTTPS with a dedicated media type and a JSON Web Key Set containing public keys. Cloudflare requires the directory response itself to be signed. That proof prevents another party from copying a public key set to a different domain and claiming the operator identity during registration.

The private key never belongs in the directory. For an Ed25519 JWK, public members include kty, crv, and x; a private d member would disclose the signing secret. Key-generation and publication tooling should enforce that boundary.

The directory format comes from an active Internet-Draft. Cloudflare documents the draft revision its service accepts, including a structured-string form for Signature-Agent. Later draft syntax may differ. A conforming implementation for one verifier therefore needs version-aware tests, not a claim of universal compatibility.

Architecture maturity and implementation profile

The current Web Bot Auth architecture draft describes how HTTP Message Signatures can authenticate automated traffic. It remains work in progress even though RFC 9421 is published.

Cloudflare's July 2026 documentation is more specific than the general architecture. Its verified-bot integration supports Ed25519 keys and requires particular signature parameters. It lists RFC 9421 components and component parameters that its verifier doesn't support. It also documents registration through the Bot Submission Form before signed requests can be recognized as a verified bot.

This is a production vendor profile over standards and drafts. A bot operator targeting Cloudflare should implement the profile actually accepted by Cloudflare. An origin building its own verifier needs its own supported algorithms, directory-fetch policy, caching rules, failure responses, and interoperability suite.

Replay resistance needs an explicit policy

Signature validity alone doesn't guarantee that a request is fresh. created and expires parameters limit the accepted time window. A nonce can distinguish requests when the verifier tracks it.

Cloudflare recommends a short expires interval, often about a minute. Its current documentation says the nonce field can be included but Cloudflare doesn't yet validate nonce reuse against a database of seen values. That limitation should be stated in a threat model. A short validity period narrows replay exposure but doesn't provide single-use enforcement.

Sensitive actions may require stronger application-level controls. Use idempotency keys for mutation, bind signatures to method and target, validate request bodies where intermediaries won't modify them, and reject replays at the business-operation layer. Keep verifier clocks synchronized and define acceptable skew.

Key rotation also affects availability. Publish overlapping public keys long enough for caches and registrations to update. Remove compromised keys through a documented emergency path. Monitor directory retrieval failures and signature rejection rates.

Receiving signed bots at an origin

A site behind Cloudflare can use verified-bot outcomes in edge policy without implementing a second signature verifier. The origin should understand what identity information the edge forwards and which policy decisions depend on it.

If verification is performed at the origin, directory retrieval becomes a server-side request. Constrain HTTPS, redirects, DNS resolution, response size, media type, caching, and fetch timeouts. Defend against server-side request forgery and cache poisoning. Don't allow a request to choose an arbitrary internal directory URL.

Failure behavior should be predictable. Distinguish malformed signatures, unknown keys, expired requests, directory failures, and valid identities that lack authorization. Avoid logging private request bodies or credentials while still capturing enough metadata for technical assessment.

Why TKOResearch doesn't publish a directory

The current TKOResearch site publishes articles and receives contact submissions. It doesn't send a public crawler or agent whose requests need operator authentication.

An empty directory describes no key. Generating a key that never signs production traffic adds rotation and monitoring obligations without supporting a service. Site TLS keys are also unsuitable because they were issued and managed for a different protocol purpose.

The correct adoption trigger is an outbound bot product with a named operator, documented destinations, request policy, key custody, rotation, registration, observability, and incident handling. The directory and signed-request support should ship with that product.

Receiving authenticated bots is a separate choice. Cloudflare can verify supported requests at the edge. That capability doesn't require the destination site to claim it is a bot operator.

Adoption checklist for a real bot operator

Before registration, confirm:

the bot's purpose, product token, and operator identity are public and stable;
the private key is generated and held in an approved key-management system;
the HTTPS directory is signed, monitored, size-limited, and free of private key members;
every required request component survives the production proxy path;
created, expires, nonce handling, and business idempotency match the threat model;
rotation and emergency revocation have been tested;
verifier-specific syntax and algorithm limits are covered by integration tests.

Also document what successful authentication permits. Verified identity should feed an explicit allow, rate, or review policy. It shouldn't bypass authorization by default.

Adoption decision

Web Bot Auth offers a stronger operator identity mechanism than mutable IP ranges or self-declared user-agent strings. RFC 9421 gives it a rigorous signature base, while the directory and automated-traffic architecture continue to evolve through IETF drafts and vendor profiles.

TKOResearch won't add a key directory simply to satisfy a discovery check. If it launches an outbound automated client, the client team will own keys, signing, registration, rotation, monitoring, and the directory. Until then, absence is accurate.