Agent-Readable Web Files and Protocols Field Guide

An agent-readiness scan can produce a long list of missing files. That list is useful only after one question has been answered: what service does each file describe?

Discovery metadata must follow a real service. It must never be published to imply a capability that the organization hasn't built, secured, staffed, and committed to operate. A missing optional discovery feature isn't automatically a site defect. Sometimes absence is the accurate result.

That rule shaped our review of 17 mechanisms suggested for TKOResearch. We implemented controls that describe the public site as it exists: links to its RSS feed and security contact, a crawler access policy, a content-use policy, an informal /llms.txt index, and an operations path for Markdown representation at the edge. We didn't advertise an API catalog, authorization server, MCP server, agent skill, browser tool, or payment service. TKOResearch doesn't currently operate those public services.

This field guide separates established standards from active drafts and vendor proposals. It also gives an adoption decision for each mechanism. The decisions are specific to the present TKOResearch site, not a universal scorecard.

The decision table

Feature	Purpose	Artifact or endpoint	Maturity	Appropriate adopters	Main security concern	TKOResearch decision
Link headers	Attach typed relationships to an HTTP response	`Link` response field	Standards Track RFC 8288	Sites with real alternate representations, policies, or related resources	A stale or false link can misdirect automated clients	Publish links to the existing RSS feed and RFC 9116 security contact
DNS-AID	Discover an organization's agent endpoints through DNS	SVCB records beneath proposed agent names	Active Internet-Draft, built on RFC 9460	Operators of reachable, maintained agent services	DNSSEC authenticates zone data, not the behavior of the advertised agent	Do not publish because no public agent endpoint exists
Markdown content negotiation	Return a Markdown representation when a client asks for it	`Accept: text/markdown` and a variant response	Cloudflare product feature, not an IETF standard	Content sites using a supported Cloudflare plan	Conversion, caching, and policy headers must remain correct across variants	Prepare Cloudflare enablement and rollback; keep the origin policy authoritative
AI crawler rules	Express crawler access preferences by product token and path	`/robots.txt`	Standards Track RFC 9309, with operator-specific tokens	Public sites that want documented automated access boundaries	A named group can accidentally lose wildcard exclusions	Publish explicit groups that repeat protected paths and retain wildcard coverage
Web Bot Auth	Authenticate outbound automated HTTP requests	Signed requests plus an operator key directory	RFC 9421 foundation with active architecture and directory drafts	Bot operators and sites that verify their traffic	Key ownership, replay resistance, rotation, and intermediary changes	Do not publish a key directory because TKOResearch isn't operating an outbound bot
Content Signals	State permitted post-access uses for public content	`Content-Signal` response field and robots directive	Publisher convention implemented by Cloudflare	Publishers prepared to declare training, search, and AI-input preferences	Enforcement depends on recipients honoring the declaration	Publish `ai-train=yes, search=yes, ai-input=yes` at the origin and in robots
API Catalog	Give machines a maintained index of public APIs	`/.well-known/api-catalog` with Linkset JSON	Standards Track RFC 9727, using RFC 9264	Organizations that publish supported developer APIs	An obsolete catalog exposes retired surfaces and wrong documentation	Do not publish; the contact form isn't a developer API
OAuth and OpenID Connect discovery	Publish authorization-server or identity-provider metadata	RFC 8414 or OpenID well-known metadata	Standards Track RFC 8414 and final OpenID Connect Discovery	Operators of real OAuth authorization servers or OpenID Providers	Invented issuers and endpoints create severe trust failures	Do not publish because this domain isn't an authorization server or OpenID Provider
OAuth Protected Resource Metadata	Tell clients which authorization servers protect a resource	`/.well-known/oauth-protected-resource` variants	Standards Track RFC 9728	OAuth-protected APIs and MCP resources	Resource identifiers and authorization-server bindings must match enforcement	Do not publish because there is no public OAuth-protected resource
`auth.md`	Give agents human-readable registration and authorization instructions	Proposed `auth.md` document	WorkOS open proposal, not an IETF standard	Services with an implemented OAuth flow that needs supplementary guidance	Narrative instructions can become stale or conflict with machine metadata	Do not publish until a real authorization service exists
MCP Server Cards	Describe an MCP server before a client connects	Candidate well-known card document	Current open draft SEP-2127, not a settled standard	Operators of maintained remote MCP servers	Static capability claims can be stale, overbroad, or trusted without verification	Do not publish because TKOResearch has no public MCP server
Agent Skills discovery	Index downloadable agent skill artifacts	Proposed `/.well-known/agent-skills/` index	Cloudflare v0.2.0 proposal, not an IETF standard	Publishers that maintain reviewed, versioned skill packages	A digest checks bytes, not whether instructions are safe	Do not publish an empty index or unowned packages
WebMCP	Let a page register tools for browser-mediated agent use	APIs under `document.modelContext`	Web Machine Learning Community Group draft work	Web applications with narrow, user-visible actions	Tool invocation can cross user intent and page authorization boundaries	Do not register tools on the current informational site
x402	Request payment before serving an HTTP resource	HTTP 402 plus x402 payment fields	x402 Foundation specification, an industry protocol rather than an IETF standard	APIs with priced resources, wallets, and settlement operations	Replay, settlement finality, refunds, and idempotent fulfillment	Do not add payment middleware because no priced machine resource exists
Machine Payments Protocol	Carry payment terms through HTTP authentication	`Payment` authentication scheme and payment metadata	Protocol documentation plus an Internet-Draft whose draft text is dated July 11, 2026	Services with explicit machine pricing and controlled fulfillment	Payment authorization must bind to the exact request and fulfillment state	Do not implement because the site has no machine-purchasable product
Universal Commerce Protocol	Negotiate merchant services, checkout, and payment handlers	`/.well-known/ucp` business profile and transport bindings	UCP specification, an active industry specification	Merchants with operational product, checkout, payment, and support systems	Merchant-of-record duties and signed state transitions must remain coherent	Do not publish a business profile for a consulting contact form
Agentic Commerce Protocol	Let agents manage merchant-owned checkout sessions	ACP checkout API and product feeds	OpenAI and Stripe beta specification; the latest stable ACP snapshot, `2026-04-17`, omits `/.well-known/acp.json`, while the official repository carries discovery as an unreleased proposal	Merchants accepted into the beta with production commerce operations	Delegated payment tokens, idempotency, and merchant-owned state require strict controls	Do not implement; an unreleased discovery proposal cannot describe a checkout service that doesn't exist

Classify the mechanism before checking the URL

The 17 items don't form one technical stack. They occupy different layers and have different owners.

Link headers, robots rules, and several OAuth documents have standards-track specifications. Those specifications define syntax and processing behavior, but they don't require every website to deploy the feature. RFC 9727 is a valid API discovery standard. A site without public APIs still has no catalog to publish.

DNS-AID, the Web Bot Auth architecture, and MPP discovery are Internet-Drafts. An Internet-Draft is work in progress. It can change, expire, or be replaced. A team can experiment with one when the use case and maintenance capacity justify it, but production documentation should preserve that maturity label.

Other items are community or vendor work. Cloudflare Markdown for Agents is an account-level conversion feature. Content Signals is a publisher convention. The MCP, Agent Skills, WebMCP, x402, UCP, and ACP materials come from their respective project maintainers. Calling all of them “web standards” would erase meaningful differences in governance and interoperability.

Procurement and architecture review depend on this classification. An RFC gives implementers a stable reference, but it doesn't warrant that a product integration is safe. Vendor features can be ready for production inside one network and remain non-portable. Draft work needs an owner who can absorb incompatible revisions.

Discovery is a pointer, not a capability

A discovery artifact reduces the cost of finding something. It doesn't create the thing.

Consider an API catalog. RFC 9727 expects a publisher to list APIs and useful metadata, with a required Linkset JSON representation. The catalog needs monitoring and maintenance because consumers will treat its targets as intentional public surfaces. Pointing it at a contact form would misclassify a form-processing route as a supported developer interface. An empty catalog would be equally unhelpful.

The same reasoning applies to authorization metadata. An issuer value isn't a brand identifier. It identifies an authorization server and anchors exact endpoint relationships. Protected Resource Metadata binds a resource to the authorization servers that may issue usable credentials. Publishing invented URLs to satisfy a scanner can train clients to send credentials to the wrong place.

MCP cards, skill indexes, and commerce profiles have a similar dependency. A card presupposes a running MCP server. A skill index presupposes reviewed packages with owners and update procedures. A UCP profile presupposes a merchant operation. If the service disappears while its discovery file remains, automation keeps following a false promise.

Our adoption gate is therefore simple: name the live service, its owner, its authentication policy, its threat model, its monitoring, and its retirement process before adding discovery metadata. If those facts can't be supplied, the correct state is absence.

What a content site can adopt accurately

An informational site still has useful machine-readable controls.

First, publish typed links to resources that actually exist. TKOResearch sends a Link response field for /feed.xml with rel="alternate" and for /.well-known/security.txt with rel="security". RFC 9116 registers the security relation and defines the security contact file. A client can verify both targets with ordinary GET or HEAD requests.

Second, make crawler policy explicit. The wildcard group remains the baseline for unlisted crawlers. Named groups document policy for operator tokens verified against current operator documentation. Because a named match doesn't inherit the wildcard group's rules under RFC 9309, every named group repeats /api/, /admin/, and /private/ exclusions. This prevents a documentation block from weakening access boundaries.

Third, separate access from use. Robots rules concern whether a compliant crawler may fetch a path. The approved Content Signals policy declares permitted uses after content has been accessed. TKOResearch permits all three defined categories for its public material:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

That statement is sent as an origin response field and repeated in robots. It doesn't override authentication, authorization, copyright law, contracts, or protected-path rules.

Fourth, offer a lean index and representation. /llms.txt is an informal convention, not a standard. On this site it lists real public pages, articles, feeds, and policies from maintained application data. Cloudflare Markdown for Agents can return a text/markdown variant when a client sends Accept: text/markdown. Enabling that feature belongs in edge operations, so the repository holds a runbook with enablement, verification, and rollback instructions rather than pretending a code commit changed the Cloudflare account.

Security review by layer

Each layer fails differently. A single “present or missing” score hides these failure modes.

Publication controls

Robots and Content Signals are declarations. They depend on recipient behavior. Protected routes still need authentication and authorization, and sensitive material shouldn't be published on the assumption that a crawler will comply. Policy values should be version-controlled, reviewed, and tested at the deployed origin and edge.

Markdown conversion creates another representation of the same resource. Test caching with Vary: Accept, content type, canonical links, structured data handling, token-count fields, and origin header preservation. Cloudflare documents a 2 MB origin-response limit and HTML-only conversion. Rollback should be rehearsed before broad enablement.

Discovery controls

Link headers, DNS records, catalogs, metadata documents, cards, and indexes create routing decisions. Their targets need ownership checks and continuous availability checks. DNSSEC can authenticate that a DNS record came from a validated zone, but the DNS-AID draft correctly warns that authentic metadata doesn't prove a discovered agent is safe. The same principle holds for a signed directory or a package digest.

Clients should constrain redirects, validate origins, reject insecure schemes where required, and treat downloaded descriptions as untrusted input. Publishers should plan rotation and retirement before launch. Stale discovery is a security and reliability defect because machines may act on it without a human noticing the mismatch.

Authentication and action controls

Web Bot Auth applies HTTP Message Signatures to outbound automated traffic. The bot operator creates keys, publishes the public directory, registers it where required, and signs requests. A content origin may verify those requests through its edge provider, but it doesn't publish an operator directory merely because it receives bots.

OAuth metadata describes trust relationships, not generic login instructions. MCP and WebMCP expose actions. Their review must cover scopes, per-tool authorization, user presence, parameter validation, tenant boundaries, rate limits, and revocation. Discovery is the beginning of that assessment, never the end.

Payment and commerce controls

Payment starts a contract between a request and a result. Before an origin can collect money, its operator needs a priced product, authoritative inventory or capacity, tax and refund policy, settlement handling, customer support, fraud controls, observability, and idempotent execution. None of the four protocols supplies those business systems.

Their technical boundaries differ. x402 places payment terms directly on an HTTP resource. MPP expresses payment through an HTTP authentication challenge and can advertise payable operations in OpenAPI. UCP negotiates a merchant's services, capabilities, transports, and payment handlers. ACP carries merchant-owned checkout state from product selection through order creation. Treating the paths as substitutes would promise incompatible behavior.

ACP needs one more maturity label. The official project is beta, and 2026-04-17 is its latest stable snapshot. That snapshot omits /.well-known/acp.json. The official repository now contains an Agentic Checkout Discovery document for that path, but its own header says Status: Proposal and Version: unreleased. A readiness check may observe the proposed path; it cannot call the path a released ACP conformance requirement.

Any payment or commerce launch should begin with the product's fulfillment record and system of record, then choose the protocol that matches the transaction. Discovery comes after the route, checkout, or merchant service works under production controls.

A practical implementation sequence

Start with an inventory, not a scanner result. List public representations, policies, APIs, authorization servers, protected resources, agents, tools, packages, and commerce surfaces. Mark the owner and production status of each.

Then map only live entries to their governing specifications. Verify whether the material is an RFC, an Internet-Draft, a project proposal, or a vendor feature. Use the specification's artifact name and processing rules rather than a checker's preferred shape when they differ.

For every proposed artifact, answer five operational questions:

What real service or policy does this describe?
Who owns accuracy, rotation, monitoring, and retirement?
What will a machine reasonably do after reading it?
Could the artifact expose a protected route, credential boundary, tool, or payment action?
How will deployment and rollback be verified outside the repository?

Finally, test the public behavior. Request headers with GET and HEAD, resolve every linked target, query DNS with validation when applicable, check variants with explicit Accept fields, and compare well-known metadata with the enforcing service. A repository file can be correct while a CDN, redirect, or cache serves something else.

Collecting all 17 checks would misrepresent the current site. The sound outcome is the smallest accurate machine-readable description of services the organization can defend and operate. For TKOResearch today, that means strong publication controls and honest non-implementation of service metadata. When a real API, agent, tool, or commerce product launches, its discovery work should ship as part of that service, with the same engineering owner and release gate.

Agent-Readable Web Files and Protocols: A Field Guide