Agent-Readable Web Files and Protocols: A Field Guide

A scanner can tell you that /.well-known/api-catalog is missing. It can't tell you whether your organization has an API worth publishing there. Start with the service inventory so you publish discovery metadata only for services someone maintains.

For a content site, the useful work is usually fairly small: publish readable material, describe its real representations, state crawler policy, and maintain a working security contact. An API, authorization server, agent endpoint, skill package, or checkout system brings a different set of commitments. Its discovery file belongs to that service's release.

For a publishing site, RSS and security links, robots rules, Content Signals, and an informal /llms.txt index can cover the relevant public resources. Verify account-level features such as Markdown conversion at the edge and from a client; application configuration alone doesn't establish what clients receive.

This guide covers 17 mechanisms, originally reviewed September 9 with the Cloudflare limits and current draft references checked again October 18, 2026. The table is an adoption map for a publishing origin, with the protocol roles kept separate. A business can operate several of those roles on different domains without making its marketing site an authorization server or merchant API.

The decision table

Classify the mechanism before checking the URL

The 17 items don't form one technical stack. They occupy different layers and have different owners.

Link headers, robots rules, and several OAuth documents have standards-track specifications. Those specifications define syntax and processing behavior, but they don't require every website to deploy the feature. RFC 9727 is a valid API discovery standard. A site without public APIs still has no catalog to publish.

DNS-AID, the current Web Bot Auth protocol, and MPP discovery are Internet-Drafts. An Internet-Draft is work in progress. It can change, expire, or be replaced. A team can experiment with one when the use case and maintenance capacity justify it, but production documentation should preserve that maturity label.

Other items are community or vendor work. Cloudflare Markdown for Agents is an account-level conversion feature. Content Signals is a publisher convention. The MCP, Agent Skills, WebMCP, x402, UCP, and ACP materials come from their respective project maintainers. Calling all of them “web standards” would erase meaningful differences in governance and interoperability.

Procurement and architecture review depend on this classification. An RFC gives implementers a stable reference, but it doesn't warrant that a product integration is safe. Vendor features can be ready for production inside one network and remain non-portable. Draft work needs an owner who can absorb incompatible revisions.

Give discovery the same owner as the service

If an API moves hosts, its catalog needs to change with it. If a signing key is withdrawn, directory caches become part of the incident response. Keep discovery in release-controlled configuration; clients may retain its URLs after the original service or owner has changed.

Consider an API catalog. RFC 9727 expects a publisher to list APIs and useful metadata, with a required Linkset JSON representation. The catalog needs monitoring and maintenance because consumers will treat its targets as intentional public surfaces. Pointing it at a contact form would misclassify a form-processing route as a supported developer interface. An empty catalog would be equally unhelpful.

The same reasoning applies to authorization metadata. An issuer value isn't a brand identifier. It identifies an authorization server and anchors exact endpoint relationships. Protected Resource Metadata binds a resource to the authorization servers that may issue usable credentials. Publishing invented URLs to satisfy a scanner can train clients to send credentials to the wrong place.

MCP cards, skill indexes, and commerce profiles have a similar dependency. A card presupposes a running MCP server. A skill index presupposes reviewed packages with owners and update procedures. A UCP profile presupposes a merchant operation. If the service disappears while its discovery file remains, automation keeps following a false promise.

Before adding discovery metadata, identify the live service, its owner, authentication policy, threat model, monitoring, and retirement process. Leave the metadata unpublished until those responsibilities are assigned.

What a content site can adopt accurately

An informational site still has useful machine-readable controls.

First, publish typed links to resources that actually exist. TKOResearch sends a Link response field for /feed.xml with rel="alternate" and for /.well-known/security.txt with rel="security". RFC 9116 registers the security relation and defines the security contact file. A client can verify both targets with ordinary GET or HEAD requests.

Second, make crawler policy explicit. The wildcard group remains the baseline for unlisted crawlers. Named groups document policy for operator tokens verified against current operator documentation. Because a named match doesn't inherit the wildcard group's rules under RFC 9309, every named group repeats /api/, /admin/, and /private/ exclusions. This keeps the named crawler policy consistent with the intended exclusions; it does not restrict application access.

Third, separate access from use. Robots rules concern whether a compliant crawler may fetch a path. The approved Content Signals policy declares permitted uses after content has been accessed. TKOResearch permits all three defined categories for its public material:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

That statement is sent as an origin response field and repeated in robots. It doesn't override authentication, authorization, copyright law, contracts, or protected-path rules.

Fourth, offer a lean index and representation. /llms.txt is an informal convention, not a standard. On this site it lists real public pages, articles, feeds, and policies from maintained application data. Cloudflare Markdown for Agents can return a text/markdown variant when a client sends Accept: text/markdown. Enabling it requires an account-level change. Document enablement, verification, and rollback, then check the response through the configured edge.

Security review by layer

Each layer fails differently. A single “present or missing” score hides these failure modes.

Publication controls

Robots and Content Signals are declarations. They depend on recipient behavior. Protected routes still need authentication and authorization, and sensitive material shouldn't be published on the assumption that a crawler will comply. Policy values should be version-controlled, reviewed, and tested at the deployed origin and edge.

Markdown conversion creates another representation of the same resource. Test caching with Vary: Accept, content type, canonical links, structured data handling and origin header preservation. Cloudflare documents a 6 MiB (6,291,456-byte) limit on decompressed HTML and HTML-only conversion. Its converter removes origin body validators such as ETag and Last-Modified; clients must not reuse them for the Markdown body. Rollback should be rehearsed before broad enablement.

Discovery controls

Link headers, DNS records, catalogs, metadata documents, cards, and indexes create routing decisions. Their targets need ownership checks and continuous availability checks. DNSSEC authenticates validated zone data. The DNS-AID draft's security discussion also warns that authentic metadata doesn't prove a discovered agent is safe. The same principle holds for a signed directory or a package digest.

Clients should constrain redirects, validate origins, reject insecure schemes where required, and treat downloaded descriptions as untrusted input. Publishers should plan rotation and retirement before launch. Stale discovery is a security and reliability defect because machines may act on it without a human noticing the mismatch.

Authentication and action controls

The September Web Bot Auth working-group draft succeeds the older architecture and directory work. Cloudflare still documents specific older syntax for its verifier. Likewise, experimental MCP discovery now uses /.well-known/ai-catalog.json, with a card embedded in data or fetched from url. These are reasons to pin source revisions, not reasons to publish new endpoints.

Web Bot Auth applies HTTP Message Signatures to outbound automated traffic. The bot operator creates keys, publishes the public directory, registers it where required, and signs requests. A content origin may verify those requests through its edge provider, but it doesn't publish an operator directory merely because it receives bots.

OAuth metadata describes trust relationships, not generic login instructions. MCP and WebMCP expose actions. Their review must cover scopes, per-tool authorization, user presence, parameter validation, tenant boundaries, rate limits, and revocation. Discovery is the beginning of that assessment, never the end.

Payment and commerce controls

Payment starts a contract between a request and a result. Before an origin can collect money, its operator needs a priced product, authoritative inventory or capacity, tax and refund policy, settlement handling, customer support, fraud controls, observability, and idempotent execution. None of the four protocols supplies those business systems.

Their technical boundaries differ. x402 places payment terms directly on an HTTP resource. MPP expresses payment through an HTTP authentication challenge and can advertise payable operations in OpenAPI. UCP negotiates a merchant's services, capabilities, transports, and payment handlers. ACP carries merchant-owned checkout state from product selection through order creation. Treating the paths as substitutes would promise incompatible behavior.

ACP needs one more maturity label. The official project is beta, and 2026-04-17 is its latest stable snapshot. That snapshot omits /.well-known/acp.json. The official repository now contains an Agentic Checkout Discovery document for that path, but its own header says Status: Proposal and Version: unreleased. A readiness check may observe the proposed path; it cannot call the path a released ACP conformance requirement.

Any payment or commerce launch should begin with the product's fulfillment record and system of record, then choose the protocol that matches the transaction. Discovery comes after the route, checkout, or merchant service works under production controls.

A practical implementation sequence

Start with a service inventory. List public representations, policies, APIs, authorization servers, protected resources, agents, tools, packages, and commerce surfaces. Mark the owner and production status of each.

Then map only live entries to their governing specifications. Verify whether the material is an RFC, an Internet-Draft, a project proposal, or a vendor feature. Use the specification's artifact name and processing rules rather than a checker's preferred shape when they differ.

For every proposed artifact, answer five operational questions:

1. What real service or policy does this describe?
2. Who owns accuracy, rotation, monitoring, and retirement?
3. What will a machine reasonably do after reading it?
4. Could the artifact expose a protected route, credential boundary, tool, or payment action?
5. How will deployment and rollback be verified outside the repository?

Finally, test the public behavior. Request headers with GET and HEAD, resolve every linked target, query DNS with validation when applicable, check variants with explicit Accept fields, and compare well-known metadata with the enforcing service. A repository file can be correct while a CDN, redirect, or cache serves something else.

Publish metadata that accurately describes the services you operate. Keep it limited to supported representations and endpoints. When an API, agent, tool, or commerce product launches, include discovery in that service's release, with an owner responsible for keeping it current.

Primary references

• RFC 8288: Web Linking
• RFC 9309: Robots Exclusion Protocol
• DNS-AID Internet-Draft
• Cloudflare Markdown for Agents
• Cloudflare Web Bot Auth
• Content Signals
• RFC 9727: API Catalog
• RFC 8414: Authorization Server Metadata
• RFC 9728: OAuth Protected Resource Metadata
• auth.md proposal
• MCP Server Cards current open draft, SEP-2127
• Agent Skills discovery proposal
• WebMCP Community Group repository
• x402 documentation
• Machine Payments Protocol
• Universal Commerce Protocol
• Agentic Commerce Protocol