Content Signals for AI Training, Search, and AI Input

A publisher can permit search indexing without making the same choice about model training. Content Signals puts those choices into machine-readable fields. The part I would review first is the material covered by the declaration: an all-yes value should be an intentional publishing decision, not something inherited accidentally from an edge service.

TKOResearch's application configures this value for its public content and repeats it in its robots output:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

The following sections cover those three purpose fields and the newer use field now documented for Cloudflare's managed robots feature. Content Signals is a publisher convention, not an IETF standard. A recipient can ignore it, and it cannot grant access to a resource the requester is otherwise forbidden to read.

The three purpose fields in the site policy

The Content Signals definitions place training and fine-tuning AI models under ai-train. The publisher uses yes to permit that category and no to state that it isn't permitted.

For search, the defined uses are building a search index and providing results such as hyperlinks and short excerpts. AI-generated search summaries are outside this category. A publisher can therefore allow traditional search indexing while selecting a different value for ai-input.

Agents that place current public material into a model as task input rely most directly on ai-input. Its examples include retrieval-augmented generation, grounding, and real-time generative answers.

The values are independent. Examples include:

Content-Signal: ai-train=no, search=yes, ai-input=no

and:

Content-Signal: ai-train=no, search=yes, ai-input=yes

Keep the purposes separate. Search permission doesn't settle the publisher's training choice. In a change review, state the intended value for each field and check the project's treatment of omitted values instead of silently filling them from a vendor default.

As of September 9, 2026, Cloudflare's managed robots documentation also describes use=immediate, use=reference, and use=full for retention and reuse. Its managed default includes use=reference. Those values are a separate dimension from the three yes/no purposes. This article records the site's existing three-field declaration; it doesn't silently add the managed default or authorize a policy change.

Access and use are separate controls

The Robots Exclusion Protocol decides which Allow and Disallow path rule applies to a compliant crawler. Content Signals describes preferred use after material is accessed.

Consider this group:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Content-Signal: ai-train=yes, search=yes, ai-input=yes

The three disallow lines retain the crawler access boundary. The Content-Signal line declares permitted uses for content that can be accessed. It doesn't make /private/ public, and it doesn't authorize a client to call /api/.

Named robots groups need the same protected-path repetition because a named match doesn't inherit wildcard rules. The TKOResearch route generates every group from one path list and includes the same approved Content Signals directive. This keeps access and post-access policy consistent without conflating them.

Why the response field matters

A robots file is site-level and path-oriented. A response field travels with the selected representation. It is visible to a client fetching a page directly, including a client that didn't begin with /robots.txt.

Origin publication also supports edge conversion. Cloudflare Markdown for Agents can turn an HTML response into Markdown when a client asks for text/markdown. The converted representation should carry the publisher's chosen policy rather than acquire a different value at the edge.

On July 13, 2026, Cloudflare changed Markdown for Agents header handling. If the origin sends Content-Signal, Cloudflare now preserves it on the converted response. If the origin omits the field, Cloudflare adds the default:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

The origin is therefore authoritative when it declares a policy. This is preferable even when the selected value matches Cloudflare's default. It records intent in application configuration and prevents a missing origin field from becoming an accidental policy dependency.

Implement the declaration once, verify it twice

In Next.js, a site-wide origin field can be attached through the global header configuration:

{
  key: 'Content-Signal',
  value: 'ai-train=yes, search=yes, ai-input=yes',
}

The robots route uses the same literal directive. Two publication locations create a consistency obligation, so repository verification should check both exact values.

After deployment, inspect an ordinary HTML response:

curl -sSI https://www.tkoresearch.com/

If Markdown for Agents is enabled, inspect the negotiated response too:

curl -sS -D - https://www.tkoresearch.com/ \
  -H 'Accept: text/markdown' \
  -o /tmp/tkoresearch-home.md

Both responses should contain the approved value. Request /robots.txt and count every group. Each named group should retain the protected paths and the same declaration.

Test error and private responses separately. A global field may appear on responses that contain no public article content. That doesn't convert an error body or authorization challenge into reusable public material. If the application later serves content with a different policy, it will need a narrower header scope and a documented precedence rule.

Policy design before syntax

The technical line is easy. The organizational decision is the real work.

Identify which public materials the organization controls, which third-party materials appear under license, and whether page-level exceptions exist. Confirm the policy with content, legal, product, and engineering owners. Record an effective date and a review cadence.

For an all-yes policy, confirm the organization intends to permit all three defined categories. Don't choose the value because it is a vendor default. For a restrictive policy, confirm that the robots directive, origin field, CDN configuration, and any page-specific headers agree.

Keep the declaration plain. Extra prose in a machine field can make it invalid. Put explanatory material in a public policy page and link to it through normal site navigation if users need more context.

Limits and enforcement

Content Signals states a preference. The project documentation notes that not all automated systems honor robots files or the directive. A publisher should not treat the line as a technical enforcement boundary.

A declaration of ai-train=no can't protect a published secret. The same is true of ai-input=no in front of an exposed licensed database. Keep non-public data behind authentication, authorization, suitable network controls, and data minimization.

Compliance monitoring can still be useful. Record declared values, crawl activity, takedown channels, and policy changes. If a recipient offers operator controls, configure them consistently. Keep claims sober: the declaration communicates terms to systems that choose to process it.

Change and rollback

A policy change should be atomic across origin headers and robots output. Update both in one release, run exact-string checks, build the application, and verify public responses after caches refresh.

Cloudflare conversion adds one more check. Because the origin field is preserved, a changed origin value should appear on Markdown variants. If it doesn't, disable conversion or correct the edge configuration before relying on the new publication state.

Rollback means restoring the previous approved value in both locations. Preserve the decision record so readers can tell whether a difference came from policy, deployment delay, or stale cache.

TKOResearch decision

TKOResearch permits AI training, search, and AI input for its public site content under the current Content Signals categories. The implementation is explicit:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Public crawler access remains bounded by path exclusions. Any private or administrative material still requires application access controls; publishing path exclusions doesn't establish that those controls exist. Cloudflare preserves the origin value on Markdown conversions, so the same policy follows both HTML and Markdown representations.

The next review should compare the declared policy with the actual public materials, then check the response at the origin and edge. If third-party content or a new managed setting changes that picture, resolve the policy choice before treating the header as current.