Content Signals for AI Training, Search, and AI Input

Content Signals lets a publisher declare three preferences for automated use of accessed content: AI training, search, and AI input. Each category takes yes or no, producing a compact policy such as:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

TKOResearch uses that all-yes policy for public content. The directive is sent as an origin response field and repeated in each robots group. It doesn't grant access to protected paths and it doesn't replace authentication, authorization, contract terms, or applicable law.

Content Signals is a Cloudflare-led publisher mechanism documented at contentsignals.org. It isn't an IETF standard, and recipients may ignore it. Accuracy still matters because systems that honor the declaration will use it to make post-access decisions.

The three defined values

Training and fine-tuning AI models fall under ai-train. The publisher uses yes to permit that category and no to state that it isn't permitted.

For search, the defined uses are building a search index and providing results such as hyperlinks and short excerpts. AI-generated search summaries are outside this category. A publisher can therefore allow traditional search indexing while selecting a different value for ai-input.

Agents that place current public material into a model as task input rely most directly on ai-input. Its examples include retrieval-augmented generation, grounding, and real-time generative answers.

The values are independent. Examples include:

Content-Signal: ai-train=no, search=yes, ai-input=no

and:

Content-Signal: ai-train=no, search=yes, ai-input=yes

Don't infer one category from another. Permission for search doesn't imply permission for AI input, and permission for AI input doesn't necessarily state a training preference. Publish all three so a recipient doesn't have to guess what an omitted category means.

Access and use are separate controls

The Robots Exclusion Protocol decides which Allow and Disallow path rule applies to a compliant crawler. Content Signals describes preferred use after material is accessed.

Consider this group:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Content-Signal: ai-train=yes, search=yes, ai-input=yes

The three disallow lines retain the crawler access boundary. The Content-Signal line declares permitted uses for content that can be accessed. It doesn't make /private/ public, and it doesn't authorize a client to call /api/.

Named robots groups need the same protected-path repetition because a named match doesn't inherit wildcard rules. The TKOResearch route generates every group from one path list and includes the same approved Content Signals directive. This keeps access and post-access policy consistent without conflating them.

Why the response field matters

A robots file is site-level and path-oriented. A response field travels with the selected representation. It is visible to a client fetching a page directly, including a client that didn't begin with /robots.txt.

Origin publication also supports edge conversion. Cloudflare Markdown for Agents can turn an HTML response into Markdown when a client asks for text/markdown. The converted representation should carry the publisher's chosen policy rather than acquire a different value at the edge.

On July 13, 2026, Cloudflare changed Markdown for Agents header handling. If the origin sends Content-Signal, Cloudflare now preserves it on the converted response. If the origin omits the field, Cloudflare adds the default:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

The origin is therefore authoritative when it declares a policy. This is preferable even when the selected value matches Cloudflare's default. It records intent in application configuration and prevents a missing origin field from becoming an accidental policy dependency.

Implement the declaration once, verify it twice

In Next.js, a site-wide origin field can be attached through the global header configuration:

{
  key: 'Content-Signal',
  value: 'ai-train=yes, search=yes, ai-input=yes',
}

The robots route uses the same literal directive. Two publication locations create a consistency obligation, so repository verification should check both exact values.

After deployment, inspect an ordinary HTML response:

curl -sSI https://www.tkoresearch.com/

If Markdown for Agents is enabled, inspect the negotiated response too:

curl -sS -D - https://www.tkoresearch.com/ \
  -H 'Accept: text/markdown' \
  -o /tmp/tkoresearch-home.md

Both responses should contain the approved value. Request /robots.txt and count every group. Each named group should retain the protected paths and the same declaration.

Test error and private responses separately. A global field may appear on responses that contain no public article content. That doesn't convert an error body or authorization challenge into reusable public material. If the application later serves content with a different policy, it will need a narrower header scope and a documented precedence rule.

Policy design before syntax

The technical line is easy. The organizational decision is the real work.

Identify which public materials the organization controls, which third-party materials appear under license, and whether page-level exceptions exist. Confirm the policy with content, legal, product, and engineering owners. Record an effective date and a review cadence.

For an all-yes policy, confirm the organization intends to permit all three defined categories. Don't choose the value because it is a vendor default. For a restrictive policy, confirm that the robots directive, origin field, CDN configuration, and any page-specific headers agree.

Keep the declaration plain. Extra prose in a machine field can make it invalid. Put explanatory material in a public policy page and link to it through normal site navigation if users need more context.

Limits and enforcement

Content Signals states a preference. The project documentation notes that not all automated systems honor robots files or the directive. A publisher should not treat the line as a technical enforcement boundary.

A declaration of ai-train=no can't protect a published secret. The same is true of ai-input=no in front of an exposed licensed database. Keep non-public data behind authentication, authorization, suitable network controls, and data minimization.

Compliance monitoring can still be useful. Record declared values, crawl activity, takedown channels, and policy changes. If a recipient offers operator controls, configure them consistently. Keep claims sober: the declaration communicates terms to systems that choose to process it.

Change and rollback

A policy change should be atomic across origin headers and robots output. Update both in one release, run exact-string checks, build the application, and verify public responses after caches refresh.

Cloudflare conversion adds one more check. Because the origin field is preserved, a changed origin value should appear on Markdown variants. If it doesn't, disable conversion or correct the edge configuration before relying on the new publication state.

Rollback means restoring the previous approved value in both locations. Preserve the decision record so readers can tell whether a difference came from policy, deployment delay, or stale cache.

TKOResearch decision

TKOResearch permits AI training, search, and AI input for its public site content under the current Content Signals categories. The implementation is explicit:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Public crawler access remains bounded by path exclusions. Private and administrative routes remain protected by application controls. Cloudflare preserves the origin value on Markdown conversions, so the same policy follows both HTML and Markdown representations.

This is an accurate policy declaration, not a claim that every automated system will comply. It will be reviewed as the site's content, licensing, and machine-access features change.