robots.txt Rules for AI Crawlers
How RFC 9309 group matching affects AI crawler policy, protected paths, operator tokens, and post-access content-use declarations.
Adding a named AI crawler group to robots.txt can weaken a site's policy if the publisher assumes that the crawler also inherits the wildcard group. Under the Robots Exclusion Protocol, it doesn't.
Crawler names can distract from the group-selection rule that controls the result. A crawler selects the most specific matching user-agent group, while the wildcard group serves only as a fallback when no specific group matches. Repeat protected-path rules in every named group that should retain them.
TKOResearch publishes a wildcard group and explicit groups for current operator tokens that were verified against primary operator documentation. Every group allows public content and repeats exclusions for /api/, /admin/, and /private/.
RFC 9309 matching behavior
RFC 9309 standardizes the Robots Exclusion Protocol. A crawler uses a product token to find a case-insensitive match in the file's User-agent lines.
If several groups match the same product token, their rules are combined. If no named group matches, the crawler uses the * group when one exists. If neither a named nor wildcard group applies, no robots rules apply.
This file doesn't provide inheritance between different matches:
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /private/
User-agent: ExampleBot
Allow: /
ExampleBot matches its named group. It doesn't copy the three wildcard exclusions. If the intent is to allow public crawling while keeping those paths excluded, repeat them:
User-agent: ExampleBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Generate repeated groups from one protected-path list where possible. That prevents a later edit from updating the wildcard policy while leaving named groups behind.
Most-specific path rules
Within the selected rule set, the crawler finds the matching Allow or Disallow path with the greatest number of octets. The most specific match wins. If equally specific allow and disallow rules conflict, the RFC recommends the allow rule.
For example:
User-agent: ExampleBot
Disallow: /research/
Allow: /research/public/
/research/public/report matches both rules, but the longer allow path wins. /research/internal/report matches only the broader disallow path.
Rules start matching at the beginning of the URI path. Comparison is normally case-sensitive. Percent-encoding and non-ASCII paths have processing requirements, so test the actual encoded routes served by the application rather than assuming a visual match.
An empty Disallow value does not block the site. A missing applicable group also doesn't imply a block. Parsers and policy reviews need to distinguish these states.
Verify operator tokens before publishing them
A named group should use the product token documented by the crawler operator, not a company name guessed from a request string. Operator documentation also explains whether robots rules apply to that traffic class.
For the current policy, we checked sources including OpenAI crawler documentation, Anthropic crawler documentation, Google's crawler list, Perplexity's crawler documentation, and Apple's Applebot documentation.
This review produced useful exclusions. OpenAI describes ChatGPT-User as user-triggered traffic for which robots rules may not apply, so it isn't presented as a conventional crawler block. Perplexity says Perplexity-User generally ignores robots rules. We also omitted an unverified ByteDance token rather than replacing it with a guess.
Some operator tokens, such as Google-Extended and Applebot-Extended, are robots control tokens without a standalone fetching user-agent string. That is valid when the operator explicitly documents the behavior. The publisher still needs to understand what the token controls.
Operator names change. Recheck primary documentation on a schedule and remove retired tokens. The wildcard group remains the baseline for unlisted compliant crawlers.
Access policy is not use policy
Robots rules state whether a compliant crawler may access URI paths. They don't authenticate a client, protect a secret, or define every use permitted after access.
Content Signals addresses a separate question: how a publisher wants accessed content used for AI training, search, or AI input. TKOResearch permits all three categories for public content, so its groups include:
Content-Signal: ai-train=yes, search=yes, ai-input=yes
This extension is outside the core Allow and Disallow processing defined by RFC 9309. Systems that understand it can process the declared policy; other robots parsers may ignore the line. The declaration doesn't change which path rule wins.
Access and use can therefore be reviewed independently. A public article can be allowed for fetching and permitted for all three declared uses. An API path can remain excluded from crawler access even though the site's public-content policy is permissive. Authentication and authorization continue to govern any protected route regardless of the file.
robots.txt isn't a security boundary
The file is public and advisory. A harmful client can ignore it. Listing a private path also discloses the path name.
Keep secrets behind authorization. Return appropriate status codes, validate sessions and tokens, rate-limit expensive operations, and avoid placing sensitive material in public static output. Use edge controls and application policy for enforcement.
The file itself is untrusted input to crawlers. RFC 9309 sets a minimum parsing limit and requires UTF-8. Clients should cap processing and handle redirects carefully. Publishers should serve text/plain at the exact origin-root /robots.txt path.
A maintainable implementation
TKOResearch uses an explicit Next.js text route and one renderer for all groups. The relevant structure is:
const protectedPaths = ['/api/', '/admin/', '/private/'];
function renderRules(userAgent: string) {
return [
`User-agent: ${userAgent}`,
'Allow: /',
...protectedPaths.map((path) => `Disallow: ${path}`),
'Content-Signal: ai-train=yes, search=yes, ai-input=yes',
];
}
The wildcard and verified named tokens all pass through the same function. Host and Sitemap lines are added once after the groups. While those lines can be useful to specific crawlers, they aren't part of RFC 9309's core access rules.
An explicit route is preferable here because the policy contains extension lines and repeated named groups. It also lets tests request the exact production body rather than infer behavior from framework metadata.
Verification checklist
Request the public file and inspect it as plain text:
curl -fsS https://www.tkoresearch.com/robots.txt
For each named token, verify all protected exclusions are present. Confirm the wildcard group remains first and complete. Check the sitemap URL and production host. Then test representative paths with a parser that implements RFC 9309 matching.
Don't stop at a repository snapshot. Confirm the edge isn't serving a managed or cached robots file over the application route. Cloudflare and other platforms can offer managed robots features; only one maintained source should be authoritative.
Review the file when a route, crawler policy, or operator token changes. A new named group is a policy change because it can alter which rules a crawler selects.
Adoption decision
Explicit AI crawler groups can make policy easier to read, but correctness comes from RFC 9309 matching and complete path rules. The named list isn't a substitute for wildcard coverage, and robots.txt isn't access control.
TKOResearch allows public crawling, repeats its protected paths for each verified named token, and declares an all-yes post-access policy for public content. That is accurate for the current site. If a future route must never be crawled, it will be protected in the application first and documented in every applicable robots group second.