How to Read the Scope of an AI Agent Security Assessment
A client review method for connecting an AI agent assessment to the permissions and release decision it actually covers.
The most consequential sentence in an agent security report may be its exclusion of background workers. If your agent sends approved messages through a queue, that exclusion removes the component that turns a proposed action into an external communication. A successful test of the chat interface doesn't settle the release decision.
Read the scope before the severity chart. You need to know what the assessor was allowed to examine, what configuration ran, and which claims the resulting observations support. A narrow assessment can be useful when the boundary is clear. Trouble begins when a narrow result becomes permission for a wider deployment.
This is a client-side reading method for an AI agent security assessment. My KevinBytes analysis of assurance claims compares different objects of assurance. I write both publications and am affiliated with TKOResearch; that analysis should not be treated as an independent endorsement of TKO's programs.
Put the deployment beside the scope
Ask for a configuration manifest you can compare with the intended release. Include application commit or release ID, model identifier, hosted model evaluation date where immutable versions aren't available, system prompt revision, tool schemas, connector versions, retrieval configuration, memory policy, network routes and credential permissions. Preserve configuration references without copying secret values into the report.
Model names alone are insufficient. The same model connected to a search index and a mailbox has different authority from an isolated chat interface. A tool schema change can expose a new parameter while the application retains the same public product name.
Record differences between the test and release environments explicitly. A simulated mail service can establish that the dispatcher enforces recipient approval against the fixture. It cannot establish that the production connector uses the intended account, honors the same restrictions, or reports actual delivery correctly.
NIST SP 800-115, final September 2008, provides general guidance on planning technical security assessments and understanding technique limitations. The agent-specific manifest and examples here are my application of that planning discipline, not a NIST agent certification scheme.
Follow an actual workflow
Consider an invented support assistant for tenant Cedar. It reads assigned tickets, drafts responses, and queues a message only after an authorized employee approves the recipient and content. It cannot administer users or send unrelated attachments.
The assessment scope should connect the workflow to explicit tests and observable outcomes:
| Workflow step | What belongs in scope | Material needed to interpret the result |
|---|---|---|
| Read ticket 42 | Authenticated employee, Cedar tenant, ticket assignment and attachment access | Resource authorization decision and returned fixture identifiers |
| Draft response | Retrieved content, model context and draft storage | Source references, draft version and ownership |
| Approve draft 7 | Reviewer role, recipient, content version and expiry | Approval record bound to the exact draft |
| Queue and send | Worker identity, current authorization, retries and connector | Execution request, receipt and resulting message state |
| Revoke employee access | Running session, queued job and cached policy | Revocation time and measured behavior of pending work |
Now read an exclusion such as “third-party integrations are outside scope.” Does it exclude the mail provider's internal infrastructure, which may be sensible, or every call through the connector? Those boundaries have different consequences. Ask the assessor to name the interface where visibility ends and the customer checks required beyond it.
A report should also distinguish a missing path from an inaccessible one. “Queue inspected, execution unavailable in the supplied environment” describes an untested runtime case. “Queue omitted from the supplied architecture” identifies a scope discovery problem. Neither is a pass.
Check authorization for the assessment itself
The rules of engagement should name the systems, accounts, tenant fixtures, time window, allowed techniques, stop conditions and responsible contacts. Customer ownership of an application doesn't automatically authorize disruptive activity against every provider it calls.
Prefer inert recipients and simulated effects where they answer the question. If actual sending is required, define the dedicated mailbox, permitted messages, recipient ownership and cleanup before execution. The scope should explain which claims require that additional test, rather than quietly using a live integration because it was available.
Data handling belongs here too. Agree which source files, traces and reports leave the environment, who can access them, and their retention and deletion process. A debugging transcript can contain private content even when no credentials are present. The assessment should collect enough to reproduce a finding without treating every prompt as a necessary deliverable.
Read results as bounded statements
“Prompt injection blocked” is hard to use. “The dispatcher rejected a send request for an unapproved recipient under policy revision 12; no connector call was observed in the instrumented path” identifies a control and the limit of observation.
Keep model refusal, policy rejection, downstream rejection, successful authorized action and unknown effect separate. A timeout after dispatch is not a demonstrated denial. If the assessor cannot reconcile it against downstream state, the result remains unknown.
Inspect ordinary allowed cases alongside negative cases. An assistant that denies every request may preserve one security property while failing its intended workflow. Ask whether authorized tasks completed and whether any failures were omitted from a headline score. Repeated model trials should disclose their number, inputs and configuration; a percentage without that denominator hides how little may have been exercised.
For AgentBoundary, request the applicable suite revision, scoped configuration, exceptions and publication terms just as you would for another technical program. A program label does not expand the underlying observations.
Turn the report into a release decision
The client owns acceptance of its deployment risk. The assessor should supply a clear recommendation, reproducible findings, tested fixes and remaining limitations. Keep “fixed in source,” “retested in staging” and “verified on the released configuration” as separate states.
In the Cedar example, a reasonable decision could permit ticket reading and draft preparation while withholding connector credentials until queue authorization and revocation are tested. That is a specific restriction an engineer can implement and an owner can check.
Assign each blocker an owner and a retest condition. Identify changes that reopen acceptance: a new tool, broader credential, altered retrieval boundary, changed approval semantics or replacement worker. Cosmetic copy changes need not invalidate the whole report; authority changes deserve deliberate review.
Before accepting the deliverable, make sure another engineer can find the tested configuration, replay the relevant fixture, identify the enforcement point and explain the remaining gap. If the report cannot support that handoff, resolve the missing scope detail while the assessor and implementation team still share the context.