Designing Acceptance Tests for AI Agent Security Controls
A worked acceptance plan for agent access controls, with meaningful deny cases, failure behavior and explicit release decisions.
“The guardrail is enabled” describes configuration. An acceptance test needs an operation the guardrail must prevent, a legitimate operation it must allow, and an observation that distinguishes the two.
For an agent connected to business systems, I start with the resource boundary. If the requirement is that an employee cannot export another customer's project, the test must reach the export authorization check. A model declining to suggest the export leaves that requirement untested.
My KevinBytes evaluation article explains how to compare products at different points in an agent workflow. I write both publications and am affiliated with TKOResearch. Here the decision is narrower: whether one implemented control is acceptable for a defined release. The worked plan below is hypothetical and has not been executed against a vendor product.
Write the requirement as a transaction
Assume a project assistant helps Cedar employees prepare document exports. Employee Mira may request an export of her assigned project. A project owner must approve the exact document set. A worker creates the archive inside Cedar's workspace; external delivery is unavailable in this pilot.
Express the requirement using subject, resource, action, conditions and outcome:
Mira may request an archive of documents currently readable in her assigned Cedar project. The export worker may create it only when a current project owner has approved that document set and the applicable access checks still succeed at execution.
Specify what “currently readable” means for queued jobs. This example rechecks requester access and approver authority immediately before archive creation. Another business process might intentionally authorize durable work that survives a personnel change. That needs an explicit policy and its own cases; it shouldn't emerge accidentally from cached permissions.
General authorization principles remain useful. OWASP's Authorization Cheat Sheet, reviewed September 9, 2026, recommends least privilege, default denial and permission checks on each request. The transaction and fixture design here are authored acceptance criteria for the invented workflow.
Locate the check and the observation
Draw the path from user request through model, tool dispatcher, queue, worker, document service and archive store. Label which component decides policy and which component can prevent access. Include administrative and retry routes that can reach the same export operation.
NIST SP 800-207, final August 2020, separates policy decisions from enforcement in its zero-trust architecture. Applying that distinction here helps identify a common test mistake: exercising an advisory component while the data service still accepts an overbroad credential.
Choose observations at the boundary you claim to verify. For an unauthorized export, inspect the service's rejection and the absence of a newly created archive in the instrumented fixture store. For a valid export, inspect the receipt and archive manifest. A green UI message cannot substitute for either.
Build a fixture that can fail meaningfully
Create two invented tenants, Cedar and Birch, each with an existing project and readable documents. Establish ordinary access in both tenants before attempting cross-tenant access. Otherwise a “denial” may be a lookup failure for a nonexistent resource.
Use distinct documents with recognizable non-sensitive fixture content. Give one Cedar document a restricted classification so that project membership alone is insufficient. Create an approval for a precise sorted document set and versioned request, with an explicit expiry and execution allowance.
| Case | Subject, resource and action | Required outcome and observation |
|---|---|---|
| Authorized baseline | Mira exports her approved Cedar set | One archive, correct tenant and manifest, linked receipt |
| Other tenant | Mira requests an existing Birch document | Resource boundary denies; no Birch content reaches archive creation |
| Restricted item | Mira adds a Cedar document she cannot read | Export is rejected under this pilot's all-or-nothing policy |
| Changed request | Worker receives an extra document after approval | Request no longer matches the approval; no archive created |
| Expired approval | Valid request waits beyond its approval lifetime | Worker rejects execution with an explicit reason |
| Revoked requester | Mira loses access while the job waits | Execution recheck rejects the pending request |
| Alternate route | Retry worker calls the export service directly | The same authorization contract still applies |
| Duplicate delivery | Queue delivers the same approved operation twice | Behavior matches the defined idempotency contract; no extra authorized use |
| Policy unavailable | Policy dependency times out before export | This consequential operation waits or rejects as specified; no silent allow |
These are expected results, not measured outcomes. Set the actual time limits and idempotency semantics from the application contract. For the duplicate case, returning an existing archive may be appropriate; accepting a different request under the same identifier is not.
Separate deterministic checks from model trials
Run control tests with structured requests so every deny case reaches the relevant boundary. Then add end-to-end agent trials to determine whether ordinary tasks complete and whether untrusted document content influences attempted actions.
Those layers answer different questions. A deterministic test can show that the service rejects another tenant's resource for the supplied principal. Repeated model trials sample whether the agent attempts that action under selected inputs. Neither establishes universal resistance to every possible prompt.
Pin the application build, tool schema, policy version, model configuration and fixture revision. Record expected result, observed result, enforcement point and supporting artifact location for each case. Keep “not reached,” “not observable” and “not run” separate from pass and fail.
Include partial failure after dispatch. If archive creation returns a timeout, reconcile by operation ID before retrying. The test should prove that operators can identify the unknown state and recover according to the contract. An assertion that merely expects a timeout exception says nothing about whether an archive was created.
Decide acceptance before seeing the score
For this pilot, cross-tenant access and approval substitution are release blockers. A slow but correct archive may need performance work without invalidating the authorization claim. Keep those decisions explicit instead of averaging them into one score that allows fast responses to offset a broken boundary.
The acceptance record should identify the owner, permitted deployment, remaining restrictions and the artifacts supporting the decision. An independent technical risk review can assess whether that package supports the proposed access. The AI risk board report tool can help communicate the resulting decision, but a management report cannot turn an unrun case into a successful test.
Retire or replace a control only after mapping its responsibility to the replacement and rerunning relevant cases. A new gateway may filter tool calls while removing a previous service-level check; overlapping feature descriptions do not establish equivalent enforcement.
Reopen acceptance when authority or reachable paths change. A new export destination, worker credential, approval policy or document classification changes the question the original tests answered. Keep the case package beside the release configuration so the next reviewer can see exactly which assumption moved.