AI Agent Security Assessment: What to Review Before Production
A pre-launch review list for AI agents that touch production APIs, customer data, tools, memory, or RAG context.
A useful pre-production assessment starts with the access the team is about to enable. Reading one customer's tickets, editing those tickets, and emailing the customer are three different decisions. A successful demo does not resolve any of them.
Define the proposed release boundary, then test the paths that could cross it: another tenant's records, an unapproved write, a hostile document, a stale approval, or a retry after an uncertain result. The release owner needs to know which capabilities can proceed and which still need work.
OWASP's 2025 LLM Top 10 gives useful categories for the findings. The test scope comes from the application, its permissions and the harm a failure could cause.
Follow one ticket through the system
Consider an agent that reads a support ticket, searches internal documents, drafts an answer and sends it to the customer. The same model may select the source material, choose a tool and generate its parameters. A sentence in the ticket can therefore affect an operation several steps later.
Trace that request through retrieval, model context, the tool dispatcher and the downstream mailbox. Identify where code checks the user's authority and where source text can influence the next action. Include alternate paths: a permission check in the main dispatcher does not cover a connector that calls the mail SDK directly.
The review should distinguish three observations: what the agent proposed, what the application authorized, and what the downstream system actually did. A blocked send and a completed send are materially different results, even if both began with the same hostile document.
Trace the data the agent can receive
List each source and the identity used to read it. Include user input, retrieved chunks, SaaS records, tool responses and memory. Logs deserve a place in this inventory too; a debugging tool can expose credentials that the main application carefully excludes.
For each source, ask who can change its content and whose permissions govern retrieval. An internal document may be editable by someone who has no authority to change a customer's account. Loading that document into context must not promote its author into an approver.
Test retrieval with two tenants, different roles and a permission change during a session. Check summaries and citations as well as raw document responses. Authorization must apply before protected content reaches a model, an external reranker or the user. Keep source identifiers so the reviewer can trace an answer back to the records used.
Map the actions the runtime can reach
A tool's name may hide several distinct permissions. Separate draft from send, preview from production, and a note update from a billing change. Inventory capabilities available through the credential or local process even when the agent's interface does not advertise them.
| Operation | What to establish before release |
|---|---|
| Read a record | The requester can access this tenant, resource and set of fields. |
| Create a draft or edit a record | The target, proposed fields and business rules are checked outside the model. |
| Send a message | Recipients, attachments and content fit an approved policy; any required approval covers the exact send. |
| Delete, change access or move money | A separately authorized business process defines the permitted targets, limits and approval conditions. |
| Execute code or deploy | The environment, credentials, network access and release gate constrain the resulting operation. |
Human approval is useful when someone must judge a consequential action. Bounded automation can also be appropriate. In either case, the runtime must enforce the decision; an instruction telling the model to ask first is insufficient.
Put hostile content in the normal workflow
Place an instruction in a document the support agent would normally retrieve. Have it ask for a different recipient, another customer's attachment or a change to the ticket's priority. Repeat through a ticket, webpage, tool response and memory record where those sources exist in the application.
Use the normal ingestion and retrieval path. Pasting a payload directly into the model tests a different entry point from a document that passes through parsing, chunking and search.
Record whether the content was retrieved, whether the model proposed the requested action, whether policy allowed it and whether anything changed downstream. Content labels and structured prompts may help the model interpret a source. The tool boundary must still reject an unauthorized request when those measures fail.
Check the proposed action at execution
Model output becomes application input when it supplies a SQL query, shell command, API parameter, email or record patch. Validate its structure and its authority separately. A valid JSON body can still name the wrong customer.
Use typed schemas and allowed values for tool inputs, parameterized queries for data access, and a constrained execution environment for code. For writes, check the current resource and business state. Bind an approval to the relevant action, target and parameters; changing an attachment or recipient should invalidate the earlier approval.
Test the authorized version of the operation as well. A control that blocks every send may stop the attack while also preventing the workflow the customer bought.
Exercise uncertain results and shutdown
A timeout does not establish that a write failed. The mail provider may have accepted the message before the connection broke. Retry and reconciliation behavior belong in the assessment because a repeated operation can cause damage without any new model decision.
Set bounds on calls, elapsed time, concurrency and cost. Where the downstream API supports it, use an idempotency key tied to one intended operation. Reconcile an uncertain outcome before retrying. Test what happens when a tool is disabled or a credential is revoked while work is queued or in flight.
Memory adds another recovery question: can the agent carry the bad instruction or false result into its next session? Check the copies, caches and derived records that need correction, rather than assuming that clearing one conversation removes them.
Check whether an operator can reconstruct the result
The trace needs the initiating identity and tenant, relevant source references, proposed action, policy decision, approval reference and downstream outcome. Store a safe projection of parameters when full bodies would expose sensitive material. Restrict access and retention for the logs themselves.
A provider's acceptance response, a later readback and a user-visible delivery may establish different things. Label those observations separately. Private model reasoning is neither required nor a dependable explanation of the external action.
Make the release decision specific
The assessment should produce a short release memo supported by the tested configuration, a trust-boundary diagram and reproducible findings. Each finding needs an affected capability, reproduction conditions, observed result, impact, owner and retest method. Record what was not tested.
For the support agent, the decision might allow drafts for one tenant while withholding external sends until recipient checks pass. That is more useful than declaring the entire agent either safe or unsafe. Name the permitted data classes, tools, operating limits and person who can stop the pilot.
Retest corrections against the version that will ship. A staging result with different credentials or retrieval filters does not establish the production behavior. The enterprise evaluation guide shows how to carry those constraints into an operating decision.