AI Agent Security Assessment: What to Review Before Production
A pre-launch review list for AI agents that touch production APIs, customer data, tools, memory, or RAG context.
A useful pre-production assessment starts with the access the team is about to enable. Reading one customer's tickets, editing those tickets, and emailing the customer are three different decisions. A successful demo does not resolve any of them.
I would define the proposed release boundary first, then test the paths that could cross it: another tenant's records, an unapproved write, a hostile document, a stale approval, or a retry after an uncertain result. The output should tell the release owner which capabilities can proceed and which still need work.
OWASP's 2025 LLM Top 10 includes prompt injection, sensitive information disclosure, improper output handling, and excessive agency. Those categories help organize the assessment; the actual test scope still comes from the application and the harm a failure could cause.
Why AI Agent Security Is Different
Traditional application security usually assumes that application logic is mostly deterministic. A user sends a request, the application validates it, business logic executes, and a result is returned.
AI agents change that pattern.
An AI agent may interpret user intent, inspect context, retrieve documents, select tools, generate parameters, call APIs, summarize results, update memory, and decide what to do next. That flexibility is why agents are useful. It is also why they are risky.
A production AI agent needs controls around at least five layers:
| Layer | Security Question |
|---|---|
| Prompt and instruction layer | Can untrusted input override system intent? |
| Retrieval and context layer | Can documents, emails, webpages, or records inject instructions or leak data? |
| Tool and API layer | Can the agent take actions beyond what the user or workflow intended? |
| Identity and authorization layer | Are actions scoped to the correct user, tenant, role, and business context? |
| Logging and artifact layer | Can the organization reconstruct what happened and why? |
Review the transitions between layers as well as each component. A permission check in one dispatcher does not cover an alternate SDK path.
The Pre-Production Review
A practical AI Agent Security Assessment should answer seven questions.
1. Trace the data available to the agent
Start with data exposure.
The assessment should identify every source of information the agent can access:
| Data Source | Examples | Review Focus |
|---|---|---|
| User prompt | Chat input, form input, support ticket | Input trust, injection handling, authorization |
| Retrieved documents | RAG chunks, knowledge base articles, PDFs | Tenant isolation, metadata filtering, document trust |
| SaaS records | CRM, ticketing, email, calendar, ERP | Per-user access, role boundaries, auditability |
| Tool output | API results, command output, database rows | Whether output is treated as trusted instruction |
| Memory | Short-term session state, long-term profile memory | Poisoning, cross-user contamination, stale context |
| Logs | Prompt logs, traces, tool calls, errors | Sensitive data exposure and retention |
A safe agent does not simply retrieve relevant data. It retrieves authorized data, preserves source attribution, respects tenant boundaries, and treats external content as untrusted.
2. Map reachable actions
The next question is action authority.
Every tool should be classified by impact:
| Action Class | Examples | Default Control |
|---|---|---|
| Read-only | Search docs, summarize tickets, list records | Authorize user, tenant, resource and returned fields; log the decision |
| Reversible write | Create draft, add internal note, update non-critical field | Require validation and clear attribution |
| Irreversible write | Delete record, close case, modify billing | Require explicit approval |
| External communication | Send email, post message, notify customer | Draft-first or human approval |
| Code or command execution | Run shell, deploy code, trigger CI/CD | Strong sandboxing or deny by default |
| Financial or legal action | Refund, approve payment, submit filing | Human approval and policy enforcement |
For each tool, inspect its scope, possible chains, downstream attribution and independent authorization. Include capabilities that are available to the runtime even if the interface does not advertise them.
3. Test instructions embedded in source content
Prompt injection is not limited to a user typing hostile instructions into a chat box. In agentic systems, instructions can enter through documents, webpages, emails, tickets, tool outputs, logs, memory, and retrieved content.
The assessment should test whether untrusted content can influence:
| Injection Source | Example Risk |
|---|---|
| User prompt | User asks agent to bypass workflow rules |
| Retrieved document | Document contains malicious instructions that override the task |
| Webpage | Agent summarizes attacker-controlled content that includes hidden instructions |
| Email or ticket | Incoming message attempts to redirect the agent's behavior |
| Tool output | API response includes text that changes the next tool call |
| Memory | Poisoned memory changes future behavior across sessions |
Combine content labels and structured prompts with tool schemas, current authorization and regression tests. Labels can help the model; deterministic code must still reject an unauthorized action.
4. Verify retrieval permissions
RAG security is often where AI systems fail quietly.
A RAG workflow may appear safe because the application only exposes a chat interface. But if retrieval ignores tenant, role, source, document classification, or authorization metadata, the model can surface data the user should never receive.
The assessment should validate:
| RAG Control | Review Question |
|---|---|
| Tenant isolation | Can user A retrieve user B's records? |
| Role filtering | Does retrieval respect the user's actual permissions? |
| Source attribution | Can every answer be traced back to authorized source material? |
| Document trust | Are external or untrusted documents treated differently from trusted internal sources? |
| Metadata enforcement | Are filters applied server-side or merely implied in the prompt? |
| Leakage testing | Can sensitive content be induced through summaries, comparisons, or indirect queries? |
A safe RAG system should enforce access before content reaches the model context. The model should not be trusted to decide what the user is allowed to see.
5. Validate output at the execution boundary
Improper output handling becomes more dangerous when model output is passed downstream.
If an LLM output becomes SQL, shell commands, API parameters, JSON patches, workflow transitions, emails, or code, then model output is part of the execution path. OWASP identifies improper output handling as a risk because unvalidated outputs can lead to downstream exploits, including code execution and data exposure.
The assessment should review:
| Output Path | Required Control |
|---|---|
| JSON tool parameters | Schema validation and allowed-value enforcement |
| SQL or query generation | Parameterization and server-side policy |
| Shell/code execution | Sandbox, allowlist, and strict deny rules |
| Email/message generation | Draft-first, approval, and recipient validation |
| Record updates | Server-side authorization and business rule enforcement |
| Summaries | Sensitive-data filtering and source validation |
The model should propose actions. Deterministic code should authorize, validate, and execute them.
6. Bound retries and chained actions
Autonomy is useful until the system can keep acting after it has gone wrong.
Review whether the agent can:
- Retry failed actions indefinitely.
- Chain multiple low-risk tools into a high-risk outcome.
- Convert read access into data exfiltration.
- Use one tool's output as another tool's input without validation.
- Persist manipulated state into memory.
- Trigger external systems without human confirmation.
- Create cost explosions through uncontrolled loops.
Classify actions, set runtime and retry limits, and require approval where the impact warrants it. Record the proposed action and authorization decision. Private model reasoning is neither necessary nor a reliable account of why a downstream operation happened.
7. Reconstruct the tested action path
If an AI agent causes damage and the logs only show "model responded," the system is not production-ready.
A production AI agent should preserve enough audit artifacts to answer:
| Question | Artifact Needed |
|---|---|
| Who initiated the action? | User identity, session, role, tenant |
| What did the agent see? | Prompt, retrieved sources, tool outputs, memory references |
| What did the agent decide? | Planned action, selected tool, generated parameters |
| What authorized execution? | Policy result, approval event, server-side authorization |
| What happened downstream? | API response, record change, external message, error state |
| What was exposed? | Output shown to user, files accessed, data returned |
This does not mean logging sensitive data recklessly. It means designing traceability, redaction, retention, and access controls into the agent runtime before launch.
What The Review Should Leave Behind
The useful output is not a long report that nobody can act on. It is a small set of artifacts that lets engineering and leadership decide whether the agent can launch, should stay in pilot, or needs more work.
| Deliverable | Description |
|---|---|
| Go/No-Go Launch Memo | Decision-ready memo stating whether the agent is ready for production, pilot-only, or blocked. |
| Agent Trust-Boundary Map | Diagram of model, memory, RAG, tools, APIs, credentials, logs, and approval gates. |
| Abuse-Case Matrix | Structured tests for prompt injection, indirect injection, RAG leakage, unsafe tool use, and excessive autonomy. |
| Findings Register | Severity-ranked findings with artifacts, impact, recommended fix, owner, and validation method. |
| Remediation Roadmap | Prioritized engineering plan for pre-launch blockers and post-launch hardening. |
What Production-Ready Looks Like
An AI agent is closer to production-ready when:
- It uses least-privilege tools.
- Write actions are classified and gated.
- RAG retrieval enforces authorization before model context.
- External content is treated as untrusted.
- Model output is validated before downstream execution.
- Sensitive operations require deterministic server-side authorization.
- Logs preserve prompt, retrieval, tool-call, approval, and output artifacts.
- Memory is scoped, validated, and isolated.
- There is a documented rollback, disable, or kill-switch path.
- The system has regression tests for prompt injection and tool misuse scenarios.
What Usually Blocks Launch
Common blockers include:
| Blocker | Why It Matters |
|---|---|
| Broad service credentials | The agent inherits more authority than any user should have. |
| Prompt-only access control | The model is being asked to enforce policy it can be manipulated to ignore. |
| High-impact writes without an effective approval or bounded automation policy | Manipulated context can cause an unauthorized change. |
| RAG without tenant isolation | Users may retrieve or summarize unauthorized data. |
| Tool outputs treated as instructions | Malicious API/content responses can influence next actions. |
| Missing audit traces | The organization cannot reconstruct failures or prove bounded operation. |
Make the release decision specific
A finding should name the affected capability, reproduction conditions, observed result, and limitation. A recommendation to keep the agent in pilot should name the permitted tenants, tools, data classes, operating limits, and owner who can stop it.
Retest the corrected path against the version that will ship. A clean result on a staging configuration with different credentials or retrieval filters does not establish the production boundary. The enterprise evaluation guide shows how to turn those constraints into an operating decision.