AI Agent Security Assessment: What to Review Before Production
A pre-launch review list for AI agents that touch production APIs, customer data, tools, memory, or RAG context.
Start Here
If your AI agent can call tools, retrieve documents, write to SaaS systems, access customer data, send messages, run code, or make workflow decisions, it should not be reviewed as "just a chatbot."
It should be reviewed as a software system with authority.
The core question is no longer whether someone can make the model say something strange. The real question is this:
If the model, prompt context, retrieval layer, tool output, or memory is manipulated, what can the system actually do?
That is the purpose of an AI Agent Security Assessment: to identify whether an agent is ready for production, limited pilot, enterprise review, or further remediation before it touches real data and real business systems.
OWASP's LLM guidance identifies prompt injection, insecure output handling, sensitive information disclosure, insecure plugin design, and excessive agency as major LLM application risks. Those risks become materially more serious when the LLM is connected to tools, APIs, memory, and production workflows.
Why AI Agent Security Is Different
Traditional application security usually assumes that application logic is mostly deterministic. A user sends a request, the application validates it, business logic executes, and a result is returned.
AI agents change that pattern.
An AI agent may interpret user intent, inspect context, retrieve documents, select tools, generate parameters, call APIs, summarize results, update memory, and decide what to do next. That flexibility is why agents are useful. It is also why they are risky.
OWASP describes AI agents as LLM-powered systems that can reason, plan, use tools, maintain memory, and take actions. The OWASP AI Agent Security Cheat Sheet also highlights agent-specific risks including prompt injection, tool abuse, privilege escalation, data exfiltration, memory poisoning, goal hijacking, excessive autonomy, cascading failures, denial-of-wallet, and sensitive data exposure.
A production AI agent needs controls around at least five layers:
| Layer | Security Question |
|---|---|
| Prompt and instruction layer | Can untrusted input override system intent? |
| Retrieval and context layer | Can documents, emails, webpages, or records inject instructions or leak data? |
| Tool and API layer | Can the agent take actions beyond what the user or workflow intended? |
| Identity and authorization layer | Are actions scoped to the correct user, tenant, role, and business context? |
| Logging and artifact layer | Can the organization reconstruct what happened and why? |
If those layers are not reviewed together, the assessment will miss the actual failure mode.
The Pre-Production Review
A practical AI Agent Security Assessment should answer seven questions.
1. What Can The Agent See?
Start with data exposure.
The assessment should identify every source of information the agent can access:
| Data Source | Examples | Review Focus |
|---|---|---|
| User prompt | Chat input, form input, support ticket | Input trust, injection handling, authorization |
| Retrieved documents | RAG chunks, knowledge base articles, PDFs | Tenant isolation, metadata filtering, document trust |
| SaaS records | CRM, ticketing, email, calendar, ERP | Per-user access, role boundaries, auditability |
| Tool output | API results, command output, database rows | Whether output is treated as trusted instruction |
| Memory | Short-term session state, long-term profile memory | Poisoning, cross-user contamination, stale context |
| Logs | Prompt logs, traces, tool calls, errors | Sensitive data exposure and retention |
A safe agent does not simply retrieve relevant data. It retrieves authorized data, preserves source attribution, respects tenant boundaries, and treats external content as untrusted.
2. What Can The Agent Do?
The next question is action authority.
Every tool should be classified by impact:
| Action Class | Examples | Default Control |
|---|---|---|
| Read-only | Search docs, summarize tickets, list records | Allow with logging |
| Reversible write | Create draft, add internal note, update non-critical field | Require validation and clear attribution |
| Irreversible write | Delete record, close case, modify billing | Require explicit approval |
| External communication | Send email, post message, notify customer | Draft-first or human approval |
| Code or command execution | Run shell, deploy code, trigger CI/CD | Strong sandboxing or deny by default |
| Financial or legal action | Refund, approve payment, submit filing | Human approval and policy enforcement |
OWASP's agent guidance recommends least privilege for tools, per-tool permission scoping, separate tool sets for different trust levels, and explicit authorization for sensitive operations.
A good assessment should not only ask whether a tool exists. It should ask whether the tool is scoped, whether the agent can chain it with other tools, whether the action is attributable to the correct user, and whether the server independently enforces authorization.
3. Can Untrusted Content Become Instruction?
Prompt injection is not limited to a user typing hostile instructions into a chat box. In agentic systems, instructions can enter through documents, webpages, emails, tickets, tool outputs, logs, memory, and retrieved content.
OWASP's prompt injection prevention guidance calls out RAG poisoning and agent-specific attacks such as thought/observation injection, tool manipulation, and context poisoning.
The assessment should test whether untrusted content can influence:
| Injection Source | Example Risk |
|---|---|
| User prompt | User asks agent to bypass workflow rules |
| Retrieved document | Document contains malicious instructions that override the task |
| Webpage | Agent summarizes attacker-controlled content that includes hidden instructions |
| Email or ticket | Incoming message attempts to redirect the agent's behavior |
| Tool output | API response includes text that changes the next tool call |
| Memory | Poisoned memory changes future behavior across sessions |
The right control is not better prompting alone. The system needs trust boundaries, content labeling, structured tool schemas, server-side authorization, output validation, and regression tests for known abuse classes.
4. Are RAG Boundaries Real Or Cosmetic?
RAG security is often where AI systems fail quietly.
A RAG workflow may appear safe because the application only exposes a chat interface. But if retrieval ignores tenant, role, source, document classification, or authorization metadata, the model can surface data the user should never receive.
The assessment should validate:
| RAG Control | Review Question |
|---|---|
| Tenant isolation | Can user A retrieve user B's records? |
| Role filtering | Does retrieval respect the user's actual permissions? |
| Source attribution | Can every answer be traced back to authorized source material? |
| Document trust | Are external or untrusted documents treated differently from trusted internal sources? |
| Metadata enforcement | Are filters applied server-side or merely implied in the prompt? |
| Leakage testing | Can sensitive content be induced through summaries, comparisons, or indirect queries? |
A safe RAG system should enforce access before content reaches the model context. The model should not be trusted to decide what the user is allowed to see.
5. Is Output Validated Before It Becomes Action?
Improper output handling becomes more dangerous when model output is passed downstream.
If an LLM output becomes SQL, shell commands, API parameters, JSON patches, workflow transitions, emails, or code, then model output is part of the execution path. OWASP identifies insecure output handling as a risk because unvalidated outputs can lead to downstream exploits, including code execution and data exposure.
The assessment should review:
| Output Path | Required Control |
|---|---|
| JSON tool parameters | Schema validation and allowed-value enforcement |
| SQL or query generation | Parameterization and server-side policy |
| Shell/code execution | Sandbox, allowlist, and strict deny rules |
| Email/message generation | Draft-first, approval, and recipient validation |
| Record updates | Server-side authorization and business rule enforcement |
| Summaries | Sensitive-data filtering and source validation |
The model should propose actions. Deterministic code should authorize, validate, and execute them.
6. Can The Agent Loop, Escalate, Or Compound Errors?
Autonomy is useful until the system can keep acting after it has gone wrong.
Review whether the agent can:
- Retry failed actions indefinitely.
- Chain multiple low-risk tools into a high-risk outcome.
- Convert read access into data exfiltration.
- Use one tool's output as another tool's input without validation.
- Persist manipulated state into memory.
- Trigger external systems without human confirmation.
- Create cost explosions through uncontrolled loops.
OWASP identifies excessive autonomy as an agent risk when high-impact actions occur without appropriate oversight.
The fix is not to remove all autonomy. The fix is to classify actions, define thresholds, require confirmation for high-impact operations, and log the reasoning and authorization path.
7. Can You Reconstruct What Happened?
If an AI agent causes damage and the logs only show "model responded," the system is not production-ready.
A production AI agent should preserve enough audit artifacts to answer:
| Question | Artifact Needed |
|---|---|
| Who initiated the action? | User identity, session, role, tenant |
| What did the agent see? | Prompt, retrieved sources, tool outputs, memory references |
| What did the agent decide? | Planned action, selected tool, generated parameters |
| What authorized execution? | Policy result, approval event, server-side authorization |
| What happened downstream? | API response, record change, external message, error state |
| What was exposed? | Output shown to user, files accessed, data returned |
This does not mean logging sensitive data recklessly. It means designing traceability, redaction, retention, and access controls into the agent runtime before launch.
What The Review Should Leave Behind
The useful output is not a long report that nobody can act on. It is a small set of artifacts that lets engineering and leadership decide whether the agent can launch, should stay in pilot, or needs more work.
| Deliverable | Description |
|---|---|
| Go/No-Go Launch Memo | Decision-ready memo stating whether the agent is ready for production, pilot-only, or blocked. |
| Agent Trust-Boundary Map | Diagram of model, memory, RAG, tools, APIs, credentials, logs, and approval gates. |
| Abuse-Case Matrix | Structured tests for prompt injection, indirect injection, RAG leakage, unsafe tool use, and excessive autonomy. |
| Findings Register | Severity-ranked findings with artifacts, impact, recommended fix, owner, and validation method. |
| Remediation Roadmap | Prioritized engineering plan for pre-launch blockers and post-launch hardening. |
What Production-Ready Looks Like
An AI agent is closer to production-ready when:
- It uses least-privilege tools.
- Write actions are classified and gated.
- RAG retrieval enforces authorization before model context.
- External content is treated as untrusted.
- Model output is validated before downstream execution.
- Sensitive operations require deterministic server-side authorization.
- Logs preserve prompt, retrieval, tool-call, approval, and output artifacts.
- Memory is scoped, validated, and isolated.
- There is a documented rollback, disable, or kill-switch path.
- The system has regression tests for prompt injection and tool misuse scenarios.
What Usually Blocks Launch
Common blockers include:
| Blocker | Why It Matters |
|---|---|
| Broad service credentials | The agent inherits more authority than any user should have. |
| Prompt-only access control | The model is being asked to enforce policy it can be manipulated to ignore. |
| Write-capable tools without confirmation | Prompt injection can become production impact. |
| RAG without tenant isolation | Users may retrieve or summarize unauthorized data. |
| Tool outputs treated as instructions | Malicious API/content responses can influence next actions. |
| Missing audit traces | The organization cannot reconstruct failures or prove bounded operation. |
Final Thought
AI agent security is not about proving the model is perfectly safe. It is about proving that the system remains bounded when the model is wrong, manipulated, confused, or exposed to hostile context.
Before your AI agent reaches production, you should be able to answer one question with concrete artifacts:
What is the worst thing this agent can do if something goes wrong?
If the answer is unclear, the agent is not ready.
What TKOResearch Reviews
TKOResearch performs principal-led AI Agent Security Assessments for teams preparing production launch, enterprise review, investor diligence, or internal security sign-off.