Evaluate AI Agents for Enterprise Use

An enterprise does not adopt an AI agent in the abstract. It gives a particular system a role in a particular workflow, with access to particular data and tools.

That is the unit that needs to be evaluated.

A model comparison can tell you something about response quality. It cannot tell you whether an accounts-payable agent should read vendor records, whether a support agent should send customer email, or whether an engineering agent should receive a token that can modify a repository. Those are system and operating decisions.

This guide provides a practical way for security, platform, and product teams to make them together.

An Illustrative Accounts-Payable Agent

Consider an enterprise team evaluating an accounts-payable agent. This is an illustrative scenario, not a description of a TKOResearch client engagement.

The product proposal sounds reasonable. The agent will:

Read invoices received through an approved intake channel.
Compare invoice fields with vendor records and purchase orders.
Check approval history and flag mismatches.
Draft an entry for the enterprise resource planning system.
Explain why it believes an invoice is ready or requires review.

The proposal becomes risky when the boundary is vague. A vendor demonstration may use one broadly privileged ERP service account. The agent may be technically able to create vendors, change bank details, post entries, approve exceptions, or release payments even though the product team only intends it to flag mismatches.

That gap between intended use and available authority is where the evaluation should begin.

For this scenario, three actions are out of scope from the start:

Approving a new vendor.
Changing vendor bank details.
Releasing a payment.

The evaluation is not trying to prove that the agent will never make a mistake. It is deciding whether the workflow remains controlled when the model is wrong, uncertain, manipulated, or unavailable.

1. Start With the Workflow, Not the Model

Write down the work before discussing model quality.

For the accounts-payable agent, the business outcome might be:

Reduce the time required to reconcile ordinary invoices while preserving the existing approval authority for vendor changes and payment release.

That sentence is more useful than "automate accounts payable." It names the task and preserves the human decision boundary.

The first review should answer five questions:

Question	Accounts-payable example
What starts the workflow?	An invoice arrives through the approved intake queue.
What does the agent produce?	A mismatch flag, source references, and a draft ERP entry.
Who relies on the result?	Accounts-payable staff and the assigned approver.
What must remain a human decision?	Vendor creation, bank-detail changes, exceptions, and payment release.
What would make the workflow unsuccessful?	Incorrect posting, missed duplicate invoices, cross-entity data access, or an unauditable recommendation.

Product owns the usefulness of the workflow. Security owns control requirements. Platform owns the runtime boundary. Operations owns the procedure when the system is uncertain or unavailable. The evaluation fails if one team is expected to answer for all four.

2. Define the Complete System Boundary

An agent is more than the model endpoint. The review boundary should include every component that can affect what the system sees, decides, or does.

For the example, the boundary contains:

Invoice intake
  -> document parser
  -> agent orchestrator
  -> model provider
  -> vendor and purchase-order retrieval
  -> mismatch rules
  -> ERP draft-entry tool
  -> approval queue
  -> audit and monitoring systems

The system also depends on identities, credentials, prompts, tool definitions, retrieval filters, memory, policy code, queues, and vendor configuration. If one of those changes, the evaluated system may have changed even when the model name has not.

Use a boundary table that records ownership and trust, not only architecture:

Component	Owner	Data handled	Trust concern	Required control
Invoice intake	Finance operations	Invoice and sender data	Sender-controlled content	Approved channel, malware handling, source tracking
Document parser	Platform	Extracted invoice fields	Parsing errors and hidden content	File limits, field validation, retained source reference
Model provider	Platform	Selected invoice context	External dependency and retention	Enterprise terms, configured retention, provider inventory
Vendor retrieval	ERP team	Vendor master data	Excess data and cross-entity access	Entity-scoped query, field allowlist, user attribution
Draft-entry tool	ERP team	Proposed accounting entry	Unauthorized write or wrong ledger	Draft-only role, typed fields, deterministic policy checks
Approval queue	Finance operations	Recommendation and source references	Approval confusion	Named approver, clear preview, parameter-bound approval
Audit store	Security operations	Action and policy records	Sensitive-data overcollection	Structured records, redaction, retention, access control

If the team cannot draw this boundary and name an owner for each part, it is not ready to approve production access.

3. Decide What Data the Agent May Receive

Data review should be field-specific. "ERP access" and "invoice access" are too broad to support a decision.

For each data source, record:

The business reason for access.
The fields required for the task.
The user, legal entity, tenant, or region boundary.
Whether the source contains instructions or text controlled by another party.
How long the data remains in prompts, traces, memory, and provider systems.
Whether the system can function with less data.

The illustrative agent needs invoice amount, currency, vendor identifier, purchase-order reference, line items, and approval status. It does not need vendor bank-account numbers to detect a price or quantity mismatch. Excluding those fields reduces both exposure and the consequences of a bad output.

A practical data decision might look like this:

Data	Decision	Reason
Invoice fields	Allow	Required for reconciliation
Purchase-order lines	Allow	Required for quantity and price comparison
Vendor name and internal ID	Allow	Required for matching
Vendor bank details	Deny	Not required for the approved workflow
Prior invoice notes	Allow with limits	Useful for known exceptions, but may contain untrusted text
Other legal entities' records	Deny	Outside the assigned entity boundary
Full email thread	Deny by default	Broader and less structured than the task requires

Authorization must happen before restricted content enters model context. A prompt telling the model not to reveal another entity's invoices is not an access-control mechanism.

4. Map Identity, Credentials, and Authorization

Ask whose authority the agent is using at each step.

There are usually three distinct identities:

The initiating user. The person who starts or reviews the workflow.
The agent workload. The service identity used by the orchestration layer.
The downstream actor. The identity recorded by the ERP, ticketing, email, or other target system.

These should not collapse into one broad service account.

For the accounts-payable pilot, the agent can use a workload identity that reads records for one legal entity and creates draft entries in a designated queue. The request must also carry the initiating user's identity so policy and audit records remain attributable. The ERP role cannot create vendors, alter payment details, approve an entry, or release funds.

Reviewers should ask:

Owner	Question
Security	Can a user or agent reach data outside the initiating user's assigned entity?
Platform	Where are credentials issued, scoped, rotated, and revoked?
Product	Does the interface clearly distinguish a draft from an approved transaction?
Finance operations	Who owns the queue when the agent cannot reconcile an invoice?
Legal and privacy	Which providers receive invoice or vendor data, and under what terms?
Procurement	Can the vendor change models, retention, or subprocessors without notice?

The model can recommend an action. It should not establish the identity or authorization under which that action runs.

5. Classify Tools and Action Authority

List every tool. Do not stop at the tools shown in the user interface. Include internal APIs, retrieval functions, queues, message senders, administrative endpoints, and maintenance functions available to the runtime.

Classify each action by what it changes and how difficult it is to reverse:

Tool action	Class	Pilot decision	Required control
Read purchase order	Read	Allow	Entity-scoped query and field allowlist
Search vendor record	Read	Allow	Exact-match preference and entity boundary
Flag mismatch	Reversible write	Allow	Source references and user attribution
Draft ERP entry	Reversible write	Allow with review	Typed schema, policy checks, draft-only status
Create vendor	Administrative write	Deny	Outside the approved workflow
Change bank details	High-impact write	Deny	Separate established business process
Release payment	Financial action	Deny	Human approval in the ERP system

This classification should drive the technical permission set. It is not enough for a product requirement to say the agent "will not" release payment when its credential can call that operation.

For a deeper implementation treatment, see How to Give AI Agents Access to Tools Securely.

6. Test the Workflow, Not Just the Response

Evaluation should use the records, edge cases, and failure conditions the agent will encounter in the actual process.

A useful test set for this scenario includes:

Ordinary invoices with clean purchase-order matches.
Partial shipments and split invoices.
Duplicate invoices with minor formatting differences.
Currency and tax variations.
Vendors with similar names.
Missing purchase-order references.
Conflicting approval history.
Scanned documents with weak extraction quality.
Invoice notes that contain instructions addressed to the agent.
Records from another legal entity that must remain inaccessible.
A model timeout after a draft has been requested.
A repeated request with the same invoice and idempotency key.

Measure the whole workflow:

Measure	What it answers
Correct reconciliation rate	Does the agent reach the right operational conclusion?
False-clear rate	How often does it mark a problematic invoice as ready?
Escalation quality	Does it route uncertain cases with useful context?
Unauthorized-access denials	Do identity and entity boundaries hold under negative tests?
Draft accuracy	Are account, amount, currency, and references correct?
Duplicate prevention	Can retries create more than one draft?
Reviewer correction rate	How often must staff materially change the result?
Time saved per accepted case	Does the workflow provide enough value to justify its controls and operating cost?

One aggregate accuracy number hides the failures that matter. A system can perform well on common invoices and still be unsuitable if it occasionally crosses an entity boundary or creates duplicate drafts.

NIST's AI Risk Management Framework organizes AI risk work around Govern, Map, Measure, and Manage. That is a useful reminder that testing is not a one-time model contest. The organization must define the context, measure the system, assign responsibility, and manage it after deployment.

7. Evaluate Failure Handling and Operational Control

Ask what happens when the system cannot complete the task safely.

For the pilot, acceptable failure behavior is boring:

If document extraction is uncertain, route the invoice for manual review.
If the vendor match is ambiguous, do not select one.
If the policy service is unavailable, do not create a draft.
If the model times out, preserve the current state and avoid an automatic write retry.
If duplicate detection fails, stop the workflow and alert the owner.
If the wrong entity appears in retrieved context, terminate the session and preserve the relevant trace identifiers.

The system also needs an operational stop path. A product owner should be able to disable the agent workflow without waiting for a code deployment. Platform teams should be able to revoke its credential. Finance operations should have a documented way to take over queued work. Security should know which records support technical review.

These are product requirements, not cleanup work for later.

8. Review Provider and Change Dependencies

An enterprise agent depends on more than its own code. Record the model provider, model version or alias, orchestration framework, retrieval services, tool servers, and important data processors.

Review:

Provider retention and training settings.
Supported regions and data-transfer terms.
Subprocessors and notification terms.
Model-update controls and deprecation policy.
Availability commitments and rate limits.
Security reporting channels.
Export and deletion capabilities.
Contract terms for customer content and generated output.
How the team will re-test after a model, prompt, tool, retrieval, memory, or policy change.

NIST SP 800-218A extends secure software development practices for AI model producers, AI system producers, and acquirers. Enterprise buyers should treat the acquired agent and its dependencies as software with a lifecycle, not a fixed appliance that was evaluated once.

9. Record the Proposed Boundary

A small machine-readable manifest can force useful precision. This is an example artifact, not a substitute for architecture review or testing.

agent: invoice-reconciliation-assistant
workflow_owner: finance-operations
business_outcome: reduce manual reconciliation time for ordinary invoices
data_classes:
  - invoice-fields
  - purchase-order-lines
  - vendor-id-and-status
  - approval-history
identities:
  initiating_user: required
  workload: ap-agent-draft-writer
tools:
  - read_invoice
  - read_purchase_order
  - find_vendor
  - create_draft_erp_entry
allowed_actions:
  - compare invoice with purchase order
  - flag mismatch
  - create draft entry in the AP review queue
prohibited_actions:
  - create or approve vendor
  - read or change vendor bank details
  - approve entry
  - release payment
approval_gates:
  draft_entry: assigned AP reviewer
model_dependency:
  provider: approved-enterprise-provider
  version: recorded-in-each-run
logging:
  - initiating-user
  - source-record-ids
  - retrieved-record-ids
  - proposed-entry
  - policy-result
  - reviewer-decision
  - downstream-result
kill_switch:
  owner: finance-platform-on-call
  action: disable workflow and revoke ap-agent-draft-writer

Version this record with the agent configuration. When a proposed change expands a tool, data class, or allowed action, require a new review.

10. Make a Decision, Not a General Recommendation

The review should end with one of three outcomes:

Outcome	Meaning
Proceed	The workflow and controls are ready for the approved operating scope.
Limited pilot	The system may operate within named constraints while the team measures specific open questions.
Do not deploy	A blocker makes the proposed use unacceptable until the design changes.

For the illustrative accounts-payable agent, a reasonable decision record is:

Decision: Limited pilot

Approved scope: One legal entity, approved invoice intake, read-only vendor and purchase-order access, mismatch flags, and draft entries in a separate review queue.

Prohibited scope: Vendor creation, bank-detail access or change, exception approval, payment approval, and payment release.

Acceptance criteria:

No successful cross-entity retrieval in the negative test suite.
No duplicate draft from retries or repeated invoice intake.
Every draft includes the initiating user, source record identifiers, policy result, and model version.
False-clear and reviewer-correction rates remain below limits set by finance operations before the pilot starts.
Staff can disable the workflow and resume manual processing within the documented recovery target.

Stop conditions:

Any unauthorized data access.
Any attempt to call a prohibited action.
Any duplicate draft caused by workflow retry.
Missing audit fields on a write attempt.
A provider or configuration change that has not completed regression testing.

Owners: Finance operations owns workflow quality. The ERP team owns permissions and draft-entry policy. Platform owns runtime reliability and credential lifecycle. Security owns abuse-case testing and control review. Product owns user experience and pilot measurement.

This decision is useful because it tells the enterprise what may happen next. "Proceed with caution" does not.

Common Evaluation Mistakes

Several shortcuts produce weak decisions:

Shortcut	Why it fails
Buying from a familiar vendor without mapping the workflow	Vendor posture does not define your data, tools, or approval boundary.
Comparing models before defining the task	General benchmark performance may not predict workflow reliability.
Treating a successful demonstration as a production test	Demonstrations rarely include hostile content, outages, ambiguous records, or constrained credentials.
Letting the model's confidence determine authority	Confidence is not identity, authorization, or approval.
Approving broad access for implementation convenience	Unused permission still expands the failure boundary.
Measuring only time saved	A fast workflow can still create unacceptable operational or data risk.
Evaluating once	Model, prompt, retrieval, tool, and policy changes can invalidate the prior result.

A Practical Enterprise Checklist

Before approving an agent, confirm that the team can answer each item with a current artifact:

The business workflow and owner are named.
The agent's intended output and prohibited decisions are explicit.
The full system and provider boundary is documented.
Required data fields are separated from convenient access.
Identity and authorization are enforced outside the model.
Every tool is inventoried and classified by impact.
High-impact actions remain gated or unavailable.
Representative, negative, and failure tests exist.
Pilot acceptance criteria and stop conditions are measurable.
Audit records connect user, context, policy, action, approval, and result.
Credentials and workflow access can be revoked quickly.
Material changes trigger review and regression testing.
Operations can continue when the agent is disabled.

The AI Agent Blast Radius Mapper can help a team turn the agent's data, tools, credentials, and actions into a concrete review boundary.

Final Point

The right enterprise question is not "Which AI agent should we buy?"

It is:

Should this agent receive this role, with this data and authority, under these controls?

That question is specific enough to test and specific enough to approve. It also gives product teams a path forward. A risky proposal can often become a useful limited pilot when data access narrows, tools become draft-only, approval stays with the responsible person, and operating limits are explicit.

TKOResearch performs principal-led AI Agent Security Assessments for teams making production, procurement, and enterprise deployment decisions.

How to Evaluate AI Agents for Enterprise Use