Evaluate AI Agents for Enterprise Use

Evaluate the agent in the workflow it will operate, with the data and tools it will receive. A model comparison can help assess response quality. It cannot decide whether an invoice assistant should read bank details or release payments.

The evaluation needs both a usefulness test and an authority boundary. The example below follows an accounts-payable assistant from its proposed task to a limited-pilot decision, with named owners and stop conditions.

An Illustrative Accounts-Payable Agent

Consider an enterprise team evaluating an accounts-payable agent. This is an illustrative scenario, not a description of a TKOResearch client engagement.

The product proposal sounds reasonable. The agent will:

Read invoices received through an approved intake channel.
Compare invoice fields with vendor records and purchase orders.
Check approval history and flag mismatches.
Draft an entry for the enterprise resource planning system.
Explain why it believes an invoice is ready or requires review.

The proposal becomes risky when the boundary is vague. A vendor demonstration may use one broadly privileged ERP service account. The agent may be technically able to create vendors, change bank details, post entries, approve exceptions, or release payments even though the product team only intends it to flag mismatches.

That gap between intended use and available authority is where the evaluation should begin.

For this scenario, three actions are out of scope from the start:

Approving a new vendor.
Changing vendor bank details.
Releasing a payment.

The evaluation is not trying to prove that the agent will never make a mistake. It is deciding whether the workflow remains controlled when the model is wrong, uncertain, manipulated, or unavailable.

1. Define the Workflow and Its Approval Boundaries

Write down the work before discussing model quality.

For the accounts-payable agent, the business outcome might be:

Reduce the time required to reconcile ordinary invoices while preserving the existing approval authority for vendor changes and payment release.

That sentence is more useful than "automate accounts payable." It names the task and preserves the human decision boundary.

The first review should answer five questions:

Question	Accounts-payable example
What starts the workflow?	An invoice arrives through the approved intake queue.
What does the agent produce?	A mismatch flag, source references, and a draft ERP entry.
Who relies on the result?	Accounts-payable staff and the assigned approver.
What must remain a human decision?	Vendor creation, bank-detail changes, exceptions, and payment release.
What would make the workflow unsuccessful?	Incorrect posting, missed duplicate invoices, cross-entity data access, or an unauditable recommendation.

Assign responsibility for workflow quality, security controls, the runtime and manual fallback. In a small organization, one person may hold several roles. Each decision still needs an owner with the knowledge and authority to make it.

2. Define the Complete System Boundary

An agent is more than the model endpoint. The review boundary should include every component that can affect what the system sees, decides, or does.

For the example, the boundary contains:

Invoice intake
  -> document parser
  -> agent orchestrator
  -> model provider
  -> vendor and purchase-order retrieval
  -> mismatch rules
  -> ERP draft-entry tool
  -> approval queue
  -> audit and monitoring systems

The system also depends on identities, credentials, prompts, tool definitions, retrieval filters, memory, policy code, queues, and vendor configuration. If one of those changes, the evaluated system may have changed even when the model name has not.

Use a boundary table that records ownership and trust, not only architecture:

Component	Owner	Data handled	Trust concern	Required control
Invoice intake	Finance operations	Invoice and sender data	Sender-controlled content	Approved channel, malware handling, source tracking
Document parser	Platform	Extracted invoice fields	Parsing errors and hidden content	File limits, field validation, retained source reference
Model provider	Platform	Selected invoice context	External dependency and retention	Enterprise terms, configured retention, provider inventory
Vendor retrieval	ERP team	Vendor master data	Excess data and cross-entity access	Entity-scoped query, field allowlist, user attribution
Draft-entry tool	ERP team	Proposed accounting entry	Unauthorized write or wrong ledger	Draft-only role, typed fields, deterministic policy checks
Approval queue	Finance operations	Recommendation and source references	Approval confusion	Named approver, clear preview, parameter-bound approval
Audit store	Security operations	Action and policy records	Sensitive-data overcollection	Structured records, redaction, retention, access control

If the team cannot draw this boundary and name an owner for each part, it is not ready to approve production access.

3. Decide What Data the Agent May Receive

Data review should be field-specific. "ERP access" and "invoice access" are too broad to support a decision.

For each data source, record:

The business reason for access.
The fields required for the task.
The user, legal entity, tenant, or region boundary.
Whether the source contains instructions or text controlled by another party.
How long the data remains in prompts, traces, memory, and provider systems.
Whether the system can function with less data.

The illustrative agent needs invoice amount, currency, vendor identifier, purchase-order reference, line items, and approval status. It does not need vendor bank-account numbers to detect a price or quantity mismatch. Excluding those fields reduces both exposure and the consequences of a bad output.

A practical data decision might look like this:

Data	Decision	Reason
Invoice fields	Allow	Required for reconciliation
Purchase-order lines	Allow	Required for quantity and price comparison
Vendor name and internal ID	Allow	Required for matching
Vendor bank details	Deny	Not required for the approved workflow
Prior invoice notes	Allow with limits	Useful for known exceptions, but may contain untrusted text
Other legal entities' records	Deny	Outside the assigned entity boundary
Full email thread	Deny by default	Broader and less structured than the task requires

Authorization must happen before restricted content enters model context. A prompt telling the model not to reveal another entity's invoices is not an access-control mechanism.

4. Map Identity, Credentials, and Authorization

Ask whose authority the agent is using at each step.

There are usually three distinct identities:

The initiating user. The person who starts or reviews the workflow.
The agent workload. The service identity used by the orchestration layer.
The downstream actor. The identity recorded by the ERP, ticketing, email, or other target system.

These should not collapse into one broad service account.

For the accounts-payable pilot, the agent can use a workload identity that reads records for one legal entity and creates draft entries in a designated queue. The request must also carry the initiating user's identity so policy and audit records remain attributable. The ERP role cannot create vendors, alter payment details, approve an entry, or release funds.

Reviewers should ask:

Owner	Question
Security	Can a user or agent reach data outside the initiating user's assigned entity?
Platform	Where are credentials issued, scoped, rotated, and revoked?
Product	Does the interface clearly distinguish a draft from an approved transaction?
Finance operations	Who owns the queue when the agent cannot reconcile an invoice?
Legal and privacy	Which providers receive invoice or vendor data, and under what terms?
Procurement	Can the vendor change models, retention, or subprocessors without notice?

The model can recommend an action. It should not establish the identity or authorization under which that action runs.

5. Classify Tools and Action Authority

List every tool. Do not stop at the tools shown in the user interface. Include internal APIs, retrieval functions, queues, message senders, administrative endpoints, and maintenance functions available to the runtime.

Classify each action by what it changes and how difficult it is to reverse:

Tool action	Class	Pilot decision	Required control
Read purchase order	Read	Allow	Entity-scoped query and field allowlist
Search vendor record	Read	Allow	Exact-match preference and entity boundary
Flag mismatch	Reversible write	Allow	Source references and user attribution
Draft ERP entry	Reversible write	Allow with review	Typed schema, policy checks, draft-only status
Create vendor	Administrative write	Deny	Outside the approved workflow
Change bank details	High-impact write	Deny	Separate established business process
Release payment	Financial action	Deny	Human approval in the ERP system

This classification should drive the technical permission set. It is not enough for a product requirement to say the agent "will not" release payment when its credential can call that operation.

For a deeper implementation treatment, see How to Give AI Agents Access to Tools Securely.

6. Test the Complete Workflow

Evaluation should use the records, edge cases, and failure conditions the agent will encounter in the actual process.

A useful test set for this scenario includes:

Ordinary invoices with clean purchase-order matches.
Partial shipments and split invoices.
Duplicate invoices with minor formatting differences.
Currency and tax variations.
Vendors with similar names.
Missing purchase-order references.
Conflicting approval history.
Scanned documents with weak extraction quality.
Invoice notes that contain instructions addressed to the agent.
Records from another legal entity that must remain inaccessible.
A model timeout after a draft has been requested.
A repeated request with the same invoice and idempotency key.

Measure the whole workflow:

Measure	What it answers
Correct reconciliation rate	Does the agent reach the right operational conclusion?
False-clear rate	How often does it mark a problematic invoice as ready?
Escalation quality	Does it route uncertain cases with useful context?
Unauthorized-access denials	Do identity and entity boundaries hold under negative tests?
Draft accuracy	Are account, amount, currency, and references correct?
Duplicate prevention	Can retries create more than one draft?
Reviewer correction rate	How often must staff materially change the result?
Time saved per accepted case	Does the workflow provide enough value to justify its controls and operating cost?

One aggregate accuracy number hides the failures that matter. A system can perform well on common invoices and still be unsuitable if it occasionally crosses an entity boundary or creates duplicate drafts.

NIST's AI Risk Management Framework 1.0 organizes AI risk work around Govern, Map, Measure, and Manage. The framework is voluntary; NIST lists its revision as work in progress as of September 9, 2026. The organization must define the context, measure the system, assign responsibility, and manage it after deployment.

7. Evaluate Failure Handling and Operational Control

Ask what happens when the system cannot complete the task safely.

Agree on the pilot's failure behavior before testing:

If document extraction is uncertain, route the invoice for manual review.
If the vendor match is ambiguous, do not select one.
If the policy service is unavailable, do not create a draft.
If the model times out, preserve the current state and avoid an automatic write retry.
If duplicate detection fails, stop the workflow and alert the owner.
If the wrong entity appears in retrieved context, terminate the session and preserve the relevant trace identifiers.

The system also needs an operational stop path. A product owner should be able to disable the agent workflow without waiting for a code deployment. Platform teams should be able to revoke its credential. Finance operations should have a documented way to take over queued work. Security should know which records support technical review.

Include these operating requirements in the pilot acceptance criteria.

8. Review Provider and Change Dependencies

An enterprise agent depends on more than its own code. Record the model provider, model version or alias, orchestration framework, retrieval services, tool servers, and important data processors.

Review:

Provider retention and training settings.
Supported regions and data-transfer terms.
Subprocessors and notification terms.
Model-update controls and deprecation policy.
Availability commitments and rate limits.
Security reporting channels.
Export and deletion capabilities.
Contract terms for customer content and generated output.
How the team will re-test after a model, prompt, tool, retrieval, memory, or policy change.

NIST SP 800-218A, published in July 2024, provides AI-specific practices that supplement the Secure Software Development Framework. Enterprise buyers should treat the acquired agent and its dependencies as software with a lifecycle, not a fixed appliance that was evaluated once.

9. Record the Proposed Boundary

A small machine-readable manifest can force useful precision. This is an example artifact, not a substitute for architecture review or testing.

agent: invoice-reconciliation-assistant
workflow_owner: finance-operations
business_outcome: reduce manual reconciliation time for ordinary invoices
data_classes:
  - invoice-fields
  - purchase-order-lines
  - vendor-id-and-status
  - approval-history
identities:
  initiating_user: required
  workload: ap-agent-draft-writer
tools:
  - read_invoice
  - read_purchase_order
  - find_vendor
  - create_draft_erp_entry
allowed_actions:
  - compare invoice with purchase order
  - flag mismatch
  - create draft entry in the AP review queue
prohibited_actions:
  - create or approve vendor
  - read or change vendor bank details
  - approve entry
  - release payment
approval_gates:
  draft_entry: assigned AP reviewer
model_dependency:
  provider: approved-enterprise-provider
  version: recorded-in-each-run
logging:
  - initiating-user
  - source-record-ids
  - retrieved-record-ids
  - proposed-entry
  - policy-result
  - reviewer-decision
  - downstream-result
kill_switch:
  owner: finance-platform-on-call
  action: disable workflow and revoke ap-agent-draft-writer

Version this record with the agent configuration. When a proposed change expands a tool, data class, or allowed action, require a new review.

10. Record the Outcome and Operating Conditions

The review should end with one of three outcomes:

Outcome	Meaning
Proceed	The workflow and controls are ready for the approved operating scope.
Limited pilot	The system may operate within named constraints while the team measures specific open questions.
Do not deploy	A blocker makes the proposed use unacceptable until the design changes.

For the illustrative accounts-payable agent, a reasonable decision record is:

Decision: Limited pilot

Approved scope: One legal entity, approved invoice intake, read-only vendor and purchase-order access, mismatch flags, and draft entries in a separate review queue.

Prohibited scope: Vendor creation, bank-detail access or change, exception approval, payment approval, and payment release.

Acceptance criteria:

No successful cross-entity retrieval in the negative test suite.
No duplicate draft from retries or repeated invoice intake.
Every draft includes the initiating user, source record identifiers, policy result, and model version.
False-clear and reviewer-correction rates remain below limits set by finance operations before the pilot starts.
Staff can disable the workflow and resume manual processing within the documented recovery target.

Stop conditions:

Any unauthorized data access.
Any attempt to call a prohibited action.
Any duplicate draft caused by workflow retry.
Missing audit fields on a write attempt.
A provider or configuration change that has not completed regression testing.

Owners: Finance operations owns workflow quality. The ERP team owns permissions and draft-entry policy. Platform owns runtime reliability and credential lifecycle. Security owns abuse-case testing and control review. Product owns user experience and pilot measurement.

This record identifies what the pilot may do, who owns it, and which events require it to stop.

Check the decision against the proposed access

A familiar vendor and a strong model benchmark do not define this agent's permissions. Compare the pilot record with the actual credentials, data fields and tool registry. Remove unused authority before asking users to rely on the workflow.

Measure correction, escalation and operating work alongside time saved. A fast draft that routinely needs repair can make the overall process slower. A high average score also cannot compensate for a cross-entity disclosure or an unauthorized payment path.

Walk through the record with the operating team

Have the workflow owner identify a permitted invoice, a denied operation and a failure that returns work to staff. Ask the operator to show the disable procedure and the logs that distinguish a draft from an approved transaction.

Resolve missing ownership, inaccessible artifacts or ambiguous stop conditions before the pilot begins. The AI Agent Blast Radius Mapper can help organize the data, tools and credentials for that discussion.

Preserve the decision when the system changes

The pilot record should identify the approved data, tools, users and operating limits. When the agent gains a new connector or a draft tool gains a send operation, reopen that decision even if the product keeps the same name.

For the accounts-payable example, a useful first pilot leaves payment authority in the existing finance process and measures whether staff accept the drafts with less total effort. Include time spent correcting, escalating and operating the agent. If those costs outweigh the benefit, narrowing or stopping the pilot is a valid result.

How to Evaluate AI Agents for Enterprise Use