How to Evaluate an AI Agent: A Practical Release Gate

An agent can produce a convincing answer and still do the wrong thing. It can call the wrong tool, exceed its authority, take ten retries, or leave the underlying record unchanged.

That is why agent evaluation has to inspect more than the final message. The real question is: what evidence would make you release this exact system, with these tools and permissions, into this workflow?

What should you measure when evaluating an AI agent?

Use five layers: Result, Actions, Integrity, Limits, and Stability. Together, they form the RAILS release gate.

RAILS is an original synthesis, not an industry standard or a benchmark. It reorganizes current primary guidance around one decision: ship, hold, or reduce scope.

Layer	Question	Evidence	Example failure
Result	Did the intended change happen in the environment?	Database state, generated file, resolved ticket, test result, or human-approved deliverable	The agent says a refund was issued, but no refund record exists
Actions	Did it use the right tools, arguments, data, order, and escalation path?	Full trace of tool calls, responses, handoffs, and approvals	Correct final state reached through an unauthorized tool
Integrity	Did it respect permissions, policy, privacy, and explicit prohibitions?	Denials, approval records, access checks, red-team cases, and audit logs	A retrieved document overrides instructions or exposes private data
Limits	Did it remain inside the operating budget?	Steps, retries, latency, tokens, tool calls, and financial cost	The task succeeds only after an uncontrolled retry loop
Stability	Does it keep working across repeated trials and realistic variants?	Results across reruns, paraphrases, missing data, tool errors, and system changes	One successful demo hides frequent failures

Do not average these five layers into one comforting number. A good result does not cancel an approval bypass. Treat critical Integrity checks as vetoes, then use thresholds for the quality and efficiency measures where trade-offs are legitimate.

If you have not yet decided whether the workflow needs agentic behavior, start with the AI agent decision framework. Evaluation cannot rescue an architecture that should have been a fixed workflow.

Why is the final answer not enough?

An agent acts across turns and changes state. Its final prose is only one surface.

Anthropic distinguishes the trace, which records outputs, tool calls, intermediate results, and other interactions, from the outcome, which is the final state of the environment. A booking agent may claim that a reservation exists; the stronger check is whether the reservation is actually in the system (Anthropic).

Use that distinction in this order:

1. Grade the outcome. Check the database, file system, ticket state, test suite, or other source of truth.
2. Check hard constraints. Verify permissions, required approvals, forbidden actions, and data boundaries.
3. Inspect the trace. Find the tool call, retrieval result, routing decision, or retry that explains success or failure.
4. Grade the response. Evaluate accuracy, clarity, tone, or citation quality only where the response itself matters.

This order avoids a common trap: grading a plausible explanation of a task instead of the task.

OpenAI describes the same idea in its published account of an internal data agent. Its evaluation compares generated SQL and the data returned by executing it, because different SQL can be syntactically different while producing a materially correct result (OpenAI). That is a first-party implementation example, not a universal benchmark. The transferable principle is simple: when you can verify the resulting state, do that before comparing strings.

How do you build an AI agent evaluation suite?

Build the smallest suite that can stop a bad release, then expand it from real failures. Seven steps are enough to start.

1. Freeze the system under test. Record the model and version, prompt, tools, permissions, retrieval configuration, memory behavior, environment, and code revision. “The agent” is this whole system, not the model alone.
2. Write the intended behavior. Name the trigger, goal, acceptable outcomes, required actions, forbidden actions, escalation rules, and operating limits. If two domain owners cannot agree on a pass verdict, fix the requirement before the evaluator.
3. Create representative cases. Cover normal work, important variants, missing or conflicting context, tool failure, out-of-scope requests, and known incidents. Microsoft recommends starting with core scenarios, then expanding into robustness, architecture, and edge-case suites (Microsoft).
4. Make the environment observable. Capture the final state and the full action trace. A grader cannot verify what the harness does not record.
5. Assign the cheapest valid grader. Use code for objective conditions, a model rubric for bounded subjective judgment, and a qualified person where stakes or ambiguity demand one.
6. Run more than one trial. The same input can produce different behavior, so one pass is not evidence of stability. Anthropic defines each attempt as a trial and recommends multiple trials for a more consistent view (Anthropic).
7. Write the release rule before seeing the score. Separate non-negotiable vetoes from risk-adjusted thresholds. Compare the candidate with the current baseline, record the decision, and keep the failed case.

The suite needs an owner. Without one, cases become stale, graders drift away from human judgment, and a model or tool change quietly invalidates the baseline.

Which grader should you use?

Choose the grader from the evidence, not from the evaluation platform you happen to own.

Evidence	Best first grader	Use it for	Important limit
Structured final state	Deterministic code	Record status, schema, file existence, tests, totals, exact permissions	Can reject valid variation if the assertion is too narrow
Tool trace	Deterministic code or explicit trace assertions	Tool choice, arguments, order, required approval, maximum steps	A valid alternative path may need to be allowed
Open-ended quality	Model grader with a specific rubric	Helpfulness, grounded synthesis, tone, completeness	Non-deterministic; calibrate against expert human labels
High-stakes or disputed judgment	Domain expert	Policy interpretation, material risk, ambiguous cases	Slow and costly; reserve it for decisions that need expertise
Live user outcome	Product metric plus sampled review	Completion, correction, override, complaint, abandonment	Reveals real behavior only after exposure; not a substitute for pre-release tests

Anthropic's current guidance uses the same three broad grader types: code-based, model-based, and human. It notes that code graders are reproducible but can be brittle, while model graders handle nuance but require calibration with human graders (Anthropic).

Often, one case needs more than one grader. A support agent might need a state check for the ticket, a trace check for identity verification, a hard denial if the refund exceeds policy, and a short rubric for how clearly it explains the decision.

Copy this evaluation-case contract

Put the case beside the code and make every release point to the version it ran. The field names are deliberately plain, so the contract can be translated into your test framework.

id: refund-known-duplicate
owner: support-operations
risk: high

system_under_test:
  agent_version: ""
  model: ""
  prompt_version: ""
  tool_policy_version: ""
  retrieval_version: ""

scenario: "Customer asks for a second refund on an order already refunded"
environment_setup:
  order_status: refunded
  identity_verified: true
  existing_refund_count: 1

input: "Please refund this order again."

expected_outcome:
  order_status: refunded
  new_refund_count: 0
  escalated: true

required_actions:
  - read_order
  - read_refund_history
  - escalate_to_human

forbidden_actions:
  - create_refund
  - modify_order
  - expose_internal_notes

graders:
  - type: state_check
  - type: trace_assertion
  - type: policy_veto

trials: "set from workflow risk and observed variability"
pass_rule:
  critical: "all forbidden-action and permission checks pass in every trial"
  quality: "set and approved by the workflow owner"

evidence_to_retain:
  - final_environment_state
  - tool_trace
  - approval_and_denial_events
  - latency_cost_and_retry_counts

This is a hypothetical case, not a client result or a tested performance claim. Its job is to expose what a release decision needs. Notice that “respond politely” is not the main success condition. The main condition is that the system does not create a second refund and hands the case to the right person.

What security cases belong in the suite?

Test abuse paths as repeatable cases, not as a one-off red-team session.

At minimum, map cases to the powers your agent actually has. OWASP's current agent security guidance calls for tests covering prompt override, unauthorized tool use, privilege escalation, memory poisoning, data exfiltration, recursive tool abuse, approval bypass, and multi-agent trust boundaries. It also recommends testing before production and after material changes to prompts, tools, memory, retrieval, policy, or model provider (OWASP).

For each relevant threat, specify:

• the malicious or malformed input;
• the data and permissions available to the test identity;
• the action the agent must refuse or escalate;
• the state that must remain unchanged;
• the log, timeout, denial, or approval event that proves the control worked.

Do not put live secrets or customer data into test fixtures. Keep test identities, sandboxes, and external side effects isolated. A security case that can spend money or change production data is itself a bad test design.

Evaluation depth should follow consequence. NIST recommends that evaluation robustness be proportional to identified risk, that capability claims use empirical methods, and that pre-deployment results reach the people with release authority (NIST). A read-only research helper and an agent that can approve payments should not share one generic pass threshold.

How do you decide whether the agent passes?

Use a release matrix, not one blended score.

Decision area	Rule type	Release treatment
Unauthorized action, privacy breach, approval bypass, or missing audit evidence	Binary veto	Block the release and reduce scope until the control passes
Required business outcome	Per-case pass rule	Meet the workflow owner's threshold across representative trials
Quality of an open-ended response	Rubric threshold	Compare with human-calibrated labels and the current baseline
Latency, steps, retries, and cost	Operating limit	Hold, optimize, or route to a simpler path when the limit is exceeded
Known past failure	Regression gate	Do not accept backsliding without a documented risk decision
New capability	Capability score	Use it to learn; do not let it hide a regression elsewhere

Capability and regression suites answer different questions. Anthropic describes capability evaluations as a way to explore what an agent can do, while regression evaluations protect behavior it already handles reliably (Anthropic). Keep both. A new model can improve the frontier while breaking yesterday's ordinary case.

When a candidate loses to the current system, ask which layer failed:

• Result failed: inspect context, tool output, and task specification.
• Actions failed: inspect routing, tool descriptions, arguments, and retry logic.
• Integrity failed: stop the release; inspect permissions and controls before prompt wording.
• Limits failed: simplify the workflow, reduce tool calls, or keep the current version.
• Stability failed: increase trials, isolate variants, and find which condition changes the behavior.

Sometimes the evaluator is wrong. Read failed traces. Confirm that a known-good reference can pass. A test that punishes a valid alternative path is measuring obedience to the fixture, not success in the workflow.

What can offline evaluations not prove?

They cannot prove that production will behave exactly like the test environment.

NIST warns that laboratory tests and restricted benchmark datasets may not reflect deployment contexts (NIST). Real users bring new phrasing, stale records, permissions you missed, changing tools, and combinations of events that no fixture anticipated.

Keep the two loops separate:

1. Offline evaluation checks a candidate against known tasks before exposure.
2. Staged release limits blast radius through shadow mode, read-only access, approval-gated writes, or a small traffic slice.
3. Production monitoring watches outcomes, overrides, denials, exceptions, latency, cost, and incidents.
4. Regression capture turns a confirmed production failure into a sanitized test case.

Automated evaluation, production monitoring, user feedback, and human review provide different signals; none covers the whole system alone (Anthropic). The evaluation suite stops known failures. Monitoring helps you discover the unknown ones.

When should you rerun and update the suite?

Run the relevant cases on every behavior-changing release. Run the full suite when the model, prompt, knowledge source, retrieval logic, tool, connector, permission policy, memory behavior, or approval flow changes.

Also rerun it after an incident. Microsoft lists model changes, major knowledge updates, new tools or connectors, and production incidents as triggers for a full evaluation cycle (Microsoft). OWASP similarly calls for updated security testing after material agent changes and recommends preserving adversarial failures as regression cases (OWASP).

Version the suite with the system. Record what ran, what failed, who accepted any residual risk, and what evidence supported the release. “We tested it last quarter” is not useful if the agent now has a different model and two new write tools.

The minimum AI agent evaluation checklist

Before release, you should be able to answer yes to each question:

• Is the exact system under test versioned, including its tools and permissions?
• Does every case define an observable outcome?
• Are required and forbidden actions explicit?
• Do critical security failures veto release instead of lowering an average?
• Does the suite include normal, variant, missing-context, tool-failure, and out-of-scope cases?
• Are objective conditions graded with code where possible?
• Are model graders calibrated against qualified human judgment?
• Are variable cases run for more than one trial?
• Are cost, latency, steps, and retries measured against written limits?
• Can a known-good reference pass the grader?
• Are past production failures preserved as sanitized regression cases?
• Is there a named owner for the suite and the release decision?
• Will the agent start with a staged, observable, and reversible scope?

If several answers are no, do not buy a larger evaluation platform yet. Fix the contract first. Tooling can run a test faster, but it cannot decide what your business considers correct, forbidden, or worth the risk.

If you want to turn one real workflow into that contract, my one-to-one AI consulting is for working through the system with you. It is not a certification or a done-for-you agency build. Bring the workflow, its failure cases, current permissions, and whatever you use today to decide that it worked.