Agent Evaluation-Driven Development: How to Test AI Agents Before Production
Introduction: How do you know your AI agent actually works?
An AI agent can answer questions, call APIs, search databases, update records, and execute multi-step workflows.
In a demonstration, it might look impressive. It completes a task, explains what it did, and produces a convincing final response.
But what happens when the same agent receives an ambiguous request, encounters an unavailable tool, retrieves incorrect information, or attempts an action it should not perform?
A successful demonstration does not establish production reliability.
Traditional software testing often checks whether a function returns an expected result for a given input. AI agents introduce additional complexity: their outputs can vary, they may choose different execution paths, and their actions can change external system state.
A customer-support agent, for example, might correctly identify a refund request but call the wrong API, update the wrong customer record, or claim that a refund was processed when the transaction failed.
Testing only its final response would miss these failures.
This is where agent evaluation, commonly called agent evals, becomes essential.
Agent evaluation is the systematic process of measuring whether an agent performs its intended tasks correctly, safely, efficiently, and consistently.
Instead of relying on demonstrations or subjective impressions, teams define success criteria, create representative test cases, execute the agent repeatedly, and grade the results.
The goal is straightforward:
Do not ship an AI agent because it appears intelligent. Ship it because its behavior has been tested against clearly defined requirements.
1. Why testing AI agents is different from testing traditional software
Consider a conventional function:
def calculate_total(price, quantity):
return price * quantity
For a given input, the expected result is deterministic.
assert calculate_total(10, 3) == 30
The same inputs should produce the same result, assuming the function and environment have not changed.
Now consider an AI customer-support agent.
A user asks:
I received the wrong product. Can you arrange a replacement?
The agent may need to:
- Understand the request.
- Identify the customer.
- Retrieve the relevant order.
- Check the replacement policy.
- Determine product availability.
- Call the replacement API.
- Verify that the replacement was registered.
- Explain the result.
Each step introduces possible failure modes.
The agent might retrieve the wrong order, misunderstand the policy, select an incorrect tool, provide invalid parameters, or stop before verifying the operation.
Even if the final response sounds correct, the actual business operation may have failed.
This creates four important differences between traditional software testing and agent evaluation.
| Traditional software | AI agent |
|---|---|
| Often deterministic execution | Potentially variable execution |
| Function-level assertions | Task and workflow-level evaluation |
| Expected output is frequently exact | Success may require semantic or state-based grading |
| Tests often focus on code behavior | Tests must also assess decisions, actions, and outcomes |
This does not mean traditional testing becomes unnecessary.
Unit tests, integration tests, API tests, and security checks remain essential. Agent evaluation complements them by testing the behavior of the complete AI-powered system.
2. What exactly should an agent evaluation measure?
A common mistake is to evaluate an agent using only its final response.
For production systems, success should be measured across several dimensions.
2.1 Task completion
Did the agent achieve the requested objective?
For a booking agent, success means a valid booking exists—not merely that the agent said the booking was completed.
For a database assistant, success means the intended record was correctly updated.
Task completion should be grounded in verifiable outcomes whenever possible.
2.2 Output correctness
Was the information returned to the user accurate?
Examples include:
- Correct customer information.
- Accurate calculations.
- Valid summaries.
- Relevant answers.
- No unsupported claims.
Output correctness is especially important for research, customer support, and information-retrieval agents.
2.3 Tool selection and execution
Did the agent choose the appropriate tool and provide valid arguments?
An agent may have access to several tools:
- Search customer.
- Retrieve order.
- Check inventory.
- Process refund.
- Create replacement.
A correct answer does not necessarily mean the correct tools were used.
Tool evaluation should consider whether the chosen tool was appropriate, whether its parameters were valid, and whether the resulting action was successful.
2.4 Instruction following and policy compliance
Did the agent respect its instructions and operational constraints?
For example:
- Did it request approval before a consequential action?
- Did it refuse to disclose protected information?
- Did it avoid accessing records outside the user's permissions?
- Did it escalate cases that required human intervention?
2.5 Efficiency
Did the agent complete the task without unnecessary operations?
A workflow that requires 20 tool calls to perform an operation that normally requires three may be functionally correct but operationally inefficient.
Measure:
- Number of model calls.
- Number of tool calls.
- Execution duration.
- Token consumption.
- Cost per successful task.
- Number of retries.
2.6 Robustness
Does the agent continue to behave correctly when conditions change?
Test situations such as:
- Missing information.
- Incorrect user input.
- Unavailable APIs.
- Empty database results.
- Conflicting instructions.
- Unexpected tool responses.
- Interrupted execution.
A reliable agent should not only perform well in ideal conditions. It should also handle predictable failures safely.
3. The building blocks of an evaluation system
An evaluation system can be understood through five basic components.
3.1 Tasks
A task is one scenario that the agent must complete.
Example:
A customer asks to replace an item from an order. The item is eligible, but the customer has not provided an order number.
The task should define the starting conditions and expected behavior.
3.2 Trials
A trial is one execution of a task.
Because agent behavior can vary between runs, executing a scenario once may not provide enough evidence.
The same task can be executed multiple times to estimate consistency.
For example:
- Task A: 10 trials.
- Task B: 10 trials.
- Task C: 10 trials.
This helps reveal whether an agent succeeds consistently or only occasionally.
3.3 Graders
A grader evaluates a particular aspect of the agent's behavior.
It might check:
- Whether the final answer is correct.
- Whether the database contains the expected record.
- Whether the correct tool was selected.
- Whether the agent respected an approval requirement.
- Whether the workflow stayed within a cost threshold.
One task can have multiple graders.
3.4 Traces
A trace is the record of an agent's execution.
Depending on the framework, it can contain:
- User input.
- Model requests and responses.
- Tool calls.
- Tool results.
- Agent handoffs.
- Intermediate outputs.
- Errors.
- Execution timing.
A trace helps developers understand not only whether an agent failed, but where and how the failure occurred.
3.5 Evaluation datasets
A dataset is a collection of test cases used to evaluate the agent repeatedly.
It becomes the reference point for comparing:
- Different prompts.
- Different models.
- Different agent architectures.
- Different tool configurations.
- New software versions.
Together, these components form the foundation of an evaluation harness: the infrastructure that executes test scenarios, records traces, applies graders, and aggregates results.
Anthropic's January 2026 evaluation guide provides a particularly useful explanation of tasks, trials, graders, transcripts, outcomes, and evaluation harnesses.
4. How to evaluate an agent: Four complementary approaches
Not every evaluation requires an AI judge. In fact, deterministic checks should generally be preferred whenever the expected result can be verified directly.
4.1 Deterministic evaluation
Deterministic graders use explicit rules, assertions, or reference values.
Example:
def check_refund_result(result, database):
assert result["status"] == "success"
refund = database.get_refund(
result["refund_id"]
)
assert refund is not None
assert refund["status"] == "completed"
assert refund["amount"] == 100
This checks whether the refund actually exists and has the expected properties.
Other examples include:
- HTTP response status.
- Database state.
- Required fields.
- Valid JSON.
- Exact tool parameters.
- Code execution results.
- Policy-based permission checks.
Advantages
- Reproducible.
- Relatively inexpensive.
- Easy to automate.
- Suitable for objective requirements.
Limitations
- Difficult to use when multiple valid answers exist.
- Cannot independently assess subjective quality.
- May become brittle when the task allows legitimate alternative execution paths.
Use deterministic graders for facts that can be verified directly.
4.2 LLM-as-a-judge
Some outputs cannot be evaluated using exact string matching.
Consider an agent that summarizes a customer complaint. There may be many acceptable summaries, each with different wording.
An LLM judge can evaluate the response against a structured rubric.
Example:
Evaluate the agent's response.
Criteria:
1. Does it accurately identify the customer's issue?
2. Does it preserve important details?
3. Does it avoid inventing information?
4. Does it provide an appropriate next step?
Score each criterion from 1 to 5.
Return:
- Individual scores
- Overall assessment
- Explanation for each score
This is useful for evaluating:
- Relevance.
- Clarity.
- Instruction following.
- Summary quality.
- Tone.
- Semantic correctness.
- Complex task behavior.
However, LLM judges are not ground truth.
They can make mistakes, favor particular writing styles, overlook subtle factual errors, or produce inconsistent scores.
A responsible evaluation process should:
- Define explicit scoring criteria.
- Use examples of good and bad outputs.
- Compare judge results with human assessments.
- Track disagreement.
- Recalibrate the judge when necessary.
For important decisions, use multiple evaluation signals rather than relying on a single model's opinion.
4.3 Human evaluation
Human evaluation is particularly useful when quality depends on business context, expert judgment, or complex user expectations.
For example, a legal research agent might retrieve relevant documents and produce a technically coherent answer. A qualified reviewer must still determine whether the interpretation is appropriate.
Human reviewers can evaluate:
- Factual accuracy.
- Completeness.
- Relevance.
- Policy compliance.
- Business suitability.
- Whether an answer is genuinely useful.
Human evaluation is expensive and difficult to scale, so it should be focused on important cases.
It is also valuable for calibrating automated graders.
4.4 Pairwise evaluation
Pairwise evaluation compares two versions of an agent on the same task.
Instead of asking whether a response deserves a score of 4 or 5, the evaluator asks:
Which version performed better?
Example:
- Version A uses GPT model X.
- Version B uses GPT model Y.
- Both receive the same task and environment.
- Their results are compared against the same criteria.
Pairwise comparison is useful when testing:
- Different models.
- Prompt changes.
- Alternative architectures.
- Tool-selection strategies.
- Different retrieval configurations.
It does not replace objective correctness checks. Two versions can be compared even if both are wrong.
5. Evaluate the outcome and the execution path
An important distinction in agent evaluation is the difference between the final outcome and the execution trajectory.
Consider a flight-booking agent.
The user requests a flight within a specific budget.
The agent:
- Searches for available flights.
- Finds a suitable option.
- Checks the booking policy.
- Submits a reservation.
- Receives a successful confirmation.
There are two separate questions:
Outcome evaluation: Was the correct flight actually booked?
Trajectory evaluation: Did the agent use the appropriate search and booking tools, respect the constraints, and avoid unauthorized actions?
Both matter.
An agent might reach the correct outcome through an unnecessarily expensive or risky path. Another might use an unusual but valid execution path that a rigid test would incorrectly reject.
This leads to a critical design principle:
Evaluate what matters about the execution, not necessarily every exact step the agent takes.
Some constraints should be strict:
- Never book without required approval.
- Never exceed an authorized budget.
- Never access another user's account.
Other behaviors can allow flexibility:
- The order of independent read-only searches.
- Alternative valid ways to retrieve information.
- Different but acceptable wording.
- Equivalent tool sequences that achieve the same result safely.
Anthropic's evaluation guidance highlights this problem: excessively rigid graders can penalize valid solutions simply because they differ from the test designer's expected sequence.
6. Build an evaluation dataset from real tasks
A good evaluation dataset should represent the situations an agent will encounter in production.
Starting with five ideal examples is not enough.
For a customer-support agent, a useful dataset might contain the following categories:
| Category | Example scenario | Expected behavior |
|---|---|---|
| Normal operation | Customer requests an eligible replacement | Complete replacement |
| Missing information | Customer provides no order number | Ask for clarification |
| Invalid request | Order does not exist | Explain the issue |
| Policy restriction | Item is not eligible for replacement | Refuse the operation and explain |
| API failure | Replacement service is unavailable | Report failure and avoid false confirmation |
| Ambiguous request | Customer asks to replace one of several items | Clarify which item |
| Authorization | User requests another customer's information | Deny unauthorized access |
| Adversarial input | User attempts to override system instructions | Preserve the intended policy |
This dataset should grow over time.
A particularly valuable source is production traces.
When a real agent fails:
- Identify the failure.
- Remove unnecessary sensitive information.
- Convert the scenario into a reproducible test case.
- Define the expected behavior.
- Add it to the evaluation dataset.
- Run it against future versions.
This turns real-world failures into permanent regression tests.
Avoid building a dataset that only contains easy examples
A dataset should include:
- Common requests.
- Unusual but legitimate requests.
- Boundary conditions.
- Ambiguous instructions.
- Missing information.
- Tool failures.
- Security-related scenarios.
- Previously observed failures.
Otherwise, the agent may achieve a high score without being prepared for real-world conditions.
7. Design a practical evaluation workflow
An effective evaluation process does not require a sophisticated platform from day one.
The following workflow can be implemented progressively.
Step 1: Define success before implementation
Start with the actual business objective.
For a customer-support agent, success might mean:
- Identify the correct customer.
- Retrieve the correct order.
- Apply the replacement policy.
- Complete the replacement when authorized.
- Never claim success when the operation fails.
Turn each requirement into an observable assertion or grading criterion.
Step 2: Create a representative dataset
Build a collection of scenarios covering normal operation, edge cases, errors, and policy restrictions.
Store:
- Test input.
- Initial environment state.
- Expected outcome.
- Evaluation criteria.
- Relevant metadata.
Version the dataset so that changes in evaluation coverage are visible.
Step 3: Run the agent in a controlled environment
Use a staging environment or simulated services.
Avoid testing destructive actions against real production data.
The environment should provide realistic responses while allowing the team to inspect and reset state between trials.
Step 4: Capture traces
Record the complete execution.
At minimum, capture:
- Model and prompt version.
- Tool names and arguments.
- Tool results.
- Execution errors.
- Final response.
- Task outcome.
- Duration and token usage.
Traces are essential for diagnosing failures that cannot be understood from the final response alone.
Step 5: Apply multiple graders
Use deterministic checks for objective requirements, LLM judges for semantic criteria, and human reviewers for high-impact or ambiguous cases.
Do not force every requirement into one overall score.
A single score can hide important weaknesses. An agent that scores highly on response quality but fails authorization checks should not be considered production-ready.
Step 6: Repeat trials
Execute stochastic tasks multiple times.
For example, if a task is tested 20 times and succeeds 16 times:
[ \text{Success rate} = \frac{16}{20} \times 100 = 80% ]
This is more informative than reporting one successful demonstration.
For critical tasks, use more trials and confidence intervals where appropriate. Twenty trials can reveal variability, but it is not necessarily enough to establish a precise reliability estimate.
Step 7: Compare versions and detect regressions
Run the same dataset against:
- The current production version.
- The proposed version.
- Alternative models.
- Different prompts or tool configurations.
Compare success rates, safety violations, latency, and cost.
A change that improves answer quality but increases unauthorized tool calls should not be treated as an overall improvement.
Step 8: Integrate evaluations into release decisions
Before deployment, require the agent to meet defined quality thresholds.
For example:
- Task success above a defined threshold.
- Zero critical authorization violations in the test suite.
- No regression in important scenarios.
- Cost and latency within operational limits.
The thresholds should reflect business risk. There is no universal score that makes every agent production-ready.
8. The metrics that matter
A useful evaluation dashboard should expose multiple dimensions instead of presenting one aggregate score.
| Metric | What it measures | Example |
|---|---|---|
| Task success rate | Percentage of tasks completed correctly | 92% |
| Tool correctness | Appropriate tool selection and execution | 96% |
| Argument correctness | Valid tool parameters | 98% |
| Policy compliance | Respect for operational rules | 100% in tested cases |
| Failure recovery | Appropriate handling of failures | 90% |
| Average latency | Time required to complete a task | 4.2 seconds |
| Cost per successful task | Total cost divided by successful completions | $0.04 |
| Regression rate | Previously passing cases that now fail | 2% |
The numbers above are illustrative, not industry benchmarks.
Task success rate
[ \text{Task success rate} = \frac{\text{Successful trials}}{\text{Total trials}} \times 100 ]
This is usually the most important top-level metric, but its usefulness depends on the quality of the test dataset and the definition of success.
Tool correctness
[ \text{Tool correctness} = \frac{\text{Correct tool calls}}{\text{Evaluated tool calls}} \times 100 ]
The definition of correctness must be explicit. A tool call can have the correct name but incorrect parameters.
Cost per successful task
[ \text{Cost per successful task} = \frac{\text{Total evaluation cost}}{\text{Successful tasks}} ]
This is more meaningful than comparing the cost of individual model calls.
An agent that costs less per request but requires more retries may be more expensive overall.
Safety violations
Track critical policy failures separately.
For example:
- Unauthorized data access.
- Unapproved financial operations.
- Destructive actions without authorization.
- Disclosure of protected information.
Averages should never hide critical failures. A single severe security violation can matter more than dozens of successful routine interactions.
9. Tools and frameworks for agent evaluation
Several tools now support evaluation workflows. They differ in scope, architecture, integrations, and deployment model.
****
Good starting point for developers
An open-source, Python-oriented evaluation framework with test cases, datasets, metrics, trajectory evaluation, and CI/CD integration. Useful for teams that want evaluation close to their code and test suites.
·
****
Supports offline and online evaluations, datasets, human feedback, experiment comparisons, and LLM-as-a-judge. It is not limited to LangChain-based applications.
Useful for teams that want evaluation, tracing, and collaborative review in one platform.
·
****
An open-source platform for tracing, experimentation, datasets, and evaluation. Its OpenTelemetry-based instrumentation and integrations make it relevant for teams using different agent frameworks.
Useful when teams want an open-source, self-hostable evaluation and observability foundation.
·
****
Supports prompt and model comparisons, test cases, automated checks, and adversarial red teaming. Particularly useful for testing security and policy weaknesses alongside functional quality.
·
****
OpenAI's agent evaluation documentation covers workflow traces, graders, datasets, evaluation runs, and trace grading.
A useful option for teams building with the OpenAI agent stack. Check the current API and platform lifecycle before choosing a dependency: OpenAI's older Evals platform is scheduled for shutdown on November 30, 2026, according to its transition notice.
· ·
Which one should you choose?
| Requirement | Starting point |
|---|---|
| Python-based evaluation and CI tests | DeepEval |
| Tracing, datasets, and collaborative evaluations | LangSmith |
| Open-source and self-hosted observability/evaluation | Arize Phoenix |
| Prompt testing and adversarial security checks | Promptfoo |
| OpenAI agent workflow evaluation | OpenAI Agent Evals |
These products overlap, but they are not direct substitutes in every deployment. A team may combine a test framework with an observability platform and a dedicated red-teaming tool.
The right choice depends on the agent architecture, data requirements, deployment constraints, and existing engineering stack.
10. Evaluation-driven development: Make testing part of the agent lifecycle
The most important shift is to stop treating evaluation as a final quality check.
In traditional development, a team might build a feature and then write tests to verify it.
In evaluation-driven development, the team defines the expected behavior before or alongside implementation.
The process becomes:
- Define the capability the agent should provide.
- Create test cases representing expected and problematic situations.
- Define measurable success criteria.
- Implement the agent.
- Run the evaluation suite.
- Inspect failed traces.
- Improve the agent.
- Repeat until the required quality threshold is reached.
- Run the suite before deployment and after significant changes.
This approach changes the development feedback loop.
Instead of asking, "Does the agent seem better?", the team asks, "Did the change improve the measured behavior without introducing regressions?"
Evaluation suites also make model upgrades easier to assess. A new model may be more capable, but it may also change tool-selection behavior, increase costs, or perform worse on specific business scenarios.
Running the same evaluation suite against both versions provides evidence for the upgrade decision.
Evaluation is not a one-time activity
Production data introduces new situations that may not exist in the original dataset.
A healthy evaluation program continuously incorporates:
- New user requests.
- Newly discovered failure modes.
- Security incidents.
- Model behavior changes.
- New tools and integrations.
- Changes in business policies.
Evaluation datasets therefore need ownership, versioning, and regular maintenance.
A benchmark that never changes can become an unreliable measure of a system that does.
11. Common mistakes when evaluating AI agents
Even a sophisticated evaluation framework can produce misleading results if its tests are poorly designed.
Mistake 1: Testing only successful scenarios
An agent that handles ordinary requests may still fail when information is missing or an API is unavailable.
Include edge cases and failure scenarios.
Mistake 2: Trusting the final response
An agent may say an operation succeeded even when the underlying system rejected it.
Verify external state independently.
Mistake 3: Relying entirely on LLM judges
Model-based graders can be inconsistent and biased.
Use deterministic checks where possible and calibrate LLM judges against human assessments.
Mistake 4: Evaluating only once
One successful execution does not establish reliability.
Repeat stochastic tasks and measure variability.
Mistake 5: Overly strict execution-path checks
A valid agent may find a different route to the same outcome.
Enforce strict requirements only where the exact behavior matters, such as authorization or required approvals.
Mistake 6: Using public benchmarks as proof of production readiness
Benchmarks can compare capabilities under defined conditions, but they do not necessarily represent a company's real workflows, data, policies, or risk profile.
Build task-specific evaluations in addition to using public benchmarks.
Mistake 7: Optimizing a single score
A high average score can conceal serious problems.
Track task completion, security, reliability, latency, and cost separately.
Conclusion: Reliable agents are evaluated, not assumed
AI agents introduce a different kind of software engineering challenge.
Their behavior depends not only on code, but also on models, prompts, tools, context, external systems, and multi-step decisions.
Traditional software tests remain necessary, but they are not sufficient to establish whether an agent performs its intended tasks reliably.
A mature evaluation process combines:
- Representative test datasets.
- Deterministic assertions.
- LLM-based graders.
- Human assessment.
- Trace and trajectory analysis.
- Repeated trials.
- Regression testing.
- Production feedback.
The objective is not to obtain the highest possible benchmark score.
It is to establish measurable evidence that an agent can perform its intended job, respect its constraints, recover from failures, and remain dependable as the system evolves.
Evaluation-driven development turns agent reliability from an assumption into an engineering process.
Sources and further reading
-
Anthropic (January 2026) — Demystifying evals for AI agents. A practical guide to evaluation design, tasks, trials, graders, traces, outcomes, and evaluation harnesses.
https://www.anthropic.com/research/demystifying-evals-for-ai-agents -
Yehudai et al. (ACL 2026) — A Survey on Evaluation of LLM-based Agents. Academic survey covering agent evaluation, benchmarks, capabilities, and open research gaps.
https://aclanthology.org/2026.findings-acl.1330/ -
Pan et al. (ICLR 2026) — Measuring Agents in Production. Empirical study of production agent development and evaluation, based on 20 case studies and a survey of 306 practitioners.
https://research.ibm.com/publications/measuring-agents-in-production -
OpenAI Developers — Evaluate agent workflows. Documentation on traces, graders, datasets, and agent workflow evaluation.
https://developers.openai.com/api/docs/guides/agent-evals -
OpenAI Developers — Evaluation best practices. Guidance on task-specific evaluations, regression testing, and human calibration.
https://developers.openai.com/api/docs/guides/evaluation-best-practices -
DeepEval — AI Agent Evaluation Quickstart. Practical documentation for end-to-end and component-level agent evaluation.
https://deepeval.com/docs/getting-started-agents -
LangSmith — Evaluations. Documentation on offline and online evaluation, human feedback, and experiment comparison.
https://www.langchain.com/langsmith/evaluation -
Arize Phoenix — Open-source AI observability and evaluation platform.
https://arize.com/phoenix/ -
Promptfoo — Open-source evaluation and red-teaming framework.
https://www.promptfoo.dev/