AI agent evaluation metrics are a framework of criteria for quantitatively measuring the quality and performance of AI agents and Agentic Workflows that autonomously execute tasks.

Unlike conventional AI evaluation, which measures simple response accuracy, agent evaluation combines multidimensional metrics—including the accuracy of tool calls, task completion rates across multiple steps, and the balance between cost and latency—to make assessments.

This article is intended for practitioners in AI BPO and business automation, and covers the following topics.

If the only question were whether the AI returned the "correct answer," conventional evaluation methods would suffice. However, the emergence of agents that autonomously advance tasks while calling multiple tools has undermined that premise.

Agentic Workflow Evaluation (AWE) is an evaluation framework designed to measure the quality of such autonomous workflows from multiple angles. Its fundamental difference from conventional methods lies in evaluating not just single-response accuracy, but the appropriateness of tool calls, the chaining between steps, and the entire flow through to final task completion as a continuous sequence. The following sections explore these differences in greater detail and examine what is evaluated and how.

Conventional AI evaluation was primarily designed around the accuracy axis of "can it return the correct response to a single question?" Typical examples include matching output text using BLEU scores or F1 scores, or measuring classification task accuracy.

It is tempting at first to assume that agents can be measured with the same metrics, but in practice, the true state of quality cannot be understood without evaluating the autonomous action sequence spanning multiple steps.

The main differences between conventional evaluation and agent evaluation are as follows:

• Unit of evaluation: Conventional evaluation targets a single turn of "1 input → 1 output." Agent evaluation targets the entire multi-turn trajectory of "planning → tool call → state update → next action."
• Definition of success: Conventional evaluation focuses on the accuracy of output text. Agent evaluation requires both the final goal achievement (Agent Goal Accuracy) and the accuracy of tool selection at each step (Tool Call Accuracy).
• Nature of errors: Conventional evaluation deals with errors in a single output. In agent evaluation, errors at intermediate steps propagate to subsequent processing (error propagation), requiring an evaluation design that can track which step broke down and why.
• Role of evaluators: Conventional evaluation often relies solely on human review or comparison against reference text. For agents, automated evaluation combining LLM-as-a-Judge (NVIDIA recommends 70B parameters or more) has become a practical option.

In this way, the "granularity" and "time axis" of evaluation are fundamentally different.

Conventional single-task AI systems for translation, classification, summarization, and similar tasks are completed in a single "input → output" step. This is precisely why static metrics such as accuracy and F1 scores were sufficient for evaluation. In autonomous workflows, however, agents call multiple tools, evaluate intermediate results, and independently determine the next step. This "chained decision-making" is the fundamental reason why existing metrics fall short.

Consider error propagation, for example. If a misjudgment at step 2 quietly corrupts the output at step 5, the root cause cannot be identified by examining only the final output. Furthermore, even if 9 out of 10 steps are correct, a failure at the final step causes a simple binary evaluation to record only "failure." An evaluation that does not reflect any of the intermediate progress provides almost no guidance for improvement. Additionally, whether the correct tool was called with the correct arguments must be verified independently of the correctness of the final answer. Moreover, because the execution path can vary even with the same prompt—a property known as non-determinism—a single evaluation run is insufficient; statistically robust evaluation across multiple runs is required.

The direction of evaluation design varies depending on the nature of the task. For fully autonomous agents that make decisions from start to finish, it is appropriate to center the evaluation on Goal Accuracy and Task Completion Rate. For Human-in-the-Loop collaborative designs, on the other hand, a design that prioritizes handoff timing and handoff quality over final accuracy is more effective.

Many practitioners find themselves uncertain about which agent to start evaluating first. Because the metrics that should be evaluated differ significantly depending on the type of agent, it is important to first clarify the target.

Representative agent types are as follows:

• Single-turn completion agents: A simple configuration that returns one response to a single input. RAG chatbots and FAQ response bots are typical examples, with Answer Accuracy and Topic Adherence serving as the primary evaluation axes.
• Multi-step, tool-using agents: A type that completes tasks by calling external APIs and databases multiple times. Tool Call Accuracy and Trajectory Evaluation become important here. The Policy Adherence Rate enumerated by IBM is also particularly meaningful for this type.
• Multi-agent collaborative agents: A configuration in which an orchestrator and multiple sub-agents work together. Because the quality of handoffs between agents determines overall quality, metrics such as Agent Handoff F1-Score (HF1) are necessary.
• Human-in-the-Loop agents: A design that delegates some decisions to humans, commonly adopted in AI BPO settings. Escalation appropriateness rate and response latency—which cannot be measured by automated evaluation alone—must also be included as evaluation targets.

You may have experienced starting to design an evaluation, only to find yourself midway through wondering, "What was I actually trying to measure in the first place?" The cause is almost always insufficient preparation before getting started.

Specifically, three foundations need to be established in advance: defining the scope of what to measure, test data to serve as a comparison baseline, and an infrastructure for collecting execution logs. These are what determine the accuracy of your evaluation. Each of these preparation steps will be explained in turn in the sections that follow.

It's tempting to think, "Let's just evaluate all the features," but in practice, narrowing your scope and starting with the highest-priority workflows leads to faster improvement cycles.

Before beginning an evaluation, the first question to ask is: "What is the purpose of evaluating this agent?" Whether the goal is quality assurance, release decision-making, or ongoing performance monitoring, the metrics to measure and the evaluation frequency will differ accordingly.

Key perspectives for scope definition

• Range of target workflows: Clarify whether you are dealing with a single tool call or a multi-step process involving multiple agents working in coordination
• Impact on users and business operations: Prioritize evaluation of workflows where failure carries high risk (e.g., billing processing, customer support)
• Evaluation granularity: Distinguish between measuring end-to-end task completion and measuring the accuracy of individual tool calls

What to decide when setting goals

• Thresholds for pass/fail determination (e.g., referencing industry standards such as ServiceNow's definition, where Overall Task Completeness of 90–100% is rated Excellent and 70–89% is rated Good, to establish your own internal benchmarks)
• Evaluation cycle frequency (per deployment, weekly, monthly)
• Ownership of evaluation results and the decision-making flow

If you begin an evaluation without a clear scope and goals, even a large volume of collected logs will make it difficult to identify what needs to be fixed. By first selecting one or two representative workflows and running through the entire evaluation process with them, you can ensure a smoother rollout to subsequent workflows.

The accuracy of an evaluation is largely determined by the quality of the test data and the baseline configuration. No matter how well-designed your evaluation metrics are, if the data is biased, the results will not reflect reality.

When designing test data, it is recommended to prepare the following three types:

• Happy path: Scenarios with normal inputs where the agent behaves as expected
• Edge cases: Exceptional inputs such as ambiguous instructions, incomplete information, or tool call failures
• Regression cases: Scenarios that reproduce past failures or malfunctions

Including a balanced mix of all three types helps prevent overestimation of success rates.

The approach to baseline configuration varies depending on the development phase of the agent. In the early stages of development, it is practical to use processing logs from human operators or the track record of existing rule-based systems as the baseline; once the agent is in production, the scores from the immediately preceding version of the agent serve as the baseline. Since changing the point of comparison affects how improvements are interpreted, it is important to fix the baseline once it has been set and maintain it across evaluation cycles.

Regarding data volume, it is generally advisable to secure at least several dozen samples per task type. With a small number of samples, binary scores such as Agent Goal Accuracy are subject to large statistical variance, making it difficult to assess the impact of improvements.

It is also important to ensure that the test data has a distribution close to that of production data.

"Even after looking at the logs, I had no idea which step had failed" — this is a common experience in agent evaluation. Before designing evaluation metrics, it is essential to first establish a state of observability.

Because agents chain together multiple tool calls and LLM inferences, tracking down the root cause of quality issues is difficult with simple request/response logs alone. What is needed is a mechanism that links the inputs and outputs of each step, tool call results, and elapsed time together as a single "trace."

The key elements to put in place are as follows:

• Distributed tracing: Assign a unique trace ID to each agent step and record the parent-child relationships between steps
• Tool call logs: Save the tool name, input parameters, return values, and success/failure flags as structured logs
• LLM inference logs: Record the full prompt, model output, token count, and latency in a way that allows for later reproduction
• Error classification tags: Classify error types — such as timeouts, tool failures, and malformed outputs — using tags to facilitate aggregation

Log granularity involves a cost tradeoff. In production environments, a practical approach is to adjust the sampling rate while ensuring that all failure traces are saved in their entirety.

Conclusion: Measuring agent success rates requires combining multiple metrics, including task completion rate, step accuracy, and cost efficiency.

The quality of autonomous workflows cannot be understood from a single number alone. The following H3 sections will explain the definition and measurement method of each metric in turn.

Teams new to agent evaluation tend to assess "whether a task was completed" as a binary outcome (success/failure). In practice, however, lumping cases where intermediate steps were executed correctly into the "failure" category causes you to lose sight of improvement opportunities. Measuring the partial completion rate alongside the overall rate makes it easier to identify bottlenecks.

Task Success Rate (TSR) refers to the proportion of scenarios in which an agent achieved its final goal. In ServiceNow's evaluation framework, the Overall Task Completeness score is recorded on a three-tier scale — "3: Successful / 2: Partially Successful / 1: Unsuccessful" — and managed with labels ranging from Excellent (90–100%) to Poor (0–49%).

Partial completion rate is the proportion of cases that fall into the "Partially Successful" category. The key measurement points are as follows:

• Calculate the proportion of scenarios judged as "partially complete" out of the total number of test scenarios
• When partial completions are frequent, use trace logs to identify at which step processing stalled
• When the partial completion rate remains persistently high, the cause is often tool call failures or missing context

Combining these two metrics reveals situations such as "low completion rate but high partial completion rate."

When an agent chains multiple steps together, the overall success or failure of a task is directly tied to whether each individual step was executed correctly. This is where two metrics come into focus: Step Accuracy and error propagation rate.

Step accuracy refers to the proportion of individual actions executed by the agent — such as tool calls, decision branching, and data extraction — that completed as expected. NVIDIA NeMo's Trajectory Evaluation and Tool Call Accuracy are built on this concept, recording the correctness of each step as a score from 0 to 1.

Error propagation rate indicates the degree to which errors in upstream steps cascade downstream. For example, in an invoice processing workflow, if the "amount extraction step" produces an error, it has been reported that the subsequent "approval routing" and "accounting journal entry" steps also fail in a chain reaction. Because the cost of correction grows exponentially as errors propagate, monitoring step accuracy in conjunction with this metric is critical.

The decision criteria can be summarized as follows:

• When the number of steps is small (3–5 steps or fewer), task completion rate alone is often sufficient to assess quality
• When the number of steps is large or branching is complex, step accuracy and error propagation rate should be measured separately to pinpoint exactly which steps are prone to errors

In the improvement cycle, steps with high error propagation rates are designated as "critical nodes," prioritized for log collection, and the findings are fed back into prompt revisions and tool design reviews.

"This agent has high accuracy, but costs are ballooning and we can't deploy it to production" — this is a situation commonly heard in the field. When teams focus solely on accuracy metrics, latency and cost issues tend to surface later, which is why these must be measured in parallel with success rates.

Key Latency Measurement Points

• End-to-end latency: Total elapsed time from receiving a user request to returning the final response
• Per-step latency: Break down and record LLM inference, tool calls, and external API wait times separately
• P95 / P99 percentiles: Capturing outliers in addition to averages allows you to understand the frequency of latency spikes

Breaking latency down by step makes it possible to isolate bottlenecks — for example, determining whether slowness stems from LLM inference or from waiting on external tool responses.

Cost Efficiency Measurement Points

• Token cost per task: Aggregate input and output token counts consumed to complete one task from logs
• Number of tool calls: Check for unnecessary loops or duplicate calls
• Cost per Successful Task: One of the representative metrics listed by IBM; dividing Cost by the number of successfully completed tasks quantifies cost-effectiveness

Practical Measurement Tips

Latency and cost can be centrally managed by recording span-level timestamps and token usage in a tracing infrastructure.

Conclusion: Quality evaluation can be systematized in three steps — scenario testing → human evaluation / LLM-as-a-Judge → continuous monitoring.

Once evaluation metrics have been defined, the next step is to design the actual evaluation process. The following steps explain how to ensure quality incrementally, from functional verification through to production monitoring.

It is tempting to think that "testing with as many random inputs as possible improves coverage," but in practice, designing scenarios by working backward from goals first leads to earlier problem detection.

In scenario testing, the business workflows the agent is actually responsible for are prepared across three categories: normal cases, abnormal cases, and boundary cases.

• Normal cases: Does the task complete end-to-end with expected inputs?
• Abnormal cases: Can the agent handle error conditions appropriately, such as missing required parameters or tool call failures?
• Boundary cases: Does the agent avoid selecting the wrong tool in edge cases, such as ambiguous inputs or competing goals?

It is important to define both the "expected tool call sequence" and the "final output" for each scenario. Among the key metrics listed by IBM, Success rate / task completion and Error rate can be measured directly at this stage.

After running the tests, scores are aggregated from the following perspectives:

1. Pass/Fail result per scenario
2. Tool call accuracy (whether the correct tools were called in the correct order)
3. Whether recovery behavior occurred when an error arose

NVIDIA NeMo's Tool Call Accuracy and Trajectory Evaluation correspond to this aggregation and enable quantitative comparison on a score range of 0.0–1.0.

Once scenario testing has confirmed whether features pass or fail, the next step is evaluating quality — specifically, "how natural and accurate were the responses and actions?" A dilemma teams commonly face at this stage is: "We want to automate, but we're worried about accuracy." The widely adopted approach for striking this balance is combining human evaluation with LLM-as-a-Judge.

The basic principle for deciding which to use is straightforward: it comes down to whether the correctness of an output can be clearly defined. Cases with a single definitive answer — such as tool call accuracy or binary task completion judgments — are handled through automated scoring with LLM-as-a-Judge, while cases with ambiguous criteria — such as response naturalness, ethical appropriateness, or fit with business context — are prioritized for human evaluation.

When using LLM-as-a-Judge, it is important to note that the model size of the Judge LLM directly affects evaluation accuracy. With NVIDIA NeMo's Agent Goal Accuracy, using a model of 70B parameters or more — and ideally over 405B — tends to improve accuracy. Metrics that return scores in the 0.0–1.0 range, such as Tool Call Accuracy and Topic Adherence, are well-suited for automated scoring, and explicitly including a scoring rubric in the evaluation prompt helps ensure score reproducibility.

On the other hand, having humans evaluate every single output is often not cost-feasible in practice. As a result, a common operational approach is to prioritize human review for samples that receive low scores from LLM-as-a-Judge or that fall near boundary values. Sharing evaluation criteria and concrete examples with reviewers in advance helps reduce variability between evaluators.

This two-tiered approach makes it possible to achieve both scalability and evaluation accuracy.

"It passed scenario testing, but no one noticed anything was wrong after it went live" — preventing exactly this kind of situation is the role of continuous monitoring.

One-time evaluations cannot capture degradation that occurs over time, such as model behavior drift or changes to external API specifications. Building real-time observation of the production environment into your processes is essential for maintaining agent quality.

Key metrics to track in monitoring

• Task completion rate trends: Using ServiceNow's thresholds as a reference, verify on a weekly basis whether the Excellent level (90% or above) is being maintained
• Tool call success rate: If Tool Call Accuracy begins to decline, suspect specification changes in external APIs or internal tools
• Latency and cost: Trigger alerts when average response time or token consumption exceeds a set percentage above the baseline
• Error rate: Monitor in real time for sudden spikes in unexpected exceptions or task interruptions

Approach to alert configuration

Designing alerts across two layers — "static thresholds" and "rate-of-change thresholds" — is effective.

• Static threshold: Send an immediate notification if the task completion rate falls below 70%
• Rate-of-change threshold: Issue a warning if the error rate worsens by 20% or more compared to the previous week

Leveraging managed services such as Amazon Bedrock AgentCore Evaluations allows you to build a pipeline from log collection to alert triggering in a short timeframe.

It is not uncommon to hear that evaluation metrics were introduced, yet nothing actually improved on the ground. When you dig into the cause, the problem almost always lies in how the metrics were selected or how the evaluation environment was configured. Thinking of a "we're measuring but nothing is changing" situation as a signal that what is being measured is off-target makes it easier to understand. Familiarizing yourself with the failure patterns commonly seen in AI BPO operations is a shortcut to improving the accuracy of your evaluation cycle.

It is easy to conclude that "if the task completion rate is high, quality is fine" — but in practice, risks that a single metric cannot reveal tend to accumulate.

One of the most common failures in agent evaluation is the pattern of chasing only a specific metric. For example, when Task Completion Rate is set as the sole KPI, the following blind spots emerge:

• Step quality becomes invisible: Even when a task is ultimately judged "complete," there are cases where unnecessary tool calls occurred along the way, or traces of self-corrected errors remain
• Cost and latency are ignored: Even if the completion rate is high, if the number of LLM calls or the cost per task exceeds acceptable limits, the system will not hold up in production
• Policy violations go undetected: As IBM points out with Policy Adherence Rate, compliance with internal rules and regulations is not reflected in the completion rate

ServiceNow's evaluation framework recommends recording Overall Task Completeness and Tool Performance Record on separate axes. This is intended to evaluate "whether the task was completed" and "whether the correct tools were used" independently of each other.

For improvement purposes, it is advisable to track at minimum the following three axes together:

1. Outcome metrics: Task completion rate, partial completion rate
2. Process metrics: Tool call accuracy, number of steps

An agent that achieves high scores in a test environment can sometimes experience a sharp drop in performance after being deployed to production. This "gap problem" is one of the pitfalls that tends to be overlooked during evaluation design.

The causes generally fall into three layers. The first is input distribution mismatch: test data tends to consist mainly of well-formed scenarios, so agents are often unable to handle the ambiguous instructions, typos, and colloquial expressions that appear in production. The second is differences in tool and API state: while the test environment uses mocks that always return normal responses, production environments are subject to sudden external API timeouts and specification changes. The third is context length differences: as conversation history grows in real-world use, context overflow errors that did not appear during testing begin to surface.

The appropriate response depends on the source of the gap. If the cause is input distribution mismatch, the priority is to increase the proportion of real samples mixed into the test data. If the issue stems from tools or external dependencies, it is more effective to incorporate chaos engineering-style fault injection testing into the evaluation workflow.

A practically effective technique is the "live data feedback loop," in which a portion of production logs is sampled and periodically fed back into the test set. Because evaluation scenarios naturally follow real-world input patterns, the divergence between scores and actual performance tends to narrow. In addition, preparing a staging environment that closely mirrors production and including real-connection tests with external APIs in the evaluation process can help narrow the gap further.

If you continue to use only test environment scores as your improvement metric, you risk misjudging the true state of production quality. Scores are ultimately "results on a test," and without a design that is mindful of the distance from production, those numbers offer no real assurance.

When numbers come in, it's not uncommon to simply glance at them, think "the score is low," and move on. But the true value of evaluation results lies in what you change based on them.

The first thing to focus on is identifying the bottleneck — specifically, which step has the highest concentration of failures. When task success rates are low across the board versus when failures are clustered in a specific tool call, the remedies are entirely different. The former requires revisiting the prompt's instruction design itself, while the latter is likely a problem with the tool's interface definition or input/output schema.

Once the bottleneck has been narrowed down, the principle is to limit changes to one place at a time. If you apply multiple fixes simultaneously, it becomes impossible to determine which change moved the score. If you modify the prompt, keep the tool design as-is and re-evaluate. This painstaking one-variable-at-a-time verification forms the foundation for iterating through the cycle.

After implementing an improvement, always re-measure using the same evaluation set. Even when scores improve, it's important to check whether other metrics have unintentionally degraded. For example, trade-offs like reducing the number of tool calls to lower latency only to see task completion rates drop are quite common in practice. Rather than tracking evaluation metrics as a single number, monitoring multiple metrics together allows you to catch such side effects early.

Evaluation results serve as a map showing where the pipeline is getting stuck. When you find a step with a low score, the basic approach to improvement is to dig into that step first.

Steps for Identifying Bottlenecks

• Review trace logs: Line up per-step success rates, latency, and error types on a step-by-step basis, and look for where scores drop sharply.
• Classify errors: Tally the number of occurrences by error type — such as tool call failures, hallucinations, and inability to exit loops.
• Run reproduction tests: Reproduce failing scenarios at the smallest possible unit to isolate whether the root cause lies in the prompt or the tool design.

Once you've determined that the bottleneck is in the prompt, the following approaches are effective.

Key Approaches to Prompt Improvement

• Adjust instruction granularity: Replacing vague instructions with concrete step-by-step descriptions tends to reduce tool selection errors.
• Revisit context length: Trim unnecessary information and narrow down what the agent should reference. Introduction to Context Engineering | The Next Step in Prompt Design is also a useful reference.
• Add few-shot examples: Incorporate correct examples corresponding to failure patterns into the prompt to improve judgment accuracy.

After making improvements, always re-evaluate using the same test scenarios and confirm score changes with concrete numbers.

If issues remain that prompt improvements couldn't resolve, the next steps are fine-tuning and revisiting tool design. However, jumping straight to "fine-tuning will improve accuracy" is premature — in practice, the root cause is often a flaw in tool design. It's more effective to suspect that first.

Fine-tuning is useful in a limited set of situations: when the model repeatedly fails on domain-specific vocabulary or writing style, when output formatting is unstable with prompts alone, or when Tool Call Accuracy scores are consistently low.

On the other hand, if tool call argument definitions are ambiguous or return value schemas are inconsistent, tool specifications should be addressed before fine-tuning. Before running a costly training job, go through the tool design review points one by one.

Q1. How do AI agent evaluation metrics differ from traditional chatbot evaluation?

Traditional chatbot evaluation focused primarily on measuring the accuracy of a single response (answer accuracy, BLEU score, etc.). AI agent evaluation, by contrast, assesses the completion rate of tasks spanning multiple steps, the accuracy of tool calls (Tool Call Accuracy), and whether goals were achieved (Agent Goal Accuracy) — evaluating a chain of autonomous actions in sequence. The fundamental difference is that it examines the quality of the entire process, not just a single turn.

Q2. What is the difference between Agentic Success Rate and Task Success Rate?

Task Success Rate (TSR) is a binary metric that determines whether a task was ultimately completed or not. Agentic Success Rate (ASR), on the other hand, also considers the state transitions the agent went through along the way, decomposing the evaluation into Transition Recall and Transition Precision. TSR alone tends to miss cases where the agent happened to reach the correct answer but took inefficient steps along the way; combining it with ASR allows you to capture process quality as well.

Q3. How large does the Judge LLM used for evaluation need to be?

According to the official NVIDIA NeMo documentation, a Judge LLM requires a model of at least 70B parameters, with models exceeding 405B parameters recommended. Using a smaller model as the Judge tends to reduce judgment accuracy for subtle nuances, undermining the reliability of the evaluation results themselves. When cost constraints apply, a practical approach is to use a large-scale Judge LLM only for high-priority scenarios and combine it with rule-based evaluation for the rest.

Q4. How should the Overall Task Completeness score from ServiceNow be interpreted?

Under ServiceNow's definition, the Overall Task Completeness score is recorded on a three-tier scale: 3 (Successful), 2 (Partially successful), and 1 (Unsuccessful). Achievement rate labels are also defined as Excellent (90–100%), Good (70–89%), Good (70–89%), Moderate (50–69%), and Poor (0–49%). In production operations, it is common practice to set Good or above as the minimum threshold and prioritize scenarios rated Moderate or below for improvement.

Q5. If evaluation metrics are in place but improvements aren't materializing, what should be reviewed?

When measuring metrics isn't leading to improvement, the cause in most cases is that "the metrics are too coarse-grained to identify the bottleneck." Start by adding fine-grained metrics beyond aggregate values like task completion rate — such as per-step error propagation rates and tool call accuracy. Next, examine the trace logs of failing scenarios to isolate whether the problem lies in prompt design, tool definitions, or the response quality of external APIs. Introduction to Context Engineering | The Next Step in Prompt Design is also a useful reference for driving improvements.