Skip to content
⚠️ Before You Start
Risk: Low
Duration: 1-2 days for the first suite, then under 1 hour per run
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Low (offline evaluation; the main risk is false confidence from a weak dataset)

Score agent trajectories with the Azure AI Evaluation SDK and custom scorers

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-25). Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: AGT-07-T01, AGT-07-T02, AGT-07-T03, AGT-07-T04.

1. Summary

When a new agent needs its first evaluation suite, or an existing agent is only scored on final answers. Use it to produce the Level 2 metrics: task success, step efficiency, tool accuracy and cost per task.

Risk: Low (offline evaluation; the main risk is false confidence from a weak dataset). Typical duration: 1-2 days for the first suite, then under 1 hour per run.

2. When to use and when not to

Use it when: When a new agent needs its first evaluation suite, or an existing agent is only scored on final answers. Use it to produce the Level 2 metrics: task success, step efficiency, tool accuracy and cost per task.

Do not use it when: to be written by the author.

3. What you must know first

  • The agent evaluators (IntentResolution, TaskAdherence, ToolCallAccuracy) are LLM judges and some are in preview; check status before client reporting
  • ToolCallAccuracy needs tool_definitions plus either response or tool_calls; agent-message queries must start with the system message
  • Each evaluator returns a score plus a {metric}_result pass/fail against a threshold you can override
  • Trajectory (path) metrics need the ordered trace, not only the final answer
  • evaluate() accepts plain Python callables as custom evaluators, mapped to data columns by name
  • Judge scores vary between runs; compare averages over a fixed dataset, not single cases

4. Prerequisites and access

  • Azure OpenAI judge deployment and a Foundry project you can log results to
  • Python 3.10+ with azure-ai-evaluation installed in a pinned environment
  • At least 30 representative tasks with expected tool calls agreed with the product owner
  • Agent runs captured as messages plus tool_calls and tool_definitions (JSONL)
  • Token usage per run available from traces or the model response

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Agree the Level 2 metric set and pass thresholds with the agent lead and record them in eval/thresholds.json.
  2. Export 30 or more agent runs to JSONL with query, response, tool_calls, tool_definitions, expected_steps and token counts.
  3. Configure the judge model from environment variables; never hard-code keys.
  4. Add the built-in agent evaluators and the custom step-efficiency and cost-per-task scorers.
  5. Run evaluate() locally against 5 rows first and inspect every reason field by hand.
  6. Run the full dataset and log results to the Foundry project for the team to review.
  7. Label 10 judge results by hand and check agreement before trusting the scores.
  8. Commit the dataset, scorer code and baseline metrics under version control with the agent version tag.
  9. Verify: re-run on the same data and confirm averages move by less than the agreed noise band.
  10. Rollback: if the suite is unreliable, keep the previous baseline and mark the new suite as advisory only.

Draft snippet: Run agent evaluators plus a custom step-efficiency scorer

import os

from azure.ai.evaluation import AzureOpenAIModelConfiguration, evaluate
from azure.ai.evaluation import IntentResolutionEvaluator, TaskAdherenceEvaluator, ToolCallAccuracyEvaluator

model_config = AzureOpenAIModelConfiguration(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version=os.environ["AZURE_OPENAI_API_VERSION"],
    azure_deployment=os.environ["JUDGE_DEPLOYMENT"],
)


def step_efficiency(tool_calls, expected_steps):
    used = len(tool_calls or [])
    return {"step_efficiency": min(1.0, expected_steps / used) if used else 0.0}


result = evaluate(
    data="eval/agent_runs.jsonl",
    evaluators={
        "intent": IntentResolutionEvaluator(model_config=model_config),
        "task_adherence": TaskAdherenceEvaluator(model_config=model_config),
        "tool_accuracy": ToolCallAccuracyEvaluator(model_config=model_config),
        "step_efficiency": step_efficiency,
    },
    azure_ai_project=os.environ.get("AZURE_AI_PROJECT"),
    output_path="eval/results.json",
)
print(result["metrics"])

Snippets to add:

  • snippets/eval/run_agent_eval.py: Trajectory evaluation runner with built-in and custom scorers (python)
  • snippets/eval/sample_row.json: Sample agent run row with tool_calls and tool_definitions (json)
  • snippets/eval/thresholds.json: Metric thresholds file (json)

6. Verify it works

A second engineer builds the suite for the sandbox demo agent from the playbook alone, runs it twice and gets all four Level 2 metrics. Pass: both runs complete, averages differ by less than the noise band, and a seeded wrong-tool run is flagged as a failure.

7. Known issues and fixes

Symptom Likely cause Fix
ToolCallAccuracy returns errors or empty results tool_definitions missing or tool_calls not in the expected shape Pass tool_definitions as JSON-schema style objects and each call with type, tool_call_id, name and arguments
Scores swing widely between identical runs Judge model temperature or a reasoning model used without its flag Pin the judge deployment, set is_reasoning_model=True for reasoning judges and average over the full dataset
All runs pass but users still report wrong actions Dataset only covers happy paths Add negative cases: tool outages, ambiguous requests and requests the agent must refuse
Custom scorer is ignored by evaluate() Function parameter names do not match data column names Rename parameters to match columns or add a column mapping in evaluator_config

8. Monitoring and alerts

Signal Alert when Action
Nightly evaluation job status Job fails or does not finish within its window Check judge deployment quota and dataset path; rerun before the next release gate
Judge-versus-human agreement on the labelled sample Agreement drops below the level agreed at setup Freeze releases that depend on that metric and recalibrate the judge

9. AWS and open-source equivalents

On AWS, Bedrock AgentCore Evaluations scores traced agents with built-in and custom LLM-as-judge evaluators. Open-source options are DeepEval (TaskCompletionMetric, StepEfficiencyMetric, ToolCorrectnessMetric) and LangSmith agentevals trajectory matchers.

10. Validation record

Pending. Planned dry-run scenario: A second engineer builds the suite for the sandbox demo agent from the playbook alone, runs it twice and gets all four Level 2 metrics. Pass: both runs complete, averages differ by less than the noise band, and a seeded wrong-tool run is flagged as a failure.

11. Sources