Skip to content
⚠️ Before You Start
Risk: Low
Duration: 1 day to set up, 10-60 minutes per run depending on dataset size
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Low (read-only evaluation, but judge model token spend must be budgeted)

Run RAG quality evaluation with the Azure AI Evaluation SDK

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off. Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: RAG-02-T05, RAG-02-T06, RAG-02-T07, RAG-02-T08.

1. Summary

Score your RAG answers with an LLM judge model before releasing changes. For every golden question, the Azure AI Evaluation SDK computes end-to-end scores on a 1 to 5 scale: groundedness (is the answer based on the retrieved context?), relevance (does the answer address the question?), and retrieval (is the retrieved text useful for the question?). Run this evaluation before every production release and as part of regression gates in CI.

Facts checked against Microsoft Learn on 29 October 2026. The Azure AI Evaluation SDK (azure-ai-evaluation version 1.18.10) is in public preview; check the current API version and capabilities in the official docs. The evaluation script has not been run against the live Azure AI Evaluation SDK. The dry-runner verifies its behaviour in the sandbox.

2. When to use and when not to

Use this how-to when:

  • You need end-to-end scores for groundedness, relevance, and retrieval quality, not only ranking metrics.
  • You plan to release changes to prompts, chat model deployments, or chunking, and need benchmarks against the previous release.
  • You want an automated evaluation step in CI pipelines or deployment validation stages.
  • You want to track RAG quality trends across releases and export run artefacts to Azure AI Foundry (preview).

Do not use it when:

  • You only need to verify whether the search index retrieves the expected document IDs. Use PB-RAG-02-01 (recall@k, MRR, and nDCG require no judge model tokens).
  • Users report wrong answers and you do not yet know the root cause. Use PB-RAG-02-03 to triage retrieval faults from generation faults.
  • You need a lightweight automated pull request merge gate with pass/fail thresholds in CI. Use PB-RAG-02-04, which builds on the scores from this playbook.
  • You need to investigate a live hallucination incident or citation mismatch in production. Use PB-RAG-06-04 instead.

3. What you must know first

  1. One call scores an entire dataset. evaluate() reads a JSONL file, invokes evaluators across rows, and returns aggregate metrics (metrics) alongside row-level scores (rows). Source: Local evaluation with the Azure AI Evaluation SDK.
  2. Column mapping binds input schema. The SDK uses ${data.<column_name>} syntax to route dataset columns into evaluator inputs. Evaluators fail if columns are not mapped explicitly. Source: Local evaluation with the Azure AI Evaluation SDK.
  3. Three core evaluators require no ground truth. GroundednessEvaluator needs query, context, and response. RelevanceEvaluator needs query and response. RetrievalEvaluator needs query and context. None of the three requires a ground-truth label. Source: Retrieval-augmented generation (RAG) evaluators.
  4. Scores are integers from 1 to 5. Higher scores indicate higher quality. The SDK default pass threshold is 3. Evaluators also provide boolean pass/fail flags and explanatory reasoning. Source: Retrieval-augmented generation (RAG) evaluators.
  5. A dedicated judge model runs the evaluation. Use a dedicated deployment in Azure OpenAI (such as gpt-4o or gpt-4o-mini). For reasoning models (o-series), pass --is-reasoning-model when running snippets/eval_release.py (which sets is_reasoning_model=True on evaluators). Source: Retrieval-augmented generation (RAG) evaluators. Do not share the deployment with production traffic (team guidance).
  6. Reasoning text consumes tokens. The SDK prompts the judge model to supply written rationale for each assigned score. This improves explainability but increases token spend. Check current model pricing before running large batches. Source: Local evaluation with the Azure AI Evaluation SDK.
  7. Judge scores have natural variance. Compare aggregate averages and pass rates over sets of at least 50 queries rather than single row scores. This is team guidance, not a platform limit.

4. Prerequisites and access

Item Detail
Roles Cognitive Services OpenAI User on the judge Azure OpenAI resource (Microsoft Entra ID). Use Entra ID authentication; never use API keys. Azure AI Developer if logging runs to an Azure AI Foundry project.
Judge deployment A dedicated gpt-4o or gpt-4o-mini deployment in Azure OpenAI. Isolate it from your user-facing application deployment (team guidance).
Python packages Python (check supported versions in the SDK docs), azure-ai-evaluation==1.18.10, azure-identity==1.27.0.
Golden set Labelled questions from PB-RAG-02-01 (baselines/golden_set.json).
Application contract Application search and answer functions to generate eval/cases.jsonl (see snippets/rag-eval-gate/eval/export_run.py).
Foundry project (optional) Storage Blob Data Owner and an Azure AI Foundry project for tracking (Step 7).

Export the environment variables:

export AZURE_OPENAI_ENDPOINT="https://${AZURE_OPENAI_RESOURCE}.openai.azure.com"
export JUDGE_DEPLOYMENT="${JUDGE_MODEL_DEPLOYMENT}"
# Check the current API version in official Azure OpenAI documentation
export AZURE_OPENAI_API_VERSION="2024-10-21"
# Optional: set project scope if tracking evaluation runs in Azure AI Foundry
export AZURE_AI_PROJECT="${AZURE_SUBSCRIPTION_ID}/${AZURE_RESOURCE_GROUP}/${AZURE_AI_PROJECT_NAME}"

5. Steps

Placeholders: ${AZURE_OPENAI_RESOURCE} (your Azure OpenAI resource name), ${JUDGE_MODEL_DEPLOYMENT} (deployment name of the judge model, e.g. gpt-4o), ${AZURE_SUBSCRIPTION_ID}, ${AZURE_RESOURCE_GROUP}, ${AZURE_AI_PROJECT_NAME}.

Full release evaluation script: snippets/eval_release.py. Dataset row example: snippets/cases.sample.jsonl.

Step 1. Prepare the evaluation dataset.

Generate eval/cases.jsonl from your golden query set by executing your application's search and answer logic with snippets/rag-eval-gate/eval/export_run.py.

Questions marked "unanswerable": true in the golden set are left out of this quality evaluation; PB-RAG-02-04 evaluates unanswerable questions separately with a refusal assertion (llm-rubric). Pass --skip-unanswerable so export_run.py excludes those rows from the cases output:

python snippets/rag-eval-gate/eval/export_run.py \
  --search retrieval.search:search \
  --answer app.answer:answer \
  --golden baselines/golden_set.json \
  --cases-out eval/cases.jsonl \
  --skip-unanswerable

Inspect the generated cases file:

head -n 1 eval/cases.jsonl | jq .

Verify: The line count equals the number of golden questions that are not unanswerable, and no line has an empty context or response. Each line matches snippets/cases.sample.jsonl.

Step 2. Initialise the judge model configuration.

snippets/eval_release.py initialises the judge model configuration for you using AzureOpenAIModelConfiguration and DefaultAzureCredential from environment variables:

# pseudo
import os
from azure.ai.evaluation import AzureOpenAIModelConfiguration
from azure.identity import DefaultAzureCredential

model_config = AzureOpenAIModelConfiguration(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    azure_deployment=os.environ["JUDGE_DEPLOYMENT"],
    api_version=os.environ.get("AZURE_OPENAI_API_VERSION"),
    credential=DefaultAzureCredential(),
)

Verify: The AzureOpenAIModelConfiguration object initialises locally from the environment variables. (Note: Instantiating the configuration object does not contact Azure; authentication, Entra ID role assignment, and deployment errors surface when evaluate() is called in Step 4).

Step 3. Instantiate the Groundedness, Relevance, and Retrieval evaluators.

snippets/eval_release.py instantiates the three evaluators using the judge model configuration. When evaluating with an o-series reasoning model, pass the --is-reasoning-model flag to set is_reasoning_model=True:

# pseudo
from azure.ai.evaluation import GroundednessEvaluator, RelevanceEvaluator, RetrievalEvaluator

evaluators = {
    "groundedness": GroundednessEvaluator(model_config, is_reasoning_model=is_reasoning_model),
    "relevance": RelevanceEvaluator(model_config, is_reasoning_model=is_reasoning_model),
    "retrieval": RetrievalEvaluator(model_config, is_reasoning_model=is_reasoning_model),
}

Verify: The dictionary contains instantiated GroundednessEvaluator, RelevanceEvaluator, and RetrievalEvaluator objects configured locally with prompt templates and settings. (Note: Evaluator instantiation does not call Azure OpenAI; network calls occur during evaluate() in Step 4).

Step 4. Run evaluate() on a sample, then the full dataset.

The SDK uses column_mapping to route dataset columns to evaluator inputs. snippets/eval_release.py configures this for you:

# pseudo
column_mapping = {
    "groundedness": {"query": "${data.query}", "context": "${data.context}", "response": "${data.response}"},
    "relevance": {"query": "${data.query}", "response": "${data.response}"},
    "retrieval": {"query": "${data.query}", "context": "${data.context}"},
}

Judge model token consumption and cancellation

Batch evaluation calls the judge model once per evaluator per row. Check current token pricing before running large datasets. If rate limits (TPM) are exceeded, reduce batch size or increase capacity on the judge deployment. To stop a running evaluation, cancel the pipeline job or terminate the process (Ctrl+C); this is a read-only evaluation, so no data in the application or search index changes.

Run snippets/eval_release.py on a sample of 5–10 rows first to verify configuration and column mappings before spending tokens:

python snippets/eval_release.py \
  --input eval/cases.jsonl \
  --limit 5 \
  --threshold 3.0 \
  --output reports/sample_results.json

Verify: The sample run processes 5 records and completes without authentication or schema errors.

Next, execute evaluation across the full dataset with the minimum pass rate threshold:

python snippets/eval_release.py \
  --input eval/cases.jsonl \
  --threshold 3.0 \
  --min-pass-rate 0.9 \
  --output reports/results.json

Verify: reports/results.json exists, containing metrics and rows arrays.

Step 5. Publish evaluation results as a pipeline artefact.

In your CI/CD pipeline, publish reports/results.json so results remain attached to the build even if the evaluation threshold fails the step (exit 1). Add condition: succeededOrFailed():

- task: PublishPipelineArtifact@1
  condition: succeededOrFailed()
  inputs:
    targetPath: 'reports/results.json'
    artifact: 'rag-quality-eval-$(Build.BuildId)'
    publishLocation: 'pipeline'

Verify: The artefact appears under pipeline run outputs and can be downloaded even when evaluation scores fall below threshold.

Step 6. Compare evaluation scores against the previous release baseline.

On the first run there is no baseline yet. Leave out --baseline and the script skips the comparison. After a release is approved, record its baseline by copying reports/results.json to baselines/previous_release.json and committing it to your repository, or by downloading it from that release pipeline run's published artefact (rag-quality-eval-*). Later runs use it. If the --baseline file is missing, the script only prints a warning and skips the comparison.

Run candidate evaluation with --baseline to compute metric deltas against the accepted baseline:

python snippets/eval_release.py \
  --input eval/cases.jsonl \
  --threshold 3.0 \
  --min-pass-rate 0.9 \
  --baseline baselines/previous_release.json \
  --output reports/results.json

Verify: Output prints averages and pass rates for each evaluator, displays baseline deltas (e.g. [baseline: 3.80, delta: -0.15]), lists any weak cases below 3.0, and exits with code 0 if passing or code 1 if thresholds are not met.

Step 7 (optional). Log evaluation runs to Azure AI Foundry (preview).

To track evaluation runs in the Azure AI Foundry portal, export the AZURE_AI_PROJECT environment variable in subscription/resource-group/project format:

export AZURE_AI_PROJECT="${AZURE_SUBSCRIPTION_ID}/${AZURE_RESOURCE_GROUP}/${AZURE_AI_PROJECT_NAME}"

snippets/eval_release.py automatically detects this variable, parses the project scope, and passes it to evaluate(azure_ai_project=...).

Verify: result["studio_url"] is returned and logged by the script, providing a URL to view the run in the Azure AI Foundry portal.

6. Verify it works

This section checks the evaluation setup itself, not your application. You give the script answers that you know are good and answers that you know are bad. You then check that it passes the good ones and catches the bad ones. The dry-runner runs it in the sandbox.

Build three small test files from your own eval/cases.jsonl:

  1. Good rows. Take 20 rows that you have reviewed. Each answer must be correct and fully supported by its context. Save them as eval/test_baseline_20.jsonl. If there is no application yet, write 20 simple rows by hand from the sandbox sample documents: a question, the passage that answers it as the context, and a short answer that uses only that passage.
  2. Ungrounded rows. Copy that file to eval/test_ungrounded_20.jsonl. In 3 rows, replace the response with a statement that is not in the context.
  3. Empty context row. Copy one row, set context to an empty string, and save it as eval/test_empty_context.jsonl.

Test 1. Good rows pass.

python snippets/eval_release.py \
  --input eval/test_baseline_20.jsonl \
  --threshold 3.0 \
  --output reports/test_baseline_results.json

Verify: The run finishes with exit code 0. The Groundedness, Relevance and Retrieval averages are all at or above 3.0.

Test 2. Ungrounded answers are caught. Pass --min-pass-rate 0.9. With 3 of 20 rows failing, the pass rate is 85%, which is below 90%, so the script exits with code 1. Without this option, an average score above 3.0 exits with code 0.

python snippets/eval_release.py \
  --input eval/test_ungrounded_20.jsonl \
  --threshold 3.0 \
  --min-pass-rate 0.9 \
  --output reports/test_ungrounded_results.json

Verify: The script outputs scores below 3.0 for the 3 changed rows, shows their reasons under the weak rows summary, and exits with code 1.

Test 3. Missing context scores low.

python snippets/eval_release.py \
  --input eval/test_empty_context.jsonl \
  --threshold 3.0 \
  --output reports/test_empty_context_results.json

Verify: Groundedness is below 3 for the empty-context row. An answer with nothing to support it must not pass.

Do not copy these test result files to baselines/previous_release.json. Only the results of an approved release run become the baseline.

7. Known issues and fixes

Symptom Likely cause Fix
Evaluation fails with 401 Unauthorized or PermissionDenied The executing identity lacks data-plane permissions on the judge Azure OpenAI resource Assign the Cognitive Services OpenAI User role on the judge resource to your Entra ID identity or service principal.
All rows score 1 on groundedness The context column name in JSONL does not match ${data.context} in column mapping Verify the column mapping keys in evaluator_config against field names in eval/cases.jsonl.
Score differs between identical runs LLM judges are non-deterministic across repeated evaluations Evaluate on sets of at least 50 queries (team default) and compare aggregate averages rather than individual queries.
Evaluator crashes when using reasoning models Reasoning models require explicit SDK parameter handling Pass --is-reasoning-model when running snippets/eval_release.py with an o-series judge deployment (which sets is_reasoning_model=True on evaluators).
Missing metric key in result["metrics"] Key naming changes across minor SDK versions Use snippets/eval_release.py, which checks exact metric keys, and pin azure-ai-evaluation==1.18.10. The key names have not been checked against the live SDK. The dry-runner confirms them in the sandbox.
429 Too Many Requests: Requests rate limit exceeded Concurrency exceeds TPM quota on the judge deployment Increase TPM quota on the judge Azure OpenAI deployment or run evaluations in off-peak windows.

8. Monitoring and alerts

The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Source Alert when Action
Average groundedness in the release evaluation Evaluation report Below 3.0 (team default) Block release. Check prompt instructions and citations in PB-RAG-02-03.
Average relevance in the release evaluation Evaluation report Below 3.0 (team default) Block release. Review question intent and system prompt phrasing.
Average retrieval in the release evaluation Evaluation report Below 3.0 (team default) Block release. Review search weights and index ranking in PB-RAG-02-01.
Metric pass rate (groundedness, relevance, or retrieval) Evaluation report Below 90% (team default; script reports pass rate per metric) Block release. Triage failing rows listed in reports/results.json.
Judge model token spend Azure Cost Management Exceeds 80% of monthly evaluation budget (team default) Sample evaluation queries or evaluate a subset before running full suites.
Evaluation pipeline failure CI Pipeline run Any failure (exit code 1 or 2) Inspect pipeline logs for authentication, schema, or regression failures.

9. Cost and capacity notes

  • Judge model token consumption: Every evaluated row sends prompt tokens (query, context, response, instructions) and receives completion tokens (score, reasoning text) per evaluator. Check current model pricing before running large batches.
  • Isolate evaluation spend: Deploy the judge model in a dedicated Azure OpenAI resource or deployment. This keeps evaluation expenses separate from application inference in cost reports.
  • Quota allocation: Allocate sufficient Tokens Per Minute (TPM) on the judge deployment to avoid HTTP 429 throttling during parallel evaluations.

10. AWS and open-source equivalents

Amazon Bedrock evaluation jobs provide managed RAG evaluations scored by a language model. Consult official AWS documentation for setup and scoring rubrics.

Open-source evaluation frameworks evaluate RAG outputs locally or in CI:

Framework Metric Notes
Ragas (open source) Faithfulness, Response Relevancy, Context Precision, Context Recall Evaluates retrieval and response quality against reference datasets.
DeepEval (open source) Faithfulness Evaluates whether the LLM answer is grounded in the retrieved context.

Do not compare scores directly across frameworks. Scoring rubrics and normalisation differ between frameworks.

11. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: Dry-runner deploys Azure AI Evaluation SDK with an Azure OpenAI judge model, runs snippets/eval_release.py on a 20-row test dataset containing 3 seeded ungrounded rows, and validates that Groundedness, Relevance, and Retrieval metrics are generated with ungrounded rows flagged.

Pass criteria: Script completes execution, outputs reports/results.json, identifies the 3 seeded ungrounded rows below score 3.0, and exits with code 1 when run with --min-pass-rate 0.9 (pass rate 85% falls below the 90% gate threshold).

Record

Item Detail
Playbook PB-RAG-02-02 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

12. Sources

Checked on 29 October 2026.