Skip to content
⚠️ Before You Start
Risk: Low
Duration: 1 day to set up
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Low (read-only evaluation; judge token cost must be monitored)

Score groundedness and gate releases

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off. Author: Gayathri Ramachandran · dry-run: Yasar · sign-off: Deeparani. Work items: RAG-06-T09, RAG-06-T10, RAG-06-T11, RAG-06-T12.

1. Summary

A change to a prompt, model or retrieval setting can weaken grounding without causing any visible error. Groundedness scoring measures how well each answer is supported by the context it was generated from. The steps below score the answers of a candidate release with the Azure AI Evaluation SDK GroundednessEvaluator and an Azure OpenAI judge model. The release is blocked in CI/CD when the pass rate is too low, and sampled production answers are re-scored every night.

The gate scores answers produced by the candidate prompt and model (step 2). Scoring the stored golden answers only proves the gate is wired up; it says nothing about the release.

Facts checked against Microsoft Learn and AWS documentation on 30 October 2026. Latest stable data-plane API: 2024-10-21.

2. When to use and when not to

Use this how-to when:

  • Gating pull requests and release pipelines before deploying changes to system prompts, answer templates, model deployments, or retrieval chunking strategies.
  • Running nightly scheduled evaluations over sampled production inference traces to detect grounding drift.
  • Auditing how well answers are grounded across the approved golden test set (FND-04).
  • Generating structured evaluation evidence and triage logs for ungrounded responses.

Do not use it when:

  • You are running quick local checks on a prompt before you commit. Use PB-RAG-06-05 with promptfoo instead.
  • You are generating claim-level citations in the application layer. Use PB-RAG-06-01 instead.
  • You are triaging an active live hallucination incident or customer escalation. Use PB-RAG-06-04 instead.
  • You are rolling out prompt changes using canary deployments. Use PB-RAG-07-01 instead.

3. What you must know first

  1. Groundedness scale and scoring. GroundednessEvaluator scores each response from 1 to 5. A score of 1 means the response is not supported by the context; 5 means it is fully supported. The SDK's default pass threshold is 3. This pack uses 4 as a stricter team default (--min-score 4.0); agree it with the RAG lead.
  2. Evaluation inputs contract. The evaluator requires three string fields per test instance: query (user prompt), context (concatenated text chunks retrieved from search), and response (model-generated answer).
  3. No-answer handling. A refusal such as "I could not find this in the documents you can access." makes no unsupported claim, so it should score high. Check this on your own unanswerable rows before you rely on it. Groundedness does not tell you whether a refusal was correct; check that with the no-answer targets in PB-RAG-06-02.
  4. Judge model and quota. Each row sends the query, context and response to the judge model. Use a capable chat model (this pack uses gpt-4o as a team default) on its own deployment, with enough tokens-per-minute quota to avoid 429 Too Many Requests during CI runs. Check current quotas in the docs.
  5. Groundedness vs answer relevance. Groundedness measures factual truthfulness against context only; it does not measure whether the response answered the user's specific intent. Relevance is measured separately using RelevanceEvaluator.

4. Prerequisites and access

Item Detail
Roles Cognitive Services OpenAI User on the Azure OpenAI resource hosting the judge deployment.
Judge deployment gpt-4o (version 2024-08-06 or later) deployed in Azure OpenAI with dedicated TPM quota.
Tools & packages Python 3.10+, azure-ai-evaluation>=1.1.0, azure-identity>=1.16.0.
Golden question set Approved golden questions (FND-04), for example data/golden/A-policies/A_01/questions.csv, plus unanswerable rows from data/golden/unanswerable/questions.csv. 30 to 100 questions for the PR gate (team default).
Candidate build The candidate prompt and model deployed to a non-production environment, reachable from the pipeline.

Set the environment variables:

export AZURE_OPENAI_ENDPOINT="https://${AZURE_OPENAI_SERVICE}.openai.azure.com"
export JUDGE_DEPLOYMENT="${JUDGE_MODEL_DEPLOYMENT}"
export MIN_PASS_RATE="0.95"   # team default

5. Steps

Placeholders: ${AZURE_OPENAI_SERVICE} (name of your Azure OpenAI resource), ${JUDGE_MODEL_DEPLOYMENT} (name of the judge deployment), ${MIN_PASS_RATE} (pass rate threshold, team default 0.95).

Full evaluation script: snippets/groundedness_gate.py.
Nightly pipeline definition: snippets/nightly-grounding.yml.

Step 1. Build a reference file to test the gate wiring.

This file holds the golden answers and their source passages. Use it only to prove the gate runs and fails when it should (section 6). It does not test a release.

python -c "
import csv, json, os
os.makedirs('eval', exist_ok=True)
with open('data/golden/A-policies/A_01/questions.csv', encoding='utf-8') as fin, open('eval/golden_reference.jsonl', 'w', encoding='utf-8') as fout:
    for r in csv.DictReader(fin):
        fout.write(json.dumps({'query': r['question_text'], 'context': r['source_passage'], 'response': r['answer_text']}) + '\n')
"
head -n 2 eval/golden_reference.jsonl | jq '{query, context, response}'

Verify: each line is valid JSON with non-empty query, context and response.

Step 2. Generate the candidate release answers.

Run every golden question, including the unanswerable rows, through the candidate build. Write one line per question with the question, the text of the chunks that retrieval returned, and the answer the candidate produced. For a refusal, write the approved refusal text (PB-RAG-06-02) as the response.

# pseudo
for q in golden_questions:
    chunks = retrieve(q)                                    # candidate retrieval
    answer = generate_and_verify(q, chunks)                 # PB-RAG-06-01, steps 4 and 5
    response = " ".join(c.text for c in answer.claims) if answer.answerable else REFUSAL_TEXT
    write_jsonl("eval/release_candidate.jsonl",
                {"query": q, "context": "\n\n".join(c["content"] for c in chunks), "response": response})

Verify: eval/release_candidate.jsonl has one line per golden question, and the context field holds the retrieved chunks, not the golden source passage.

The gate script builds the judge from environment variables, as below. You do not need to run this code yourself:

import os
from azure.ai.evaluation import AzureOpenAIModelConfiguration, GroundednessEvaluator
from azure.identity import DefaultAzureCredential

model_config = AzureOpenAIModelConfiguration(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    azure_deployment=os.environ["JUDGE_DEPLOYMENT"],
    credential=DefaultAzureCredential(),
)
evaluator = GroundednessEvaluator(model_config)

Step 3. Execute batch groundedness evaluation.

Run batch evaluation over the dataset using snippets/groundedness_gate.py:

python snippets/groundedness_gate.py \
  --input eval/release_candidate.jsonl \
  --output eval/eval_results.jsonl \
  --threshold ${MIN_PASS_RATE} \
  --min-score 4.0

Verify: the script prints Initialised GroundednessEvaluator, writes a score (1 to 5) per row, and exits with code 0 if the pass rate meets or exceeds ${MIN_PASS_RATE}. Exit code 1 means the gate failed. Exit code 2 means the input or the judge was not available: the gate fails closed and never falls back to a heuristic unless you pass --mock for a local test.

Judge token consumption incurs operational cost

Batch evaluation sends all context and response tokens to the judge model. Check the current price before approval. Keep pull request gates under 100 questions (team default); run larger sets nightly with snippets/nightly-grounding.yml. To stop spending, cancel the pipeline run.

Step 4. Extract failed rows for triage.

Filter the results to the failed rows and the judge's reasons:

jq -c 'select(.groundedness_result == "fail")' eval/eval_results.jsonl > eval/failed_rows.jsonl
wc -l eval/failed_rows.jsonl

Verify: failed_rows.jsonl holds only rows with "groundedness_result": "fail", each with its context, response and groundedness_reason.

Step 5. Integrate the regression gate into CI/CD pipelines.

Add the evaluation script as a required pipeline step in your release workflow:

- script: |
    python snippets/groundedness_gate.py \
      --input eval/release_candidate.jsonl \
      --output $(Build.ArtifactStagingDirectory)/eval_results.jsonl \
      --threshold $(MIN_PASS_RATE) \
      --min-score 4.0
  displayName: "Groundedness Evaluation Release Gate"
  env:
    AZURE_OPENAI_ENDPOINT: $(AOAI_ENDPOINT)
    JUDGE_DEPLOYMENT: $(JUDGE_DEPLOYMENT)

Verify: the pipeline runs step 2 and then the gate on each candidate build, and stops the deployment on any non-zero exit code.

Step 6. Schedule nightly production audit.

Add a job that exports a sample of the previous day's production traces to eval/production_sampled.jsonl (query, retrieved context, response). Remove personal data before export. Then deploy snippets/nightly-grounding.yml.

Verify: the scheduled pipeline triggers at 02:00 UTC, scores the sampled rows and publishes GroundednessNightlyResults.

6. Verify it works

Run the gate on the reference file from step 1 with one answer replaced by an ungrounded one. Use the real judge, not --mock:

python -c "
import json
with open('eval/golden_reference.jsonl', encoding='utf-8') as f:
    rows = [json.loads(line) for line in f][:10]
rows[0]['response'] = 'Out-of-domain ungrounded answer not found in source text.'
with open('eval/test_seeded_failures.jsonl', 'w', encoding='utf-8') as f:
    for r in rows: f.write(json.dumps(r) + '\n')
"
python snippets/groundedness_gate.py --input eval/test_seeded_failures.jsonl --threshold 0.95
  • Verify: The command exits with code 1, reporting a pass rate below 95%.
  • Verify: run against eval/golden_reference.jsonl (no seeded failure), the command exits with code 0.
  • Verify: with AZURE_OPENAI_ENDPOINT unset and no --mock, the command exits with code 2.

7. Known issues and fixes

Symptom Likely cause Fix
HTTP 429 Too Many Requests during batch evaluation Judge deployment exceeded TPM rate limits Allocate dedicated TPM quota for the judge model, implement backoff retries, or reduce evaluation concurrency.
Groundedness score drops across entire test set after model upgrade New generation model paraphrases more freely or injects parametric knowledge Tighten system prompt grounding instructions, require verbatim extraction, or calibrate judge threshold.
Evaluator returns error on specific rows (scored 1, reason Evaluator call error) Empty context string or null response field in JSONL Validate the input; write the approved refusal text instead of an empty response.
Gate exits with code 2 in CI Judge endpoint or deployment variable not set, or azure-ai-evaluation not installed Set AZURE_OPENAI_ENDPOINT and JUDGE_DEPLOYMENT from the variable group, and install the package. Do not add --mock.
Gate passes but users still see ungrounded answers Gate scored the golden answers instead of the candidate's answers Generate release_candidate.jsonl from the candidate build (step 2).
Long evaluation runtime in CI/CD pipeline Sequential row evaluation over large golden sets Use parallel async workers or evaluate a stratified representative sample for PR gates and run full sets nightly.
Valid numerical paraphrases flagged as ungrounded Judge model struggles with implicit unit conversions Standardise numerical representations in chunking and prompt generation templates.

8. Monitoring and alerts

The thresholds below are illustrative team defaults; tune them against your measured production baseline.

Signal Source Alert when Action
GroundednessPassRate Release pipeline output Below 95% on the release candidate set (team default) Stop the release; inspect failed_rows.jsonl.
GroundednessDropFromBaseline Release pipeline output More than 2 percentage points below the production baseline (team default) Block the release; review the prompt or model change with the RAG lead.
ProductionSampledGroundedness Nightly batch evaluation 7-day moving average below 92% (team default) Start runbook PB-RAG-06-04.
JudgeModelThrottlingRate Azure Monitor metric 429 responses above 1% of judge calls in a run (team default) Raise the judge deployment's quota or lower concurrency.

9. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon Bedrock Bedrock Model Evaluation & Guardrails Contextual Grounding Amazon Bedrock provides built-in model evaluation jobs for faithfulness and runtime Contextual Grounding checks in Bedrock Guardrails. Real-time guardrail checks return a confidence score between 0.0 and 1.0.
RAGAS (Open-source) faithfulness metric Calculates the ratio of claims in the generated answer that can be inferred from the context. Requires an OpenAI-compatible judge LLM and returns a continuous float score between 0.0 and 1.0.
DeepEval (Open-source) FaithfulnessMetric Evaluates whether the actual output contains contradictions or ungrounded claims against retrieval context. Supports customizable thresholds, reasoning output, and local LLM backends.
Promptfoo (Open-source) context-faithfulness assertion Model-graded assertion designed for CI/CD test suites. Evaluates responses against context using LLM grading prompts and exports HTML/JSON test reports.

10. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: Build golden_reference.jsonl (step 1) and replace 10% of the answers with ungrounded ones. Run the evaluation pipeline with a minimum pass rate of 95%. Verify that the gate fails the build with a non-zero exit code, flags all seeded rows in the triage log, and passes after removing seeded rows.

Pass criteria: Groundedness evaluation detects 100% of seeded ungrounded answers, logs row-level reasoning, and correctly blocks pipeline progression.

Record

Item Detail
Playbook PB-RAG-06-03 at commit [hash]
Dry-run by Yasar
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by Deeparani

11. Sources

Checked on 30 October 2026.