Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 hours to wire into CI, minutes per run (team default; measure it in your sandbox)
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Medium (a missing gate lets regressions reach users)

Evaluation gate checklist for prompt, model and index changes

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-17). Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: RAG-02-T13, RAG-02-T14, RAG-02-T15, RAG-02-T16.

1. Summary

On every pull request that touches prompts, model deployments, chunking, index schema or retrieval settings, this gate runs your application's own search and answer code on a golden set. It scores retrieval (MRR, recall and nDCG) and the answers (groundedness), and it blocks the merge if a score falls by more than the agreed tolerance.

Risk: Medium (a missing gate lets regressions reach users). Typical duration: 1-2 hours to wire into CI, minutes per run (team default; measure it in your sandbox).

Scope and boundaries

This checklist covers wiring and running the CI evaluation gate. It does not cover:

Link to those playbooks instead of repeating their content here.

2. When to use and when not to

Use it when a pull request touches any of the following:

  • Prompt changes — system prompt edits, few-shot examples, instruction rewording.
  • Model deployment changes — swapping the chat or embedding model, changing model version or API endpoint.
  • Chunking logic changes — chunk size, overlap, splitting strategy, or pre-processing steps.
  • Index schema changes — adding or removing fields, changing field types, vector dimensions, or similarity algorithm.
  • Retrieval settings changes — top-K, hybrid search weights, re-ranker, query expansion, or filter logic.

Do not use it when:

  • Production incidents requiring immediate rollback — use PB-RAG-02-03 to triage and identify the regression first, then open a fix PR that runs this gate normally.
  • Scheduled nightly / offline evaluation runs — those are handled by PB-RAG-02-02; this checklist is for PR-gating only, not batch reporting.
  • Documentation-only or test-data-only changes — changes that cannot affect retrieval or generation quality do not need to trigger the evaluation gate. Use path filters in CI to exclude them (e.g. docs/**, *.md outside prompts/).
  • Exploratory sandbox work — ad-hoc experiments that are never merged to main can skip the gate; ensure the branch is never set as a base for a production PR.

3. What you must know first

Key concepts

Term Definition Where to learn more
Golden set A fixed, reviewed set of query-expected-answer pairs used as the benchmark PB-RAG-02-01
Baseline The run (and its latency) recorded from the last accepted state of the system PB-RAG-02-01
Judge variance Random variation in LLM-judge scores across runs on identical inputs; use tolerance bands and multiple runs to manage it PB-RAG-02-01
Application contract The two functions the gate calls from your application, search and answer. Because the gate runs your real code, a pull request that changes retrieval, the prompt or the model settings changes the scores. Section 4.4

Rules of thumb

  • Gate on both ranking metrics and groundedness — neither alone is sufficient.
  • Baselines must be updated deliberately, through a reviewed commit approved by the RAG Lead.
  • promptfoo can assert context-faithfulness per test case. The default threshold is 0.5; the value 0.9 used in the snippets is a team default — adjust it after measuring in your sandbox. (promptfoo context-faithfulness)
  • The Azure AI Evaluation SDK evaluators score on a 1-5 scale with a default pass threshold of 3. Check the default for the SDK version you pin. (Local evaluation)
  • ranx needs the run and the qrels to hold the same questions. The gate scores only the questions that have qrels, so unanswerable questions do not affect the retrieval metrics. (ranx)

4. Checklist

Split into five sub-sections. §4.1 must be satisfied before any run. §4.2 is one-time setup. §4.3 runs on every PR. §4.4 covers CI wiring. §4.5 covers gate failures.

Judge token spend

LLM-judge runs consume tokens and incur cost. Check the current price of the judge model before approving the evaluation budget. Consider limiting the judge run to a sampled subset of the golden set for routine PRs.

4.1 Prerequisites

  • Pre1. RAG Lead. Confirm the golden set includes unanswerable questions, marked "unanswerable": true. See PB-RAG-02-01 or PB-RAG-06-02. Verify: the golden set holds at least 5 unanswerable questions, and at least 10% of its questions are unanswerable (team default; confirm it after the sandbox runs).
  • Pre2. RAG Lead. Confirm that a signed-off golden set of at least 50 questions (team default) and its qrels exist in the sandbox (baselines/golden_set.json and baselines/qrels.json). Verify: both files are present and signed off before the dry-run.

4.2 One-time setup

  • S1. DevOps engineer. Confirm that your application has the two functions of the contract in §4.4: search(query, top_k) and answer(query, chunks). Verify: you can import both from the repository root.
  • S2. DevOps engineer. Copy the gate files from snippets/rag-eval-gate/ in this pack into your project repository as listed in §4.4. Pin the promptfoo version in the PROMPTFOO_VERSION pipeline variable. Verify: pip install -r eval/requirements.txt succeeds, and python eval/export_run.py --help runs.
  • S3. DevOps engineer and RAG Lead. Sign in with Microsoft Entra ID, never an API key. In Azure DevOps, use an Azure service connection and store the variables of §4.4 in pipeline variables or a variable group (Pipelines → Library → Variable groups). Fork PRs must not receive secrets. Verify: the pipeline runs on an internal branch with no API key in the variables.
  • S4. DevOps engineer. Make the gate a required Build Validation policy on main (Project Settings → Repositories → Policies → Build Validation), with a path filter for your prompt, retrieval, index and config folders. Add an Automatically included reviewers policy, set to Required, for the baselines/ path with the RAG Lead as reviewer, so that a baseline change cannot merge without their approval. Add the same for eval/ if you also want the gate code protected. (Branch policies) The GitHub equivalent is a required status check plus CODEOWNERS (see §4.4). Verify: a PR that fails the gate cannot be completed on main without an approved exception, and a PR that changes baselines/ cannot be completed without the RAG Lead's approval.
  • S5. RAG Lead. Record the first baseline from the accepted state of main: run python eval/export_run.py with --out baselines/baseline.json --meta-out baselines/baseline.meta.json. Commit both files through a reviewed PR. Verify: both files exist and are signed off.

4.3 Every pull request

  • P1. CI (automated). The gate starts on a pull request that touches prompts/**, retrieval/**, index/** or config/**. The config/** path makes model deployment and temperature changes trigger the gate. Verify: the pipeline starts within 10 minutes of PR creation (team default; the dry-run records the real figure).
  • P2. CI (automated). eval/export_run.py calls your search and answer functions for every golden question. Verify: runs/ holds candidate.json, candidate.meta.json, candidate_cases.jsonl and candidate_promptfoo.jsonl, and the step reports no error.
  • P3. CI (automated). eval/eval_retrieval.py compares the run with the baseline using ranx on MRR@10, Recall@10 and nDCG@10. The tolerance is ±0.02 per metric. This is a team default — measure it in your sandbox before you lock it. Verify: the gate passes if every metric is within tolerance. It exits with code 1 if one regresses.
  • P4. CI (automated). Judge the answers. Route A (promptfoo): npx promptfoo@<pinned-version> eval --config eval/promptfooconfig.yaml runs a context-faithfulness assertion on the answers that your code produced. The echo provider returns each answer as it is, so promptfoo generates nothing. The team default threshold is 0.9. Route B (Azure AI Evaluation SDK): python eval/eval_azure_ai.py scores groundedness, with a default pass threshold of 3 on a 1-5 scale. Verify: route A: every test is at or above the threshold, the report is reports/promptfoo-report.json, and promptfoo exits with code 100 if a test fails. Route B: the average groundedness is at least 3.0, and the report is reports/azure-eval-report.json.
  • P5. CI (automated). The reports in reports/ and the runs in runs/ are published as build artefacts. Verify: links to them are visible on the PR.
  • P6. PR author. Read the reports before you ask for a review. Verify: there is no unexplained regression.
  • P7. RAG Lead. For schema PRs (index schema changes), build a candidate index in the sandbox and re-run the baseline before merging. Verify: the candidate index exists and the baseline is refreshed.
  • P8. RAG Lead. For embedding model or chunking changes, build a new index and record a new baseline before the PR can merge. Do not compare with the old-model baseline. Verify: the new index and the new baseline are committed through a reviewed PR.
  • P9. CI (automated) and PR author. Check that unanswerable questions get a refusal, not a made-up answer. export_run.py adds a refusal assertion (llm-rubric) to every golden row marked "unanswerable": true. Verify: every unanswerable row passes the assertion in reports/promptfoo-report.json.
  • P10. CI (automated) and PR author. Check the search latency. eval_retrieval.py prints the p95 latency of the candidate next to the baseline's. The pipelines pass --max-p95-increase 0.25, so the gate fails if p95 rises by more than 25% (team default; confirm it in the sandbox, because p95 on a small golden set is noisy). Verify: p95 is reported, and the rise is no more than 25%.
  • P11. CI (automated). The shape of the run is validated before scoring. Verify: a run with a missing golden question or a non-numeric score stops with SETUP ERROR and exit code 2, not with a silent pass.

4.4 CI setup — Azure and GitHub

All gate files are in snippets/rag-eval-gate/ in this pack. Copy them into your project repository. The folders the gate reads (prompts/, retrieval/, baselines/, runs/, reports/) are in your project, not in this repository.

Snippet file Copy it to Purpose
azure-pipelines-rag-eval.yml Your repository, then create a pipeline from it Azure DevOps pipeline (recommended)
github-actions/rag-eval.yml .github/workflows/ GitHub Actions equivalent
eval/export_run.py eval/ Calls your search and answer code, writes the run and the cases
eval/eval_retrieval.py eval/ ranx comparison with the baseline
eval/promptfooconfig.yaml eval/ promptfoo judge assertions (route A)
eval/eval_azure_ai.py eval/ Azure AI Evaluation SDK judge (route B)
eval/requirements.txt eval/ Pinned Python packages

The application contract

The gate must run the code that a pull request changes. So it calls two functions from your application. It does not run a query of its own.

# retrieval/search.py  (your application's own code)
def search(query: str, top_k: int) -> list[dict]:
    """Ranked results, best first: [{"id": "<document id>", "score": 0.82, "content": "<chunk text>"}]."""

# app/answer.py  (your application's own code)
def answer(query: str, chunks: list[dict]) -> str:
    """The application's answer, built with its real prompt and model settings."""

Pass them to the gate as --search retrieval.search:search and --answer app.answer:answer. The id must be the document ID used in the qrels. Your application also needs its usual settings. In CI, point them at the test index.

Pipeline variables

Variable Used by Meaning
SERVICE_CONNECTION Azure DevOps Azure service connection that signs in with Entra ID
SEARCH_FUNCTION, ANSWER_FUNCTION export_run.py The two functions of the contract
AZURE_OPENAI_HOST promptfoo Host of the judge resource, for example yourresource.openai.azure.com (a host, not a URL)
AZURE_OPENAI_ENDPOINT route B Endpoint of the judge resource
JUDGE_DEPLOYMENT route B and promptfooconfig.yaml Name of the judge deployment. In promptfooconfig.yaml the provider id is literal, so edit the name there.
PROMPTFOO_VERSION promptfoo Pinned promptfoo version, for example 0.124.1
Your application's own settings Your code Endpoint and index that your code reads, set to the CI test index

Wire the pipeline up with the Build Validation policy in S4. That policy runs the pipeline on a pull request, so the file has no pr: trigger. The pipeline publishes its reports with PublishPipelineArtifact@1, which Microsoft recommends over PublishBuildArtifacts@1 on Azure DevOps Services. (PublishPipelineArtifact@1)

# Full file: snippets/rag-eval-gate/azure-pipelines-rag-eval.yml
- task: AzureCLI@2
  displayName: Run the evaluation gate
  inputs:
    azureSubscription: $(SERVICE_CONNECTION)
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      python eval/export_run.py --search "$(SEARCH_FUNCTION)" --answer "$(ANSWER_FUNCTION)" --out runs/candidate.json
      python eval/eval_retrieval.py --candidate runs/candidate.json --baseline baselines/baseline.json
      npx promptfoo@$(PROMPTFOO_VERSION) eval --config eval/promptfooconfig.yaml

The gate steps run inside AzureCLI@2, so the Azure CLI session is signed in. promptfoo authenticates to Azure OpenAI with that session, as its Azure provider documents. (promptfoo Azure provider) The full file runs both gates and fails if either fails.

Alternative judge route: Azure AI Evaluation SDK

Teams that prefer Python to Node.js can run the judge step with the Azure AI Evaluation SDK. It reads runs/candidate_cases.jsonl, which holds the query, the retrieved context and the application's answer on each line. See PB-RAG-02-02 for the full evaluation set-up.

python eval/eval_azure_ai.py --threshold 3.0 --output reports/azure-eval-report.json

The script signs in with DefaultAzureCredential and passes it to the SDK as a credential, not an API key.

GitHub Actions (alternative)

See snippets/rag-eval-gate/github-actions/rag-eval.yml. It signs in with OIDC (azure/login), with no API key. Pull requests from forks cannot get an OIDC token, so the job runs only for branches in the same repository. Add a CODEOWNERS entry that requires a RAG Lead review for baselines/ and eval/.

4.5 When the gate fails

Situation Who approves Process
Metric regression beyond tolerance RAG Lead Triage using PB-RAG-02-03. Fix the root cause; do not lower the threshold without review.
Intentional baseline change (accepted regression trade-off) RAG Lead + Secondary Approver Open a separate PR that updates only the baseline files. Requires explicit written justification in the PR description.
Exception to blocking the merge Playbook Sign-off Owner (GenAI Practice Lead) Must be documented in the PR with a time-bound remediation plan. Use sparingly.

Role definitions for this table:

  • RAG Lead — the engineer accountable for retrieval quality on this workstream.
  • Secondary Approver — a senior engineer or architect who is not the PR author and is not the RAG Lead on this PR. Must have reviewed the justification and the metric delta before approving.
  • Playbook Sign-off Owner (GenAI Practice Lead) — the practice-level role with authority to grant exceptions to required gates. Must not be the PR author or the RAG Lead on the same PR.

5. Known issues and fixes

Symptom Likely cause Fix
Gate is flaky LLM judge variance on a small test set Raise the sample size (team default: at least 50 questions, so that one question moving a rank stays inside the ±0.02 tolerance); use tolerance bands (team default ±0.02, measure in the sandbox). The judge temperature is set to 0 in promptfooconfig.yaml.
Gate passes a PR that removes the vector query The export does not call the application's own search code Point --search at the function that the application really uses. Prove it in the dry-run with a PR that removes the vector query.
Gate passes but production degrades CI index differs from production data Refresh the CI index snapshot every month, and after any production re-index or schema change (team default). Re-record the baseline after each refresh, through a reviewed PR.
Baseline creeps down Baseline updated after each regression Require RAG Lead + secondary approval for all baseline changes (see §4.5).
429 / rate-limit errors from the judge model Judge token spend exceeds the deployment quota during the run Lower concurrency with the promptfoo --max-concurrency option, or request a quota increase. Check the current price and quota before approving the budget.
Evaluation run times out in CI The export takes too long against a large index Cap --top-k, check the p95 latency (P10), and use a smoke subset of the golden set for pull requests.
SETUP ERROR and exit code 2 from eval_retrieval.py A run or the qrels is missing a question, or has the wrong shape Fix the file, or regenerate it with export_run.py.
Cannot load 'module:function' from export_run.py Wrong function path, or the script is not run from the repository root Run from the repository root and check the --search and --answer values.
promptfoo fails to call the judge No Azure sign-in in the step, or AZURE_OPENAI_HOST is a URL Run the step inside AzureCLI@2 (or after azure/login), and give the host only.

6. Monitoring and alerts

Signal Alert when Action
CI gate failure rate on main 3 or more gate failures in a rolling 7-day window (team default) Stop merges and triage using PB-RAG-02-03.
Baseline file changed Any pull request that touches baselines/** The required-reviewer policy (S4) asks the RAG Lead to approve before the merge. If a baseline change reaches main without that approval, revert it and open a justification PR.
Judge token spend Monthly judge cost reaches 80% of the monthly budget (team default). The RAG Lead sets the budget after the dry-run has measured the real spend. Check the current price before approval, and create the alert in Azure Cost Management. Notify the RAG Lead and sample a subset of the golden set for routine PRs.

7. AWS and open-source equivalents

The same gate works with AWS CodePipeline or any CI. The application contract and the ranx comparison do not change. On AWS, Amazon Bedrock evaluations include RAG evaluations that use LLMs to score a knowledge base's retrieval and generated answers on a dataset of questions with ground truth (Amazon Bedrock evaluations). Use one as the judge step.

On open-source stacks, ranx scores retrieval and promptfoo runs the judge assertions, as in this checklist. DeepEval is another option: it runs LLM evaluations with pytest-style commands (DeepEval).

8. Validation record

Pending. Planned dry-run scenario: A second engineer opens two sandbox pull requests, one that degrades retrieval by removing the vector query. Pass if the gate blocks the bad change and passes the clean one.

9. Sources