⚠️ Before You Start

Risk: Medium

Duration: 2-3 days for the first 50 labelled questions (team default)

Last Validated: Not yet validated (draft)

Kind: How-to

Risk Note: Medium (a biased golden set gives false confidence)

Build a golden set and measure recall@k, MRR and nDCG

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-18). Author: Prasanna · dry-run: Devadharshini Raja · sign-off: Deeparani. Work items: RAG-02-T01, RAG-02-T02, RAG-02-T03, RAG-02-T04.

1. Summary

This how-to guides the creation of a ground-truth golden dataset (questions and graded document relevancies) and establishes quantitative information retrieval (IR) ranking metrics—recall@k, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG). It provides a standardized Python evaluation harness using ranx to measure baseline retrieval performance and gate search pipeline modifications before production deployment.

Risk: Medium (a biased or unrepresentative golden set produces misleading metrics and gives false operational confidence). Typical duration: 2-3 days for the first 50 labelled questions (team default).

2. When to use and when not to

Use this how-to when:

• Establishing the initial retrieval evaluation baseline for a new RAG knowledge corpus before tuning parameters.
• Re-calibrating evaluation datasets after major corpus updates or significant shifts in user query patterns.
• Comparing candidate retrieval configurations (e.g. pure keyword, dense vector, hybrid search, semantic ranker) against an established baseline.
• Validating changes to chunking strategies, embedding models, or indexing parameters against measured ranking metrics.

Do not use it when:

• Evaluating end-to-end LLM generation qualities such as groundedness, hallucinations, or answer relevance. Use PB-RAG-02-02 (Azure AI Evaluation SDK) instead.
• Triaging an active production quality incident to isolate whether an issue stems from retrieval or generation. Follow PB-RAG-02-03 instead.
• Executing a fast release readiness gate checklist for prompt, model, or index promotions. Follow PB-RAG-02-04 instead.

3. What you must know first

1. Qrels (Query Relevance Judgements). Qrels represent the ground-truth map associating query IDs with relevant document IDs and graded relevance levels (0 = not relevant, 1 = partially relevant, 2 = highly relevant; team default) .
2. Run files. A run file records the output of an Information Retrieval system, associating each query ID with retrieved candidate document IDs and rank scores.
3. Metric specialisation. recall@k measures the proportion of relevant documents captured in the top $k$; MRR@k measures how quickly the user encounters the first relevant document; nDCG@k evaluates the entire ranking order, rewarding runs that place the most relevant documents higher.
4. Stable document ID anchoring. Ground truth must reference stable document IDs (such as parent document URI or storage key) rather than ephemeral chunk IDs, because chunk hashes change when chunk size or splitter settings are modified.
5. Statistical hypothesis testing. Metric improvements must be validated with statistical hypothesis tests (such as paired Student's t-test, Fisher's randomization test, or Tukey's HSD test via ranx.compare(..., max_p=0.05)) before declaring a candidate configuration superior.
6. Unanswerable questions in the golden set. A production golden set must include unanswerable questions (marked with "unanswerable": true in the query manifest; team default: at least 5 questions and at least 10% of the set) to evaluate no-answer and refusal boundaries (PB-RAG-06-02, PB-RAG-02-04). Because unanswerable questions have no relevant documents in the corpus, they must be omitted from qrels.json. ranx evaluates ranking metrics only over queries present in the qrels, so unanswerable questions do not distort retrieval metrics while remaining available for generation evaluation.

4. Prerequisites and access

Item	Requirement
Access	Read access to the search index and access to source raw documents
Tooling	Python 3.11 with ranx>=0.3.16
Corpus Data	Ingested search index with stable document IDs or parent document metadata
Labeling Capacity	1-2 domain Subject Matter Experts (SMEs) allocated for relevance grading
Query Intake	50-100 representative questions (team default: at least 50 questions) collected from user search logs, support tickets, or workshops

5. Steps

Step 1. Sample representative queries across user archetypes

Curate a representative set of at least 50 questions (team default) covering fact-seeking, multi-hop, exploratory, and keyword-style queries from user logs or stakeholder workshops. Include out-of-scope or unanswerable queries (team default: at least 5 questions and at least 10% of the set, marked "unanswerable": true). Categorize each query into a metadata manifest:

[
  {
    "query_id": "q_001",
    "query": "What is the travel expense submission deadline?",
    "unanswerable": false
  },
  {
    "query_id": "q_002",
    "query": "Who must approve expense reports submitted after 60 days?",
    "unanswerable": false
  },
  {
    "query_id": "q_003",
    "query": "What is the company policy on pet insurance reimbursement?",
    "unanswerable": true
  }
]

Verify: The query set contains at least 50 unique questions (team default) with at least 5 unanswerable questions (at least 10% of the set; team default), saved in golden/queries.json.

Step 2. Grade relevance with domain experts (create qrels)

Have domain experts review retrieved passages for each answerable query and assign relevance scores: 0 (not relevant), 1 (partially relevant), or 2 (directly answers query) . Format the output into standard TREC/ranx JSON format:

{
  "q_001": {
    "doc_manual_v1": 2,
    "doc_faq_v2": 1
  },
  "q_002": {
    "doc_policy_2026": 2
  }
}

Omit unanswerable queries from golden/qrels.json: because no document in the corpus is relevant, unanswerable queries have no relevance judgements. ranx scores only the queries present in qrels.json, allowing candidate run files to contain unanswerable queries without distorting retrieval metrics.

Verify: File golden/qrels.json validates with json.load() and contains graded scores for every answerable query in the golden set.

Step 3. Export baseline retrieval run from the search service

Execute each evaluation query against the target search index requesting top-20 results using the evaluation export utility (snippets/rag-eval-gate/eval/export_run.py):

python snippets/rag-eval-gate/eval/export_run.py \
    --search my_app.retrieval:search \
    --golden golden/queries.json \
    --out runs/baseline.json \
    --top-k 20

The script calls your application's search function and records the query ID, document ID, and search score:

{
  "q_001": {
    "doc_manual_v1": 0.88,
    "doc_guide_v3": 0.65
  },
  "q_002": {
    "doc_policy_2026": 0.94
  }
}

Verify: File runs/baseline.json contains retrieval scores for all query IDs matching golden/queries.json.

Step 4. Compute ranking metrics and compare candidate runs

Execute the evaluation script (snippets/rag-eval-gate/eval/eval_retrieval.py) to calculate recall@10, mrr@10, and ndcg@10, and assess statistical significance ($p \le 0.05$):

python snippets/rag-eval-gate/eval/eval_retrieval.py \
    --candidate runs/candidate.json \
    --baseline runs/baseline.json \
    --qrels golden/qrels.json \
    --k 10 \
    --tolerance 0.02

Under the hood, the evaluation harness loads runs and qrels into ranx to evaluate metrics and compare distributions:

import json
import sys

from ranx import Qrels, Run, compare, evaluate

with open("golden/qrels.json") as f:
    qrels = Qrels(json.load(f))
with open("runs/baseline.json") as f:
    baseline = Run(json.load(f), name="baseline")
with open("runs/candidate.json") as f:
    candidate = Run(json.load(f), name="candidate")

metrics = ["recall@10", "mrr@10", "ndcg@10"]
print(compare(qrels=qrels, runs=[baseline, candidate], metrics=metrics, max_p=0.05))

base = evaluate(qrels, baseline, metrics)
cand = evaluate(qrels, candidate, metrics)
tolerance = 0.02
failed = [m for m in metrics if cand[m] < base[m] - tolerance]
if failed:
    print("Regression on:", failed)
    sys.exit(1)

Verify: Script executes cleanly, printing the comparison table with metric scores and p-values.

Step 5. Commit baseline scores and version golden artifacts

Record the baseline metrics in the repository catalog and commit golden/qrels.json and runs/baseline.json into version control under git LFS or release tags .

Verify: git status confirms artifacts committed; baseline scores documented in evaluation tracking logs.

Snippets referenced:

• snippets/rag-eval-gate/eval/export_run.py: Export candidate and baseline retrieval runs from the search application
• snippets/rag-eval-gate/eval/eval_retrieval.py: Evaluate and compare retrieval runs using ranx with statistical testing
• snippets/qrels.sample.json: Qrels format example (json)

6. Verify it works

A second engineer runs the verification script against a 30-question synthetic golden set in the sandbox environment:

1. Build a 30-question golden set (27 answerable and 3 unanswerable questions; team default) for the sandbox search index and generate golden/qrels.json for the 27 answerable queries.
2. Export two runs from the sandbox search index using snippets/rag-eval-gate/eval/export_run.py:
3. runs/keyword.json: baseline run using keyword-only (BM25) search.
4. runs/hybrid.json: candidate run using hybrid search (keyword + dense vector).
5. Run the retrieval evaluation comparing both runs:

python [`snippets/rag-eval-gate/eval/eval_retrieval.py`](snippets/rag-eval-gate/eval/eval_retrieval.py) \
    --candidate runs/hybrid.json \
    --baseline runs/keyword.json \
    --qrels golden/qrels.json \
    --k 10

1. Verify the outputs:
2. Verify: The comparison report outputs recall@10, mrr@10, and ndcg@10 for both runs.
3. Verify: The results are reproducible (running the evaluation again produces identical metric values).
4. Verify: The report indicates which run wins and displays the statistical significance ($p$-value via paired Student's t-test).

7. Known issues and fixes

Symptom	Likely cause	Fix
Scores drop sharply after a re-chunk with no retrieval degradation	Qrels reference ephemeral chunk IDs that changed during re-chunking	Map chunk retrieval hits back to stable parent document IDs before computing ranking metrics
Scores look artificially high (near 1.0)	Evaluation queries were extracted directly from indexed chunks	Add queries from genuine user search logs, support tickets, and natural human paraphrases
Metric deltas fluctuate between runs	Query sample size is too small (<50 queries, team default) to achieve statistical power	Expand evaluation set to $\ge 50$ queries (team default) across distinct topics and apply ranx.compare(max_p=0.05)
KeyError or ValueError during Qrels loading	Document ID mismatch between qrels JSON and search run JSON	Standardize ID casing and strip URI schemes before writing qrels and run dictionaries

8. Monitoring and alerts

The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline.

Signal	Alert when	Action
nDCG@10 in nightly regression run	Drops >2% below stored baseline on golden set	Block index/prompt release, file quality incident, and execute triage runbook PB-RAG-02-03
Recall@10 in CI pull request check	Fails statistical comparison ($p < 0.05$ with negative delta)	Reject pull request; tune hybrid search weights or re-ranking model before merging

9. AWS and open-source equivalents

Platform	Equivalent	Key differences
Azure AI Search	Evaluated via external runs	Azure AI Search does not provide a native ranking evaluation API; relevance and ranking (keyword, vector, hybrid, semantic ranker) are evaluated by exporting query results into run files and assessing with open-source ranx
Amazon Bedrock Knowledge Bases	Retrieve evaluation jobs	Managed prompt dataset evaluation with automated LLM relevance judgements
Open-source (ranx)	ranx ranking evaluation library	Standard TREC metric evaluation supporting statistical comparison (paired Student's t-test, Fisher's randomization test, Tukey's HSD)
Open-source (ragas / deepeval)	Context Precision and Context Recall metrics	LLM-assisted context assessment evaluating retrieval quality without exhaustive human labeling

10. Validation record

Pending. Planned dry-run scenario: A second engineer builds a 30-question golden set for a sandbox index, exports two runs (keyword only and hybrid), and computes the metrics. Pass if results are reproducible on rerun and the comparison report shows which run wins and with what p-value.

Record

Item	Detail
Playbook	PB-RAG-02-01 at commit [hash]
Dry-run by	[Name] (must not be author or reviewer)
Date	YYYY-MM-DD
Sandbox	[Resource group or tag, tier and region]
Scenario	[Scenario followed]
Time taken	[Hands-on time] hands-on, [waiting time] waiting
Result	[PASS / PASS WITH NOTES / FAIL]
Notes	1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by	[Reviewer name and date]

11. Sources

Checked on 26 October 2026.

• AmenRa, ranx: ranking evaluation library (GitHub): Qrels, Run, metrics, statistical comparison.
• Ragas, Ragas: list of available metrics: context precision, context recall.
• DeepEval, DeepEval: Contextual Precision metric: context ranking and precision metrics.
• AWS Documentation, Create a prompt dataset for retrieve-only RAG evaluation jobs (Amazon Bedrock): Bedrock knowledge base retrieve-only evaluation.

Related playbooks

• PB-RAG-02-02: Run RAG quality evaluation with the Azure AI Evaluation SDK (how-to).
• PB-RAG-02-03: Triage 'the answers feel wrong': retrieval or generation? (runbook).
• PB-RAG-02-04: Evaluation gate checklist for prompt, model and index changes (checklist).

📄 Snippet CODE

📋 Copy ↗ Raw ×