Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 hours per review
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Medium (an unchecked ranking change affects every query)

Relevance go-live and ranking change checklist

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Verified and completed against official documentation on 02 Nov 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-20). Author: Gayathri Ramachandran · dry-run: Yasar · sign-off: Deeparani. Work items: RAG-08-T13, RAG-08-T14, RAG-08-T15, RAG-08-T16.

1. Summary

This checklist gates every production go-live, ranking parameter update, scoring profile modification, and semantic configuration change. It ensures that retrieval quality is measured on an unexposed golden query set, configuration drift is caught before deployment, and automated rollbacks are tested.

Facts checked against Microsoft Learn on 02 November 2026. Data-plane REST API: 2024-07-01.

2. When to use and when not to

Use this checklist when:

  • You deploy a new RAG knowledge base to production for the first time.
  • You change hybrid search weights, vector query parameters (k, oversampling), or text-vector balance.
  • You create or modify semantic configurations, field priorities, or enable the semantic ranker.
  • You cut over traffic to a rebuilt search index via an index alias after chunking, analyser, or embedding updates.
  • You perform scheduled quarterly relevance audits to detect configuration drift between Git and live search services.

Do not use it when:

  • You are deciding which retrieval mode (keyword, vector, hybrid, or re-ranker) to implement from scratch. Use PB-RAG-08-01 instead.
  • You are tuning hybrid weights and testing score distributions in the sandbox. Use PB-RAG-08-02 instead.
  • You are actively diagnosing and mitigating an ongoing live relevance incident. Use PB-RAG-08-03 instead.
  • You are rebuilding an index or modifying schema fields requiring zero-downtime alias cutover. Use PB-RAG-09-01 instead.

3. What you must know first

  1. In-place updates vs index rebuilds. Semantic configurations, scoring profiles, and query-time weights can be updated in place without downtime or reindexing. In contrast, modifications to field types, analysers, tokenizers, vector dimensions, or embedding models require creating a new index and switching traffic via an alias (PB-RAG-09-01).
  2. Two-stage retrieval architecture. Hybrid search runs L1 retrieval by executing keyword (BM25) and vector (HNSW/Exhaustive KNN) searches in parallel, merging candidate lists using Reciprocal Rank Fusion (RRF). The optional semantic ranker acts as an L2 re-ranker, scoring only the top 50 documents (@search.rerankerScore from 0.00 to 4.00) using multi-lingual cross-encoder models.
  3. Reciprocal Rank Fusion score mechanics. RRF computes document scores as $\sum \frac{1}{60 + \text{rank}}$. It standardises disparate score ranges into a uniform distribution. Adjusting weight on vector queries biases rank placement before RRF fusion, altering document inclusion in the top 50 window sent to L2 re-ranking.
  4. Held-out evaluation sets. Never measure go-live quality solely on queries used during prompt drafting or weight tuning. Quality gating requires an isolated, held-out golden set with diverse query archetypes (keyword, natural language questions, multi-lingual, and domain acronyms).
  5. Configuration as code and drift prevention. Manual edits in the Azure portal cause silent divergence from infrastructure code. All index JSON schemas, semantic configurations, and application search parameters must reside in version-controlled repositories and be verified prior to release.
  6. Latency and cost constraints. Semantic ranking incurs additional latency and is billed per query (check current prices and service quotas in official documentation). Autocomplete, keystroke suggestions, and non-conversational programmatic queries must bypass semantic re-ranking.

4. Checklist

Stage 1: Pre-flight and baseline verification

  • Change ticket and approvals (RAG Lead): Verify a change ticket exists specifying candidate ranking settings, target index, rollback owner, and approved deployment window.
  • Infrastructure as Code review (DevOps engineer): Confirm that index definitions, scoring profiles, and semantic configurations are committed to Git and peer-reviewed in a pull request.
  • Production baseline capture (GenAI engineer): Record pre-change baseline metrics from production over the past 7 days: p95 latency (SearchLatency), query volume (QueriesPerSecond), and golden set nDCG@10.
  • Quota and service tier check (DevOps engineer): Verify the Azure AI Search service is on Standard tier (S1 or higher) with sufficient search units and semantic ranker quota enabled before turning on semantic search. Check current limits in official documentation.

Stage 2: Offline evaluation and quality gating

  • Held-out golden set evaluation (GenAI engineer): Execute automated evaluation across the held-out golden test set (minimum 100 queries) comparing baseline vs candidate ranking settings.
  • Relevance metric gating (RAG Lead): Confirm candidate ranking achieves $\ge$ baseline nDCG@10 and $\ge$ baseline Mean Reciprocal Rank (MRR). Any regression on critical golden queries blocks release.
  • Refusal and out-of-scope verification (GenAI engineer): Confirm out-of-scope and adversarial queries do not retrieve unrelated chunks with elevated confidence scores.
  • Semantic threshold calibration (GenAI engineer): If semantic ranking is used, verify the downstream prompt application enforces a minimum @search.rerankerScore threshold (recommended default $\ge 2.0$ on the 0–4 scale) to discard ungrounded context.

Stage 3: Deployment and live verification

  • Index definition drift check (DevOps engineer): Compare the live index definition against the committed repository JSON using Azure CLI or REST API to confirm zero unapproved schema drift:
curl -sS "${SEARCH_ENDPOINT}/indexes/${INDEX_NAME}?api-version=2024-07-01" \
  -H "Authorization: Bearer ${TOKEN}" | jq '{scoringProfiles, semanticSearch}' > live.json
diff -u git_expected.json live.json

Verify: diff outputs zero lines, confirming live configuration matches Git.

  • Query parameter scoping (GenAI engineer): Inspect application search request templates to confirm semantic ranking is enabled only for full conversational queries and explicitly disabled on autosuggest or filter-only queries (queryType=simple or queryType=full).
  • Search alias verification (DevOps engineer): If ranking change required an index rebuild (e.g. vector dimension or analyser changes), confirm the application queries an index alias, and verify the alias points to the validated index following PB-RAG-09-01.
  • Smoke test execution (GenAI engineer): Run 10 representative live queries across text, vector, and hybrid modes. Confirm HTTP 200 responses and valid citation metadata in answers.

Stage 4: Post-change monitoring and rollback readiness

  • Rollback configuration tagged and ready (DevOps engineer): Verify the previous known-good index definition or application configuration is tagged in Git, with a single-command revert or alias cutback tested.
  • Telemetry alert activation (DevOps engineer): Confirm Azure Monitor alert rules for SearchLatency (> 1.5x baseline) and ThrottledSearchQueriesPercentage (> 1%) are active with on-call routing enabled.
  • Post-deployment watch window (RAG Lead): Monitor live query latency and semantic ranker consumption for 2 hours post-release. Confirm error rates remain below 0.1%.
  • Sign-off record (RAG Lead): Record the final evaluation report link, change ticket number, and dry-run sign-off in the validation record.

5. Known issues and fixes

Symptom Likely cause Fix
Live settings drift from Git Direct manual edits in the Azure portal bypass CI/CD Restrict Search Service Contributor role in Microsoft Entra ID. Automate nightly drift detection in CI and restore from Git on failure.
Offline evaluation passes but users report worse answers The tuning query set was overfitted; held-out evaluation was skipped Expand the golden test set with production query logs. Re-evaluate across distinct query clusters following PB-RAG-08-02.
Search latency spikes by > 200 ms after enabling semantic search Semantic ranking re-ranks 50 documents containing very large text fields Specify exact, concise fields in prioritizedContentFields and prioritizedKeywordsFields. Exclude large raw JSON chunks from semantic extraction.
Semantic ranker cost jumps unexpectedly post go-live Application sends every keystroke, typeahead, or filter query to the semantic ranker Set queryType=simple for autocomplete and typeahead. Enable queryType=semantic exclusively on submitted search requests.
429 ThrottledSearchQueriesPercentage rises post-change S1 tier search unit capacity exhausted by concurrent hybrid + vector queries Add search replicas to scale query throughput, optimize vector filter selectivity, or check current limits in the docs.
Document ranking collapses or produces identical scores Vector query weight set to 0.0 or RRF reciprocal rank constant altered Check query payload: verify vector weight (default 1.0) and confirm both text and vector components return non-empty candidate sets.

6. Monitoring and alerts

The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Source Alert when Action
Nightly index definition drift CI/CD scheduled job Live index configuration differs from Git Roll back unapproved portal changes immediately by reapplying Git index JSON.
SearchLatency (Average) Azure Monitor platform metric Average query latency rises > 1.5x baseline for 10 minutes Check if semantic ranking fields are oversized or add search replicas. Revert ranking config if latency persists.
ThrottledSearchQueriesPercentage Azure Monitor platform metric Throttling exceeds 1% of total queries for 5 minutes Scale out search service replicas or reduce top-k candidate limits.
Semantic ranker query volume Application telemetry / Azure billing Daily query volume exceeds 30% above forecast Audit client application for query loops or unintended semantic calls on autocomplete requests.
Low relevance feedback rate Application telemetry (user thumbs down / no-click) Negative feedback rate exceeds 15% on live queries over 1 hour Initiate PB-RAG-08-03 to diagnose regression; execute rollback if confirmed.

7. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon OpenSearch Service Search Pipelines with Normalization Processor (normalization-processor) and neural sparse / k-NN search OpenSearch supports score normalization techniques (such as Min-Max, L2, or reciprocal rank) within search pipelines. Pipelines are defined in JSON/YAML and updated via the OpenSearch REST API (PUT /_search/pipeline/<pipeline-name>). Track pipeline configurations in Git and test with OpenSearch Benchmark.
PostgreSQL with pgvector Hybrid ranking using SQL Common Table Expressions (CTEs) and Reciprocal Rank Fusion RRF is implemented directly in SQL queries combining full-text search (ts_rank) and vector distance (<-> or <#>). Changes to weights or formulas are deployed via application code or stored procedures. Test migrations using EXPLAIN ANALYZE to guard against query plan regressions.

8. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: An engineer introduces an unapproved scoring profile and alters semantic configuration parameters directly on a sandbox Azure AI Search index. The dry-runner executes the checklist. The drift detection check catches the disparity, the offline evaluation identifies a simulated relevance regression on the held-out golden set, and the automated rollback restores the approved baseline configuration.

Pass criteria:

  • Drift detection script flags live vs Git discrepancies with non-zero exit code.
  • Offline evaluation detects artificial relevance degradation on held-out test set.
  • Rollback procedure restores baseline index configuration and passes all checklist items.

Record

Item Detail
Playbook PB-RAG-08-04 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

9. Sources

Checked against official documentation on 02 November 2026.