Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 2-8 hours to diagnose; fix time varies
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (a hasty ranking fix can improve one query type and break others for all users)

Diagnose and fix a relevance regression in production

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-16). Author: Deeparani · dry-run: Prasanna · sign-off: Yogeshwaran. Work items: RAG-08-T09, RAG-08-T10, RAG-08-T11, RAG-08-T12.

1. Summary

Relevance regressions usually follow a change: a reindex, new data, a new embedding model, a ranking setting, a prompt or a model. Users see worse answers, more "not found" replies or wrong citations. A hasty fix can help one query type and break others. Find the cause, fix it on a copy, and prove it with the golden set before it reaches users. The outcome is a golden-set score back at baseline, with the failing examples added to the golden set.

The order is always the same:

  1. Confirm the drop and list what changed in the last 7 days.
  2. Classify each failing example: content (chunk missing or stale), ranking (chunk ranks too low) or generation (chunk retrieved, answer still wrong).
  3. Fix on a sandbox copy, run the full golden set, then release or roll back.

Steps 1-11 only read production or change the sandbox copy. The one risky step is the release in step 12.

All calls use the stable REST API 2026-04-01. Reset docs and hybridSearch.maxTextRecallSize are preview (2026-08-01-preview); the runbook only points to them. Facts checked on 3 November 2026.

2. When to use and when not to

Use this runbook when:

  • The nightly golden-set score drops, or an alert in section 7 fires.
  • Users report worse answers, more "not found" replies or wrong citations.
  • Any of these follow a release, reindex, data change, or prompt or model change.

Do not use it when:

  • One user reports one bad answer and the golden set is stable. Use PB-RAG-02-03 instead.
  • You are tuning weights or a re-ranker with no regression. Use PB-RAG-08-02 instead.
  • The indexer is failing or throttled. Use PB-RAG-05-02 first, then come back if quality is still low.
  • Citations point to the wrong page while the right chunk was retrieved. Use PB-RAG-06-04 instead.

3. What you must know first

  1. Retrieval failure or generation failure. If the right chunk is not in the top k, retrieval failed. If it is in the top k and the answer is still wrong, generation failed. Fix only the failing half.
  2. Check what changed first. Most regressions follow a change to the index definition, data, embeddings, ranking settings, prompt or model. The change list (step 2) is the fastest lead.
  3. Debug subscores show which ranker moved a result. Set debug to vector, semantic or all in the search request. @search.documentDebugInfo then shows the keyword (text.searchScore) and vector (vectors) parts of the score (hybrid ranking).
  4. Semantic ranker sees only the top 50. It re-ranks the top 50 results and scores them 0 to 4 in @search.rerankerScore. A chunk outside the first 50 can never be lifted by it. An empty or * search string means no semantic ranking (semantic query).
  5. Index statistics lag. Get Index Statistics results "are not guaranteed to always be up-to-date". Use a lookup by key and $count to check content.
  6. Never tune ranking on production. Reproduce and fix on a sandbox copy, then release through the normal change process.

4. Prerequisites and access

Item Detail
Roles Search Index Data Reader on production and the sandbox copy (query and lookup only). Search Service Contributor on the sandbox to change it. Use Microsoft Entra ID, not admin keys.
Logs Search resource logs sent to Log Analytics (AzureDiagnostics). App telemetry with the query, the request body, the retrieved ids and the prompt version.
Golden set The golden set and the last good nightly results (PB-RAG-02-01).
Sandbox A copy of the production index on a sandbox service, built from the definitions in Git.
Change history Git history for index, indexer, skillset, prompts and app config; Azure activity log.
Tools Azure CLI, curl, jq, Python 3.10+ with requests and azure-identity.
Change approval Change ticket for any production change in stage 4, with the rollback owner. 
export SEARCH_ENDPOINT="https://${SEARCH_SERVICE}.search.windows.net"
export API_VERSION="2026-04-01"
export TOKEN=$(az account get-access-token --scope https://search.azure.com/.default --query accessToken -o tsv)

5. Decide your approach

Steps 1-4 are always the same. Step 4 tells you which branch to follow.

What step 4 shows Branch Go to
Expected chunk is in the top k for the app's request Generation failure Step 10
Expected chunk is missing, or its content is old Content failure Step 5
Expected chunk exists but ranks below the top k Ranking failure Step 7
Results are right in a manual query, wrong in the app Request mismatch Step 8 (replay the app's exact body)

6. Steps

Placeholders: ${INDEX} (production alias or index), ${SANDBOX_INDEX} (copy), ${KEY} (chunk key), ${KEY_FIELD} (for example id). Never paste tokens into tickets or chat.

Stage 1: Confirm and scope

Step 1. Confirm the regression. Compare the nightly golden-set score with its 7-day trend. Collect three user examples: the query, the expected document, the time. Run zero-results.kql in Log Analytics.

Verify: the golden-set drop is larger than normal noise, and you have three examples. Note the hour the zero-result rate or the score changed: it dates the cause.

Step 2. List every change in the last 7 days.

git log --since="7 days ago" --oneline -- ${DEFS_PATH} ${PROMPTS_PATH} ${APP_CONFIG_PATH}
az monitor activity-log list --resource-group ${RESOURCE_GROUP} --offset 7d \
  --query "[].{time:eventTimestamp, op:operationName.localizedValue, by:caller}" -o table

Verify: you have a list of changes with times, and at least one matches the hour from step 1. If none matches, check data changes (indexer runs, source uploads) in step 5.

Step 3. Get the app's request and retrieved ids for each example. Take them from app telemetry: the request body, the retrieved ids and the prompt and model version. Save each as one line of requests.jsonl (format in replay_queries.py).

Verify: each line has query_id, body and expected_ids.

Step 4. Classify the failure. For each example, check if the expected chunk is in the retrieved ids. Pick the branch in section 5.

Verify: every example has one branch. If examples split across branches, work the ranking or content branch first.

Stage 2: Check content

Step 5. Look up the expected chunk by key.

curl -sS "${SEARCH_ENDPOINT}/indexes/${INDEX}/docs/${KEY}?api-version=${API_VERSION}" \
  -H "Authorization: Bearer ${TOKEN}" | jq '.'

URL-encode ${KEY} if it contains special characters.

Verify: HTTP 200 and the content matches the current source. HTTP 404 means the chunk is missing.

Step 6. Count the documents. Do not use index statistics; they lag.

curl -sS "${SEARCH_ENDPOINT}/indexes/${INDEX}/docs/\$count?api-version=${API_VERSION}" \
  -H "Authorization: Bearer ${TOKEN}"

Verify: the count matches the last good count and the source. If chunks are missing or stale, follow PB-RAG-05-02 (indexer failures) or PB-RAG-09-01 (rebuild), then go to step 11.

Stage 3: Find where ranking moved

Step 7. Replay one example with debug subscores. Use debug-query.http with the app's body, top set to 50 and debug set to all. Run it on the production alias (read only) and on the sandbox copy.

Verify: you know the expected chunk's rank and which part moved it:

What you see Likely cause
Low text.searchScore, normal vector score Analyser, field or searchFields change
Low vector score, normal text score Embedding model, vectorizer or chunking change
In the top 50 but low @search.rerankerScore Semantic configuration change (title, content or keyword fields)
Not in the top 50 at all Filter, top, k or index content change

Step 8. Replay the logged app requests on the sandbox copy.

python snippets/ops/replay_queries.py requests.jsonl --endpoint "${SEARCH_ENDPOINT}" \
  --index ${SANDBOX_INDEX} --key ${KEY_FIELD} --baseline baseline.json --out results.json

Verify: the script lists each expected chunk outside the top 10 and every DROPPED chunk. The sandbox shows the same drops as production. If not, the sandbox copy differs from production: rebuild it from Git.

Step 9. Compare ranking settings with the last good version in Git.

curl -sS "${SEARCH_ENDPOINT}/indexes/${INDEX}?api-version=${API_VERSION}" -H "Authorization: Bearer ${TOKEN}" \
  | jq -S '{fields, scoringProfiles, semantic, vectorSearch}' > live.json
git show ${LAST_GOOD_COMMIT}:${DEFS_PATH}/index-${INDEX}.json \
  | jq -S '{fields, scoringProfiles, semantic, vectorSearch}' > good.json
diff good.json live.json

Also compare the app's query settings: queryType, semanticConfiguration, vector weight, k, top and filters.

Verify: you have one candidate cause that explains the subscores from step 7. Go to step 11.

Stage 4: Fix and release

Step 10. Check prompt and model (generation failure only). Compare the prompt and model version in telemetry with the last good release. Follow PB-RAG-07-01 to test and roll back a prompt change.

Verify: the answer is correct when the last good prompt and model run on the same retrieved chunks.

Step 11. Reproduce the fix on the sandbox copy. Apply one change at a time: revert the setting, or rebuild with the previous definition (PB-RAG-09-01). Run the full golden set, not only the failing examples.

Verify: the golden-set score is back to baseline, and no query type or corpus segment is worse than before.

Production change: affects every user of the index

A ranking or index change applies to all queries at once. Release it only with a change ticket and a rollback owner. Keep the previous definition and the previous index (behind the alias) until the watch window ends. Rollback: section 8.

Step 12. Release the fix, or roll back the last change. Release through the normal change process. If no fix is ready within the agreed time, roll back the change from step 2 instead.

Verify: run replay_queries.py against ${INDEX}; the failing examples are in the top 10. The next nightly golden-set run is at baseline.

Step 13. Add the failing examples to the golden set. Add each user example with its expected ids (PB-RAG-02-01).

Verify: the next nightly run includes the new examples and they pass.

7. Monitoring and alerts

Thresholds are team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Source Alert when Action
Nightly golden-set nDCG@10 Evaluation pipeline Drop of more than 0.03 versus the 7-day average Open an incident and start this runbook at step 1.
Share of answers that are refusals or "not found" App telemetry 1.5 times baseline over 1 hour Start this runbook at step 1.
Zero-result queries (Query.Search with Documents_d == 0) AzureDiagnostics 50% above baseline over 1 hour Check filters, the alias target and the document count (step 6).
SearchLatency Platform metric 1.5 times baseline for 10 minutes after a ranking release Roll back the release (section 8).

The zero-result and duration query is in zero-results.kql. Use it as the log alert query.

8. Rollback

Stages 1-3 only read production; they need no rollback.

Situation Rollback Time
Sandbox change made things worse Rebuild the sandbox copy from the definition in Git. Indexing time
Fix released by changing app query settings Redeploy the previous app config. Deployment time
Fix released as a new index behind the alias Repoint the alias to the previous index (PB-RAG-09-01, section 8). Under 5 minutes
Prompt or model change released Roll back with PB-RAG-07-01. Deployment time

Rollback trigger (any one): the golden-set score falls below the pre-fix value; any query segment drops in the next nightly run; an alert in section 7 fires within the watch window.

9. Known issues and fixes

Symptom Likely cause Fix
Expected document missing after a reindex Indexer item failures, or the deletion policy removed it Check indexer status errors and the source. Reprocess with reset docs (preview, 2026-08-01-preview) or push the document.
Expected chunk exists but ranks below the top 50 An embedding, chunking or analyser change altered matching Compare subscores before and after (step 7). Revert the change or rebuild with the previous setting.
Results look right in Search Explorer but wrong in the app The app uses different filters, alias, query type or select list Log the exact request body from the app and replay it (step 8).
No @search.rerankerScore or captions in results Query is not semantic, or the search string is empty or * Check queryType and semanticConfiguration. Semantic ranking needs a non-empty search string.
Hybrid query misses keyword matches that full-text search finds Only a limited set of text results goes into fusion Check hybridSearch.maxTextRecallSize (preview) in the sandbox only.
Sandbox replay does not show the production drop The sandbox copy differs from production Rebuild the sandbox from the definitions in Git and the same data snapshot.
Golden set improves but one query type gets worse A ranking fix tuned for one segment Report the score per segment. Do not release until no segment is worse.

10. AWS and open-source equivalents

The diagnosis order is the same everywhere: confirm, list changes, check content, check ranking, then generation.

Platform Equivalent Key differences
Amazon OpenSearch Service and OpenSearch Hybrid search pipeline with a normalisation processor; explain for per-query scores Check the search pipeline definition as well as the index mapping.
Amazon Bedrock Knowledge Bases Call the Retrieve API directly to see which chunks came back Ranking settings are fewer; most regressions are data or chunking changes.
Open source (Elasticsearch, Qdrant, Weaviate) Explain or score details per query, plus an RRF or fusion setting Check the fusion method and the re-ranker version in your own config.

11. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: a second engineer breaks a sandbox index on purpose. They rebuild it with a different analyser and drop ten documents, then follow the runbook from the alert.

Pass criteria: both causes are found within 2 hours, and the golden-set score returns to baseline after the fix.

Record

Item Detail
Playbook PB-RAG-08-03 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

12. Sources

Checked on 3 November 2026.

  • PB-RAG-02-01: build and run the golden set used in steps 1, 11 and 13.
  • PB-RAG-02-03: triage a single bad answer.
  • PB-RAG-08-02: tune weights and re-ranking once the regression is fixed.
  • PB-RAG-05-02 and PB-RAG-09-01: fix missing content and rebuild the index.
  • PB-RAG-07-01: test and roll back a prompt change.
  • PB-RAG-06-04: citations are wrong although the right chunk was retrieved.