Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-3 hours to a diagnosis
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (wrong diagnosis leads to changes that make things worse)

Triage 'the answers feel wrong': retrieval or generation?

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-04). Author: Prasanna · dry-run: Devadharshini Raja · sign-off: Sivaraj. Work items: RAG-02-T09, RAG-02-T10, RAG-02-T11, RAG-02-T12.

1. Summary

This runbook sorts "the answers feel wrong" reports into four classes: retrieval miss, ranking issue, filter issue and generation fault. You test each failing question against the index, then against the prompt, before anyone changes either. The outcome is a classified list of cases, a root cause, a severity and a handover to the playbook that fixes it. It diagnoses only; it makes no changes.

Risk: Medium (wrong diagnosis leads to changes that make things worse). Typical duration: 1-3 hours to a diagnosis.

2. When to use and when not to

Use it when: users or the business report worse answers, or the nightly golden-set evaluation drops. Use it as the first response, before anyone changes prompts or indexes.

Do not use it when:

  • The problem is a stale or missing document after an indexer failure: use PB-RAG-05-02.
  • The problem is a permission change or revocation that is not yet applied: use PB-RAG-04-03.
  • The answer cites the right passage but invents claims or a wrong citation: start at PB-RAG-06-04.
  • You need to build or extend the golden set or run a planned evaluation: use PB-RAG-02-01 or PB-RAG-02-02.

3. What you must know first

  • Working definition: if the right passage is not in the top-k, it is a retrieval fault. If it is retrieved and the answer is still wrong, it is a generation fault. (Runbook rule, not from a source.)
  • A hybrid query is one request with search and vectorQueries. The two queries run in parallel and are merged with RRF, which scores each result as 1/(rank + k). The RRF constant k is separate from the vector query k. (Hybrid search overview, RRF ranking)
  • A text query is vectorized by the service when kind is text and the vector field has a vectorizer. A vector-only query always returns k results, even poor matches, so read its rank next to the keyword and hybrid ranks. (Create a vector query)
  • A filter can run before or after vector search (vectorFilterMode: preFilter, postFilter or strictPostFilter, which is preview). preFilter is the default for indexes created after about 15 October 2023. postFilter can return too few results when the filter is selective. A security filter is an ordinary filter on a field of user or group IDs, so wrong IDs hide documents. (Vector query filters, Security filter pattern)
  • The semantic ranker runs after RRF and reranks only the top 50 results it receives. It cannot find a passage that was not in them. The debug request parameter (vector, semantic or all) returns subscores that explain a ranking. (Semantic ranking, RRF ranking)
  • RetrievalEvaluator, DocumentRetrievalEvaluator (needs ground_truth) and GroundednessEvaluator score the two halves separately on a JSONL dataset. (Local evaluation)

4. Prerequisites and access

Item Detail
Failing cases 10-20 questions, each with the time, the user and the answer that was given. Take them from support, users or the nightly evaluation.
Application logs For each question the logs must hold the retrieved chunk IDs, the filter applied for the user and the prompt sent to the model. If they do not, ask the application owner before you start.
Search access Search Index Data Reader on the index. It is read-only and can query. Use Microsoft Entra ID, not admin keys. (Azure roles)
Tools Python 3 with requests and azure-identity. Sign in with az login or another source that DefaultAzureCredential can use.
Scripts snippets/probe_query.py (steps 4 to 6) and snippets/replay_prompt.py (step 7) from this pack.
Model access Cognitive Services OpenAI User on the Azure OpenAI resource, to replay prompts in step 7. Use Microsoft Entra ID, not a key. (Entra ID for Azure OpenAI)
What to query The alias or index that the application queries, not the index behind an alias. An alias maps a name to one index. (Index alias)
Vector field Normally the vector field has a vectorizer, and the probe sends the question as text. If an index has none, use the embedding fallback: set EMBEDDING_ENDPOINT, pass --embedding-deployment and --embedding-api-version, and give the identity the Cognitive Services OpenAI User role on the Azure OpenAI resource. Use the same embedding model that built the index, or the ranks are meaningless. Before you use it, check the token audience and the api-version for your Azure OpenAI resource in the documentation. (Entra ID for Azure OpenAI)
Baseline The latest golden set results from PB-RAG-02-01 and the tolerance from PB-RAG-02-04.
Change log Releases, re-index runs, alias swaps, and model, prompt, data and permission changes since the last good evaluation.
Triage record A copy of snippets/rag-triage-record.md.
Test user A test user or group for the filter check. Never use a real user's access to test.

5. Steps

Placeholders: ${INCIDENT_ID}, ${SEARCH_SERVICE}, ${INDEX_OR_ALIAS}, ${VECTOR_FIELD_NAME}, ${ID_FIELD_NAME}, ${QUESTION}, ${EXPECTED_ID}, ${USER_FILTER} and ${SEMANTIC_CONFIG}. Never paste keys, tokens or real names into tickets or chat.

This runbook is read-only. Steps 4 to 6 send search queries. Steps 7 and 9 call a model. Nothing here changes the index, the application or anyone's access.

Step 1. Copy the triage record and add one case for each failing question.

cp snippets/rag-triage-record.md ${INCIDENT_ID}-triage.md

Verify: the file exists. Each case has the time, the user, the question, the answer given and the logged chunk IDs.

Step 2. Check the change log for the window of the first failure. Write down every release, re-index run, alias swap, model change, prompt change, data load and permission change between the last good evaluation and the first failure.

Verify: the window is written in the record, with each change that falls in it, or with "no changes found".

Step 3. Find the expected passage for each question by hand. Open the source document and note its ID as it appears in ${ID_FIELD_NAME} in the index.

Verify: each case has an expected document ID, and a second person agrees with it.

Step 4. Run the probe for each case, with the filter that the user's query applied. Set the environment once, then run the probe.

export SEARCH_ENDPOINT="https://${SEARCH_SERVICE}.search.windows.net"
export SEARCH_INDEX="${INDEX_OR_ALIAS}"
export VECTOR_FIELD="${VECTOR_FIELD_NAME}"
export ID_FIELD="${ID_FIELD_NAME}"

python snippets/probe_query.py "${QUESTION}" "${EXPECTED_ID}" --filter "${USER_FILTER}"

The probe prints the rank of the expected document in keyword-only, vector-only and hybrid mode. Because you gave a filter, it also runs hybrid without the filter. Then it suggests a class. --top is how many results the application passes on (default 10), and --depth is how far down the probe looks (default 50). If the application sets vectorFilterMode, pass the same value with --filter-mode. If the index has no vectorizer, add --embedding-deployment ${EMBEDDING_DEPLOYMENT} --embedding-api-version ${EMBEDDING_API_VERSION} (see the prerequisites). Each case sends three to five search queries. Check the current limits in the docs before you probe many cases at once.

Verify: the output shows a rank, or -, for every mode, and a suggested class. The hybrid rank agrees with the position of the expected chunk in the application's logged chunk IDs. If it does not, the probe is not querying what the application queries (a different alias, filter or filter mode). Fix that before you go on.

Step 5. Confirm the class of each case. The suggested class is a guide. Use this table.

Probe result Class Next
Hybrid rank within the top 10 Not a retrieval fault Go to step 7.
Found only without the filter Filter issue Confirm with a test user (see the warning). Hand over in step 10.
Hybrid finds it, but below the top 10 Ranking issue Go to step 6.
Only keyword finds it, or only vector finds it Ranking issue Go to step 6. Keyword only: check that vectors are filled and which embedding model was used. Vector only: check the analyser and field choices.
Not found in any mode, with or without the filter Retrieval miss Look the document up in the index by key, or with a filter on ${ID_FIELD_NAME} if it is filterable. If it is missing, hand over in step 10. If it is there, check that its vector field is filled.

Do not widen access to test a filter

Test with a test user or group, or in the sandbox. Never add a real user to a group or relax a filter on a live index to see whether a document appears. If the index uses document-level permissions, an elevated read (preview) can bypass them to investigate wrong results. It needs the Search Index Data Contributor role, so ask the security owner first. (Azure roles)

Verify: each case has exactly one class that you agree with.

Step 6. For ranking issues, rerun the probe with semantic ranking and debug output.

Semantic ranking is billed per query

Charges apply when a request has queryType=semantic and a search string. Check the current price before approval, and run only the ranking cases. (Semantic ranking)

python snippets/probe_query.py "${QUESTION}" "${EXPECTED_ID}" --filter "${USER_FILTER}" --semantic "${SEMANTIC_CONFIG}" --debug

Verify: the output has a hybrid rank and a hybrid + semantic rank, and the debug hit shows the subscores returned for the expected document. The semantic ranker reranks only the top 50 results it receives. A document outside the first 50 in hybrid cannot be lifted by it.

Step 7. For generation faults, replay the logged prompt with the same chunks. Use snippets/replay_prompt.py. It sends the chat messages that the application logged, with the retrieved chunks already in them, to the model deployment and prints the answer. Run it three times: as logged, with the previous prompt, and with the previous model version.

This step calls the model and spends tokens

Replay only the failing cases. Check the current price of the model deployment before you replay many cases. A replay is a test. Do not deploy the old prompt or model.

export MODEL_ENDPOINT="https://${OPENAI_RESOURCE}.openai.azure.com"
export MODEL_DEPLOYMENT="${CURRENT_DEPLOYMENT}"

# 1. The prompt as it was logged, with the current deployment
python snippets/replay_prompt.py --messages ${LOGGED_MESSAGES_FILE} --expect "${EXPECTED_TEXT}"
# 2. The previous prompt: the logged messages from before the change
python snippets/replay_prompt.py --messages ${PREVIOUS_MESSAGES_FILE} --expect "${EXPECTED_TEXT}"
# 3. The previous model version, with the logged messages
python snippets/replay_prompt.py --messages ${LOGGED_MESSAGES_FILE} --deployment ${PREVIOUS_DEPLOYMENT} --expect "${EXPECTED_TEXT}"

${EXPECTED_TEXT} is a phrase from the expected passage. The script reports whether the answer contains it, as a quick aid. You still read the answer.

Verify: with the same chunks the answer still contradicts the passage. If the previous prompt or model gives a correct answer, the change is the cause. This is a generation fault.

Step 8. Record the class of every case in the triage record.

Verify: every case has exactly one class and one line of evidence. Copy the probe output into the evidence.

Step 9. Run the golden set to size the problem. Run it as in PB-RAG-02-01 and PB-RAG-02-02.

A golden-set run calls the judge model and spends tokens

Check the current price before you run it. Use a sampled subset for a first look.

Verify: for retrieval classes, recall@10 and nDCG@10 are compared with the baseline, using the tolerance in PB-RAG-02-04. For generation faults, groundedness and relevance scores are compared the same way. The result is a severity: isolated or systemic.

Step 10. Hand over to the fix playbook for each class. This runbook does not make the change.

Class and likely cause Hand over to
Retrieval miss: document not indexed, or an indexer failure PB-RAG-05-02
Retrieval miss or ranking issue after a re-index run or an alias swap PB-RAG-09-01
Filter issue: wrong IDs or a permission change PB-RAG-04-03
Ranking issue: embedding, analyser or semantic configuration change The pack owner. Run the gate in PB-RAG-02-04 before the fix is released.
Generation fault: prompt or model change PB-RAG-06-04 and PB-RAG-07-01

Verify: the incident owner has the class, the root cause, the severity and the playbook to follow.

Step 11. Write the incident note with the root cause and the new golden questions to add.

Verify: the note is saved, and the new questions are passed to PB-RAG-02-01.

Snippets used in the steps

The steps use three files from the snippets/ folder of this pack:

The probe prints a table like this one. The values are an example, not a measured result.

Question:    Q?
Expected id: DOC-42
Mode                Rank (within the first 50)
keyword             -
vector              3
hybrid              3
Suggested class: in the top 10: retrieval looks fine, check generation (step 7)

6. Monitoring and alerts

Signal Alert when Action
Nightly golden set recall@10 Below baseline by more than the tolerance in PB-RAG-02-04 (team default 0.02 per metric) Start this runbook
User thumbs-down rate At or above 2x the trailing 7-day average (team default) Start this runbook and sample the flagged questions
Share of answers that say 'no information found' At or above 50% over the trailing 7-day average (team default) Check indexer status (PB-RAG-05-02) and filters first

7. Rollback

This runbook is read-only. It queries the index, replays prompts and runs the golden set, and it changes no index, alias, prompt, model, filter or permission. There is nothing to undo.

  • Replaying with the previous prompt or model (step 7) is a test run only. Do not deploy it.
  • To undo the cause you found, use the playbook named in step 10, for example PB-RAG-09-01 for an alias swap.

Do not widen access to test a filter

Test with a test user or group, or in the sandbox. Nothing in this runbook needs more access than Search Index Data Reader.

8. Known issues and fixes

Symptom Likely cause Fix
Expected document not returned at all Document not indexed, or excluded by a security or metadata filter Check indexer status and the filter applied for that user
Passage found only by keyword search Embedding model changed or vectors missing for some documents Check vector field population and embedding deployment
Correct passage retrieved but answer contradicts it Prompt change or model version change Replay with the previous prompt and model; roll back if it fixes
Drop started right after an alias swap New index built with different settings or incomplete load Repoint the alias to the previous index and investigate
Passage is in the top 10 without semantic ranking but drops with it The semantic configuration fields do not carry the passage text Check the semantic configuration fields and their order
Passage is found only at rank 51 or lower The semantic ranker reranks only the top 50 of the initial result Raise the vector k to 50 and review the field and analyser choices
The probe rank differs from the application's logged chunks The probe queries a different index, alias, filter or filter mode Use the application's alias and filter, and pass --filter-mode. Do not go on until the two agree
The probe fails with HTTP 403 The signed-in identity cannot query the index Assign Search Index Data Reader on the index. Do not use an admin key
The probe fails on the vector query The vector field has no vectorizer, so the service cannot vectorize the text Check vectorizers in the index definition (see Create a vector query), or use the embedding fallback (--embedding-deployment)

9. AWS and open-source equivalents

On AWS, call the Amazon Bedrock Retrieve API to see what a knowledge base returns for a question without generating an answer (Retrieve). RetrieveAndGenerate queries the knowledge base and generates the response in one call (RetrieveAndGenerate). Compare the two to separate retrieval from generation. A document that Retrieve does not return is a retrieval fault. A document that it returns, with a wrong answer from RetrieveAndGenerate, is a generation fault.

On open-source stacks, trace each request with Langfuse, which records the retrieval steps and the model calls with their inputs and outputs (Langfuse observability). Log the retrieved IDs in the trace and apply the same split.

10. Validation record

Pending. Planned dry-run scenario: A second engineer receives a sandbox with two seeded faults (a filter that hides a document, and a prompt that ignores context). Pass if both are classified correctly within 2 hours using only this runbook.

11. Sources

Checked on 29 October 2026. Each source is followed by what it backs in this runbook.