Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-4 hours to root cause
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (in regulated settings a wrong cited answer can have legal impact)

Investigate an ungrounded answer or wrong citation incident

Draft: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is dry-run by someone other than the author and signed off (planned 2026-11-12). Author: Deeparani · dry-run: Gayathri Ramachandran · sign-off: Prasanna. Work items: RAG-06-T13, RAG-06-T14, RAG-06-T15, RAG-06-T16.

1. Summary

A wrong answer that carries a citation does more harm than no answer, because users trust the citation. When one is reported, contain the harm first and find the cause second. The citation verifier is run on the logged trace and sorts the fault into one of three classes: generation, renderer or stale index. Each class has its own fix path. The steps below take the incident from first report to root cause, fix and closure.

Facts checked on 28 Oct 2026. No preview features are used.

2. When to use and when not to

Use this runbook when:

  • A user, reviewer or monitor reports an answer that its cited sources do not support.
  • A citation opens the wrong document or the wrong page.
  • An alert in section 7 fires.

Do not use it when:

3. What you must know first

  1. Three fault classes, three fix paths. Generation: the model cited a chunk it was not given, or quoted text that is not in it. Renderer: the answer is right, but the link or page shown is wrong. Stale index: the source changed or was deleted after indexing.
  2. Structured outputs fix the shape, not the facts. They make the model follow your JSON Schema. They do not check that a cited chunk id was retrieved, so verify citations in code. They are not supported with "On Your Data" (bring your own data). (Structured outputs)
  3. Indexers detect changes, not deletions. Blob indexers pick up changed content from timestamps. Deletions are removed only with a soft-delete detection policy set before the first run; without it, deleted documents stay searchable. A restored blob may not be reindexed until its LastModified timestamp changes. (Changed and deleted blobs)
  4. Re-chunking changes chunk ids. Old citation links can point to chunks that no longer exist. See PB-RAG-01-03.
  5. Contain before you diagnose. When users are at risk, stop the harm first. Root cause can wait an hour; a wrong cited answer in a regulated setting cannot.

4. Prerequisites and access

Item Detail
Report details Conversation id, trace id or timestamp of the reported answer
Traces Read access to logged traces: prompt, retrieved chunks, model output and rendered links, under one trace ID (PB-RAG-06-01, step 6)
Search access Search Index Data Reader on the search service, and read access to indexer status
Feature flags App Configuration Data Owner on the store that holds ${SAFE_MODE_FLAG}, if you use Azure App Configuration
Change log Release history for prompts, models, index and renderer for the last 7 days
Tools Azure CLI, curl, jq, Python 3, snippets/verify_citations.py
Incident record Copy of snippets/grounding-incident.md in the incident ticket 
export SEARCH_ENDPOINT="https://${SEARCH_SERVICE}.search.windows.net"
export API_VERSION="2026-04-01"
TOKEN=$(az account get-access-token --resource https://search.azure.com --query accessToken -o tsv)

5. Decide your approach

Run steps 1 to 7 every time. The verifier result (step 5) and the replay (step 8) pick the fix path.

Finding Fault class Fix path
[generation] ... was never retrieved (invented id) Generation Turn on code-level citation verification (PB-RAG-06-01); drop unverified claims
[generation] quote not found Generation Same as above. If it started after a release, roll back the prompt or model (section 8)
[renderer] rendered page ... or source_url mismatch Renderer Fix the chunk-to-link mapping or page metadata; roll back the renderer release if recent
[stale_index] content ... has changed Stale index Reindex the changed source (PB-RAG-05-02) and check the refresh schedule
[stale_index] no longer in the index Stale index Check deletion detection (PB-RAG-05-03) or a recent re-chunk (PB-RAG-01-03)
No faults, but the answer is still wrong Generation (unsupported claim with a valid citation) Replay (step 8); score with the groundedness gate (PB-RAG-06-03)

6. Steps

Placeholders: ${SEARCH_SERVICE}, ${INDEX_NAME}, ${INDEXER_NAME}, ${CONTENT_FIELD} (the chunk text field), ${APPCONFIG_NAME}, ${SAFE_MODE_FLAG}, ${TRACE_ID}.

Stage 1: Contain

Step 1. Decide whether the fault is systemic. Search the last 24 hours of traces and user reports for the same topic, document or release. One answer is an isolated case; several answers on the same topic or since the same release are systemic.

Verify: the incident note records "isolated" or "systemic", with the evidence.

Changes what users see

Safe mode turns off generated answers for every user it targets. Agree it with the service owner first. Roll back by disabling the flag (section 8).

Step 2. If systemic, switch on safe mode. Safe mode shows the retrieved passages with links, without a generated answer. Use your app's flag; with Azure App Configuration:

az appconfig feature enable --name "${APPCONFIG_NAME}" --feature "${SAFE_MODE_FLAG}" \
  --label "${ENV_LABEL}" --auth-mode login --yes

Verify: az appconfig feature show --name "${APPCONFIG_NAME}" --feature "${SAFE_MODE_FLAG}" --label "${ENV_LABEL}" --auth-mode login shows "state": "on", and a test question in production returns passages only.

Stage 2: Collect evidence

Step 3. Export the trace for the reported answer from your trace store to trace.json, in the schema shown at the top of verify_citations.py. Use page_number for page fields, as in the renderer contract from PB-RAG-06-01.

Verify: jq '.retrieved | length, (.answer.claims | length)' trace.json prints two numbers greater than 0.

Step 4. List changes in the 7 days before the report. Note every prompt, model, index, indexer or renderer release, with its time.

git log --since="7 days ago" --oneline -- "${PROMPTS_PATH}" "${CONFIG_PATH}" "${RENDERER_PATH}"

Verify: the incident note lists each change, or says "none".

Stage 3: Diagnose

Step 5. Run the citation verifier.

python snippets/verify_citations.py trace.json

Verify: exit code 0 prints all citations verified. Exit code 1 prints one [generation] or [renderer] line per fault. Copy the lines into the incident note.

Step 6. Check whether the cited chunks have changed in the index. Build current.json with Documents - Get, then re-run the verifier with it.

echo '{}' > current.json
for id in $(jq -r '.retrieved[].chunk_id' trace.json); do
  code=$(curl -sS -o doc.json -w '%{http_code}' -H "Authorization: Bearer ${TOKEN}" \
    "${SEARCH_ENDPOINT}/indexes('${INDEX_NAME}')/docs('${id}')?api-version=${API_VERSION}&\$select=${CONTENT_FIELD}")
  case "$code" in
    200) val=$(jq ".${CONTENT_FIELD}" doc.json) ;;
    404) val=null ;;
    *) echo "unexpected HTTP ${code} for ${id}"; cat doc.json; break ;;
  esac
  jq --arg k "$id" --argjson v "$val" '.[$k] = $v' current.json > tmp.json && mv tmp.json current.json
done
python snippets/verify_citations.py trace.json --current current.json

URL-encode chunk ids that contain special characters.

Verify: current.json has one entry per retrieved chunk. Any [stale_index] line means the index content differs from what the model saw.

Step 7. If a chunk is stale, check when the indexer last ran.

curl -sS -H "Authorization: Bearer ${TOKEN}" \
  "${SEARCH_ENDPOINT}/indexers('${INDEXER_NAME}')/search.status?api-version=${API_VERSION}" \
  | jq '{status, last: .lastResult | {status, endTime, itemsFailed}}'

Verify: compare lastResult.endTime with the source document's last change. If the source changed after the last successful run, the index is behind (PB-RAG-05-02). If the source was deleted and the chunk is still there, deletion detection is missing (PB-RAG-05-03).

Step 8. Replay the question with the previous prompt or model version in a non-production environment, using the same retrieved chunks from trace.json.

Verify: if the fault disappears with the previous version, the release caused it. Record "fault gone" or "fault remains" in the incident note.

Stage 4: Fix

Production release

A prompt, model or renderer rollback changes behaviour for all users. Use the normal release process, and keep the rolled-back version ready to redeploy (section 8).

Step 9. Apply the fix path from section 5. Roll back the faulty release if step 8 points to it; otherwise fix forward through a reviewed PR.

Verify: re-run step 5 (and step 6 for stale faults) on a fresh trace of the same question. Expected: all citations verified.

Step 10. Switch off safe mode if you switched it on in step 2.

az appconfig feature disable --name "${APPCONFIG_NAME}" --feature "${SAFE_MODE_FLAG}" \
  --label "${ENV_LABEL}" --auth-mode login --yes

Verify: az appconfig feature show ... shows "state": "off", and the test question returns a generated, cited answer.

Stage 5: Close

Step 11. Add the question to the golden set with its correct answer and source, and run the groundedness gate (PB-RAG-06-03).

Verify: the gate passes, and the new case id is in the incident note.

Step 12. Complete the incident note from snippets/grounding-incident.md. Inform the client if the contract requires it.

Verify: every field is filled, the root cause is one paragraph, and each follow-up action has an owner role and a due date.

7. Monitoring and alerts

The thresholds below are team defaults, not platform recommendations. Tune them against two weeks of baseline data.

Signal Source Alert when (team default) Action
Citation verification failures (CitationVerificationFailureRate, from PB-RAG-06-01 step 5) App telemetry Above 2% of claims over 1 hour Start this runbook at step 1
User reports tagged "wrong source" Support tool Any new report Start this runbook within 1 business hour
ungroundedDetected rate (if Content Safety groundedness detection, preview, is used) Content Safety response Above 2 times the 7-day baseline for 1 hour Start this runbook and check releases in step 4
Indexer lastResult.status Indexer status API transientFailure on 2 consecutive runs Use PB-RAG-05-02; stale answers will follow
Rendered page or link mismatch ([renderer] faults) App telemetry Any, after a renderer release Roll back the renderer (section 8)

8. Rollback

Undo each action in this runbook as follows. "Newer" means the version you rolled back from.

Action taken How to undo it Time
Safe mode switched on (step 2) az appconfig feature disable ... (step 10) Minutes
Prompt release rolled back (step 9) Redeploy the newer prompt version from source control One release cycle
Model deployment rolled back (step 9) Point the app back at the newer deployment name One release cycle
Renderer release rolled back (step 9) Redeploy the newer renderer build One release cycle
Golden set case added (step 11) Remove the case in a PR Minutes
Reindex started for stale content No rollback: the reindex brings the index up to date with the source Indexing time

Rollback trigger (any one): the fix makes the verifier fail on a fresh trace; the groundedness gate fails; user reports rise after the fix.

9. Known issues and fixes

Symptom Likely cause Fix
[generation] ... was never retrieved (invented id) Model invented the chunk id; no code-level check Verify citations in code before rendering (PB-RAG-06-01)
[generation] quote not found but the meaning is right Model paraphrased instead of quoting Tighten the "quote exact text" instruction; drop claims that fail the check
Quote is correct but the link opens the wrong page Page metadata lost in chunking, or renderer page offset (for example 0-based vs 1-based) Fix metadata propagation or the renderer, and add a renderer test
[stale_index] content ... has changed Index not refreshed after the source changed Run the indexer (PB-RAG-05-02); check the schedule
Deleted document still cited No deletion detection policy, or it was added after the first run Set the soft-delete policy; rebuild the index if it was added late (PB-RAG-05-03)
[stale_index] no longer in the index after a chunking change Re-chunking created new chunk ids Expected for old traces; re-ask the question and verify the new trace (PB-RAG-01-03)
Restored document never reappears Restored blob kept its old LastModified timestamp Resave the blob metadata to update the timestamp, then run the indexer
Problem started after a prompt release Prompt change weakened grounding instructions Roll back the prompt and run the groundedness gate (PB-RAG-07-01)
Verifier passes but the answer is still wrong Claim not supported by a correctly quoted chunk Score with the groundedness evaluator (1 to 5) in PB-RAG-06-03

10. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon Bedrock Knowledge Bases RetrieveAndGenerate returns citations: each has a generatedResponsePart (text span) and retrievedReferences (content, location, metadata) Map each span to its references and run the same verifier checks. guardrailAction shows INTERVENED when a guardrail acted. Not available for managed knowledge bases.
Amazon Bedrock Guardrails Contextual grounding check Scores grounding and relevance per response; see PB-RAG-06-02.
Open source Tracing tool (for example Langfuse or Arize Phoenix) plus verify_citations.py Export the trace to the same JSON shape; the verifier is platform-neutral.

11. Validation record

Pending. Planned dry-run scenario: A second engineer gets three sandbox traces: an invented citation, a renderer page offset and a stale document. Pass if each is diagnosed and the right fix path chosen within 2 hours.

Sandbox scenario: the author prepares three traces from the data/golden/A-policies/A_04 corpus in the sandbox, without labels:

  1. A trace that cites a chunk id not in retrieved.
  2. A trace whose rendered page is one higher than the chunk's page.
  3. A trace whose chunk was edited in the index after the answer.

The dry-runner also switches safe mode on and off once (steps 2 and 10) to test the rollback.

Pass criteria:

  • Each trace gets the right fault class and fix path from section 5.
  • All three are diagnosed within 2 hours in total.
  • Safe mode switches on and off, and each switch is verified.
  • An incident note is completed for at least one trace.

12. Sources

Checked on 28 Oct 2026.