Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 hours per change
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Medium (silent regressions reach users)

Prompt or model change without grounding regressions checklist

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off. Author: Gayathri Ramachandran · dry-run: Yasar · sign-off: Deeparani. Work items: RAG-06-T17, RAG-06-T18, RAG-06-T19, RAG-06-T20.

1. Summary

A change to a RAG prompt, system message, decoding setting or model deployment affects every live answer at once. Small wording changes can weaken grounding, invent citations or shift the refusal rate without any error being raised. The checklist below is the review gate for such changes, completed by the author of the change and a reviewer before the change merges or reaches users. It reuses the tools from the rest of the pack: code-level citation checks (PB-RAG-06-01), refusal targets (PB-RAG-06-02) and the groundedness gate (PB-RAG-06-03). Record the result in snippets/prompt-change-checklist.md.

2. When to use and when not to

Use this checklist when:

  • Reviewing pull requests that modify system prompt instructions, grounding rules, citation format requirements, or few-shot examples.
  • Switching model deployments, upgrading date-stamped model versions, or changing model families (for example, between gpt-4o and gpt-4o-mini).
  • Adjusting decoding parameters (such as temperature, top_p, or reasoning effort) that alter generation entropy and factual adherence.
  • Modifying retrieval context formatting or the placement of instructions relative to retrieved context.

Do not use it when:

  • You are orchestrating the canary deployment, traffic routing, or production rollout. Use PB-RAG-07-01 instead.
  • You are establishing the automated evaluation pipeline or configuring groundedness gating thresholds in CI. Use PB-RAG-06-03 instead.
  • You are triaging an active live ungrounded answer incident or citation defect. Use PB-RAG-06-04 instead.
  • You are changing chunking profiles or embedding models. Use PB-RAG-01-01 and PB-RAG-01-03 instead.

3. What you must know first

  • Model version drift. Changing date-stamped model versions alters instruction-following nuance and refusal behaviour even when using identical prompt text.
  • Structured outputs compatibility. Structured outputs (response_format with json_schema and strict: true) need a supported model version; see the Azure OpenAI structured outputs documentation. Unsupported models return 400 BadRequest.
  • Deployment version policies. Azure OpenAI deployments support version upgrade policies: NoAutoUpgrade (pinned to the current version), OnceNewDefaultVersionAvailable and OnceCurrentVersionExpired. Team rule: production deployments use NoAutoUpgrade, so a model version changes only through this checklist.
  • Context faithfulness ratio. In promptfoo, context-faithfulness is the share of claims in the output that the context supports: supported claims divided by all claims. A score below 1.0 means at least one claim is not supported by the context.
  • Grounding and answerability trade-off. Tightening grounding rules to suppress hallucination can increase false refusals on valid answerable questions; loosening rules invites confabulation. Evaluations must test answerable and unanswerable golden datasets simultaneously.
  • Instruction placement. With long retrieved context, the model can follow instructions placed only at the top less well. Repeat the grounding rules after the retrieved context as well.
  • Code verification necessity. System prompts cannot guarantee 100% citation veracity on their own; deterministic code verification (PB-RAG-06-01) must validate that every cited chunk ID and quote exists verbatim in the context.

4. Checklist

Production risk: prompt changes affect all live answers at once

Modifying system prompts or updating model deployments alters instruction-following across 100% of concurrent live sessions immediately. Never apply prompt or model version changes directly in production without completing all checklist items and securing signed-off evaluation results.

1. Prerequisites and change documentation

  • Verify test environment credentials: Author and Reviewer confirm access to sandbox Azure OpenAI resource and target deployment:
export AOAI_ENDPOINT="https://${AZURE_OPENAI_SERVICE}.openai.azure.com"
export CHAT_DEPLOYMENT="${CHAT_MODEL_DEPLOYMENT}"
export PROMPT_VERSION="${PROMPT_VERSION}"
  • Inspect git diff: Author and Reviewer inspect git diff origin/main to confirm only intended prompt text, system messages, or hyperparameters are altered.
  • Record change rationale: Author records the business trigger in the pull request description; Reviewer verifies ticket reference.
  • Confirm baseline evaluation: Author captures current production groundedness pass rate and refusal rate on golden test set (PB-RAG-06-03) for regression comparison.
  • Verify sampling parameters: Author confirms temperature is pinned to 0.0 as an illustrative team convention for standard chat models. Reasoning models (o1, o3-mini) do not accept a temperature parameter and must omit it; Reviewer validates config.

2. Model and schema compatibility

Breaking change risk: unversioned model updates cause runtime failures

Upgrading to an unsupported model deployment or allowing unversioned auto-updates can trigger runtime 400 BadRequest errors or silent citation drops. Ensure the target model supports structured outputs before routing production traffic.

  • Verify structured outputs support: Author confirms target model deployment is listed in the Azure OpenAI structured outputs documentation; Reviewer verifies deployment properties.
  • Verify Pydantic schema constraints: Author and Reviewer confirm all models retain model_config = ConfigDict(extra="forbid"), all properties are required, and unsupported keywords (minLength, pattern, format) are absent (PB-RAG-06-01).
  • Confirm explicit model version: Author confirms deployment configuration sets version update policy to NoAutoUpgrade rather than unversioned auto-updates; Reviewer verifies in deployment configuration.

3. Grounding evaluation and regression gating

  • Execute promptfoo evaluation: Author executes npx promptfoo eval -c snippets/eval/promptfooconfig.yaml against golden query set (FND-04); Reviewer reviews report:
npx promptfoo eval -c snippets/eval/promptfooconfig.yaml
npx promptfoo view -p 15500
  • Verify context-faithfulness score: Author and Reviewer confirm all answerable queries pass the context-faithfulness assertion with threshold 0.95 or higher (team default), and every unanswerable query returns answerable: false.
  • Run groundedness release gate: Author runs the gate on answers generated by the changed prompt or model (PB-RAG-06-03, steps 2 and 3); Reviewer confirms exit code 0 (team defaults: pass rate 0.95 or higher, row score 4 or higher) and no drop against the baseline.
  • Verify citation veracity in code: Author executes code-level verification (PB-RAG-06-01); Reviewer confirms zero invalid chunk IDs or hallucinated quotes on answerable test queries:
# pseudo
from snippets.answer_with_citations import verify_answer_citations

verified_answer, warnings = verify_answer_citations(raw_answer, chunks)
assert verified_answer.answerable is True
assert len(warnings) == 0

4. No-answer and refusal calibration

  • Test unanswerable query set: Author executes unanswerable test queries (PB-RAG-06-02); Reviewer verifies the non-answer rate meets targets.non_answer_rate_on_unanswerable in no-answer-policy.yaml (team default 0.90), with answerable: false and empty claims.
  • Check false refusal rate: Author compares refusal rate on valid answerable questions against production baseline to confirm stricter prompt wording does not reject valid inquiries (PB-RAG-06-02); Reviewer checks calibration report.
  • Verify refusal message format: Author and Reviewer check that refusals use the approved wording (R1, R2, R3) from snippets/refusal-wording.md, not text written by the model.

5. Latency, token budget, and rollout readiness

  • Measure prompt token budget: Author counts tokens using tiktoken to confirm the prompt template remains within allocated budget without preamble inflation; Reviewer verifies count.
  • Check P95 latency headroom: Author benchmarks end-to-end response latency against latency targets (illustrative team budget: P95 <= 5000 ms); Reviewer validates headroom.
  • Confirm fast rollback mechanism: Author and Reviewer confirm prompt version environment variable (PROMPT_VERSION) or deployment routing toggle is configured for instantaneous rollback without redeploying code (PB-RAG-07-01).
  • Complete change review sign-off: Author and Reviewer fill in all checklist fields and record signatures in snippets/prompt-change-checklist.md.
  • Configure canary observation: Author schedules post-release monitoring window (illustrative team default: 24 to 48 hours) to inspect live citation verification metrics; Reviewer approves canary plan.

Representative promptfoo test configuration from snippets/eval/promptfooconfig.yaml:

providers:
  - id: azure:chat:${CHAT_MODEL_DEPLOYMENT}
    config:
      apiHost: ${AZURE_OPENAI_SERVICE}.openai.azure.com
      temperature: 0.0
tests:
  - description: Unanswerable query must return answerable false with no claims
    vars:
      query: What is the company policy on pet insurance reimbursement?
      context: Corporate Travel and Expense Policy covers airfare, rail, lodging, and meals.
    assert:
      - type: is-json
      - type: javascript
        value: JSON.parse(output).answerable === false && JSON.parse(output).claims.length === 0

The prompt returns JSON (answerable and claims), so test refusals on the JSON fields, not on refusal text.

5. Known issues and fixes

Symptom Likely cause Fix
Faithfulness drops on long contexts Grounding instructions placed only at the top of prompt decay across long contexts ("lost in the middle") Frame grounding rules both in the system message and directly adjacent to the user query before the response.
Structured output API returns 400 BadRequest after model switch New model deployment or API version does not support JSON schema strict mode Verify model version is listed in the supported models table and data plane API version supports structured outputs.
Refusal rate spikes on answerable queries Stricter prompt instruction wording causes over-refusal on partial evidence Recalibrate prompt instructions against the no-answer decision guide (PB-RAG-06-02) and run evaluation against golden queries.
Hallucinated quotes pass schema validation Model satisfies JSON schema format with fabricated quote text Execute code-level citation verification (PB-RAG-06-01) to drop claims referencing non-existent quotes or chunk IDs.
Initial query after deployment exhibits latency spike Initial inference request triggers schema parsing and grammar initialization overhead Execute a synthetic warm-up inference request before routing production user traffic.

6. Monitoring and alerts

The thresholds below are illustrative team defaults; tune them against your measured production baseline.

Signal Source Alert when Action
ContextFaithfulnessRate CI evaluation pipeline Below 95% on golden query set Block merge in pull request. Inspect prompt diff and failed claims.
CitationVerificationFailureRate Application metrics Above 2% of claims over 1 hour Inspect recent prompt changes and start runbook PB-RAG-06-04.
No-answer rate Application metrics See the no-answer rate alert in PB-RAG-06-02 Check for an over-refusal prompt regression, then follow PB-RAG-06-02.
SchemaParsingErrorRate Application logs Above 0.1% over 15 minutes Check for unversioned model redeployments or SDK schema mismatches.
P95InferenceDuration APM / Platform metric P95 above 5 seconds for 10 minutes Review prompt token length and schema nesting depth.

7. AWS and open-source equivalents

Platform Equivalent Key differences
AWS Bedrock Contextual Grounding Check (Guardrails) & Prompt Management Bedrock Guardrails evaluates contextual grounding natively with float thresholds (0 to 0.99; 1.0 is disallowed) and blocks ungrounded output. Prompt Management provides built-in versioning (DRAFT and immutable numbered versions 1, 2, 3) referenced by version ARN.
Open-source (promptfoo) context-faithfulness assertion Model-graded assertion in CI that evaluates supported claims over total claims using LLM-as-a-judge. Runs locally or in CI pipelines without managed platform dependencies.
Open-source (DeepEval / Ragas) FaithfulnessMetric (DeepEval) / faithfulness (Ragas) Computes factual consistency between retrieval context and generated response. DeepEval supports local thresholding and test runner integration.

8. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A second engineer reviews two prepared prompt modifications against the golden dataset (FND-04):

  1. A valid prompt refinement adjusting citation tone while retaining grounding instructions.
  2. A regressed prompt that removes the negative constraint ("Do not use external knowledge") and softens verbatim quote instructions.

Pass criteria: The checklist and CI evaluation gate cleanly approve modification 1 and catch modification 2 with context-faithfulness failures and citation verification drops.

Record

Item Detail
Playbook PB-RAG-06-05 at commit [hash]
Dry-run by Yasar
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by Deeparani

9. Sources

Checked on 30 October 2026.