Skip to content

Prompt or Model Change Review Checklist

  • Pull Request: ${PR_NUMBER}
  • Ticket / Work Item: ${TICKET_ID}
  • Author: ${AUTHOR_NAME}
  • Reviewer: ${REVIEWER_NAME}
  • Target Deployment / Model: ${MODEL_DEPLOYMENT} (${MODEL_VERSION})

1. Change Motivation and Scope

  • Change rationale documented: Author records the specific business trigger (user feedback, failure log, tone tuning, or model upgrade) in the pull request description; Reviewer confirms reference.
  • Diff inspected: Author and Reviewer inspect git diff origin/main to confirm only intended prompt templates, instructions, or hyperparameters are modified.
  • Sampling parameters: Author confirms temperature is pinned to 0.0 as an illustrative team convention for standard chat models (omit parameter for reasoning models such as o1 and o3-mini); Reviewer verifies in deployment configuration.

2. Model and Schema Compatibility

  • Model version compatibility: Author verifies the target model is listed in the Azure OpenAI structured outputs supported models table; Reviewer verifies in deployment configuration.
  • Explicit version pinning: Author confirms deployment configuration sets version update policy to NoAutoUpgrade (or an explicit date-stamped version rather than unversioned auto-upgrade); Reviewer checks deployment configuration.
  • Pydantic schema validation: Author confirms the JSON schema uses model_config = ConfigDict(extra="forbid") and all fields are required (PB-RAG-06-01); Reviewer confirms schema tests pass.

3. Evaluation and Regression Results

  • promptfoo eval executed against golden query set:
  • Context faithfulness rate: ${CONTEXT_FAITHFULNESS_RATE}% (Illustrative team default threshold: >= 95%) — Author runs test; Reviewer checks CI log.
  • Latency P95: ${P95_LATENCY} ms (Illustrative team budget: <= 5000 ms) — Author benchmarks; Reviewer validates.
  • Automated Groundedness Gate (PB-RAG-06-03):
  • Groundedness pass rate: ${GROUNDEDNESS_PASS_RATE}% (Release Gate: >= 95% of queries with evaluator score >= 4.0 / 5.0) — Author runs pipeline; Reviewer checks build status.
  • Zero regression on answerable queries against production baseline.
  • Code-level citation verification (PB-RAG-06-01):
  • 100% of cited chunk IDs and verbatim quotes verified against context — Author runs test suite; Reviewer checks verification report.
  • Zero fabricated citations emitted.

4. No-Answer and Refusal Calibration

  • Unanswerable test queries (PB-RAG-06-02):
  • Non-answer rate on the unanswerable set meets targets.non_answer_rate_on_unanswerable in no-answer-policy.yaml (team default 0.90; model returns answerable: false and empty claims) — Author verifies in test output; Reviewer inspects results.
  • False refusal rate on answerable queries does not increase over baseline (PB-RAG-06-02) — Author verifies on golden set; Reviewer validates.
  • Standard refusal template preserved without conversational drift — Reviewer inspects rendered messages.

5. Rollback and Deployment Readiness

  • Fast rollback toggle verified: Author confirms rollback environment variable PROMPT_VERSION (or blue-green routing toggle) is configured; Reviewer tests rollback switch.
  • Canary monitoring window scheduled: Author schedules post-release observation window (illustrative team default: 24 to 48 hours) to monitor live CitationVerificationFailureRate and UnanswerableRate; Reviewer signs off.

Sign-off

Role Name Signature / Approval Date
Author ${AUTHOR_NAME}
Reviewer / RAG Lead ${REVIEWER_NAME}