⚠️ Before You Start

Risk: High

Duration: 2-5 days including evaluation and canary

Last Validated: Not yet validated (draft)

Kind: Runbook

Risk Note: High (every answer changes; retirement dates are fixed, so a late start leaves no time to fix regressions)

Re-validate prompts on a model version upgrade or retirement

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off.

1. Summary

This runbook guides teams through re-validating existing RAG prompts and migrating live traffic when a foundation model faces retirement, automatic upgrade, or replacement with a newer snapshot. You deploy the candidate version in parallel with automatic upgrades turned off, refresh the baseline and run the golden set through both deployments, canary live traffic via an application configuration switch, update snippets/prompts/registry.yaml, and keep the old deployment until sign-off or retirement. Facts were checked against Microsoft Learn on 30 October 2026. Management REST API version is 2024-10-01; data-plane inference API version is 2024-10-21 . The local evaluation SDK (azure-ai-evaluation) is in preview.

2. When to use and when not to

Use this runbook when:

• An Azure OpenAI model version approaches deprecation or its published retirement date.
• Upgrading to a newer, faster, or cheaper model snapshot (for example, moving between GPT-4o snapshots) behind an existing prompt.
• Microsoft announces an automatic rolling upgrade for a standard deployment tier, and you must verify grounding beforehand.
• Verifying whether existing system prompts maintain citation accuracy and token length budgets under new model weights.

Do not use it when:

• You only alter prompt wording, few-shot examples, or output schemas on an unchanged model deployment. Use PB-RAG-07-01 instead.
• You upgrade an embedding model across vector search indexes. Use PB-RAG-09-01 instead.
• You investigate an unexpected grounding drop or citation anomaly on the live system without a model change. Use PB-RAG-06-04 instead.
• You build initial RAG prompt templates from scratch. Use PB-RAG-07-03 instead.

3. What you must know first

1. Model lifecycle and retirement timelines. Microsoft sets Generally Available (GA) model retirement dates 18 months from launch. Models enter deprecation at 12 months. Official retirement notices are issued at least 60 days before retirement. Once retired, inference calls return 410 Gone.
2. Version upgrade policies. Standard deployments support three upgrade policies: Opt out of automatic model version upgrades, Upgrade once new default version becomes available, and Once the current version expires.
3. Provisioned throughput (PTU) requires manual upgrades. Provisioned deployments never auto-upgrade upon model retirement. Teams with PTU allocations must manually provision successor deployments and migrate traffic before the announced cutoff date.
4. Behavioural drift under identical prompts. Successor model weights alter token generation probabilities and refusal thresholds. An unchanged prompt on a new snapshot may produce longer answers, drop citations, or decline valid queries.
5. Parameter compatibility and breaking changes. Candidate models may reject legacy parameters (such as max_tokens versus max_completion_tokens) or enforce stricter JSON schemas. Verify model parameters on Microsoft Learn before cutover.
6. Decoupled routing via application configuration. Always deploy the candidate model alongside the current deployment under a distinct name. Route traffic via application configuration or feature flags so rollback completes within the team target of under 10 minutes without redeploying code.
7. Local evaluation SDK is preview. The Azure AI Evaluation SDK (azure-ai-evaluation) is in preview. Pin the exact package version in your CI/CD dependencies.

4. Prerequisites and access

Item	Detail
Roles	Cognitive Services OpenAI Contributor (manage deployments), Cognitive Services OpenAI User (execute inference), and App Configuration Data Owner (update routing keys). Use Microsoft Entra ID.
Tools	Azure CLI (az), Python 3.10+, openai Python SDK (>= 1.0.0), azure-ai-evaluation (pinned preview), curl, jq.
Deployment inventory	Current inventory recording deployment names, model versions, upgrade policies, and retirement dates (snippets/ops/model-inventory.yaml).
Judge deployment & workspace	A judge model deployment (${JUDGE_DEPLOYMENT}) for GroundednessEvaluator and a Log Analytics workspace (${WORKSPACE_ID}) receiving AzureDiagnostics.
Golden evaluation set	50–200 domain questions with reference context and expected refusals (exported from your project golden query set as JSONL with query, context fields).
Quota allocation	Sufficient Tokens Per Minute (TPM) quota (${CAPACITY_UNITS}) in the target subscription and region for parallel deployments.
Configuration switch	App Configuration keys (illustrative team keys: RAG:ModelDeployment:CanaryShare and RAG:ModelDeployment:Active) plus application routing code that reads those keys to split live traffic between deployments.
Change approval	Change ticket detailing candidate version, cutover window, rollback owner, and validation criteria.

Set the environment variables and obtain an access token:

export AZURE_OPENAI_ENDPOINT="https://${RESOURCE_NAME}.openai.azure.com"
export AZURE_SUBSCRIPTION_ID="${SUBSCRIPTION_ID}"
export RESOURCE_GROUP="${RESOURCE_GROUP_NAME}"
export LOCATION="${REGION}"
export AOAI_RESOURCE_ID="${AOAI_RESOURCE_ID}"
export JUDGE_DEPLOYMENT="${JUDGE_DEPLOYMENT}"
export WORKSPACE_ID="${WORKSPACE_ID}"
export CAPACITY_UNITS="${CAPACITY_UNITS}"
export TOKEN="$(az account get-access-token --resource https://cognitiveservices.azure.com --query accessToken --output tsv)"

5. Steps

Placeholders: ${RESOURCE_NAME}, ${AOAI_RESOURCE_ID}, ${RESOURCE_GROUP}, ${RESOURCE_GROUP_NAME}, ${SUBSCRIPTION_ID}, ${REGION}, ${LOCATION}, ${CANDIDATE_DEPLOYMENT}, ${CURRENT_DEPLOYMENT}, ${MODEL_NAME}, ${CANDIDATE_VERSION}, ${CAPACITY_UNITS}, ${JUDGE_DEPLOYMENT}, ${WORKSPACE_ID}, ${APP_CONFIG_NAME}.

Stage 1: Inventory and prepare candidate deployment

Step 1. Audit model lifecycle and upcoming retirement dates.

Query the Microsoft Cognitive Services management API to inspect model lifecycle states across the region:

MANAGEMENT_TOKEN="$(az account get-access-token --resource https://management.azure.com --query accessToken --output tsv)"
curl -sS -X GET \
  "https://management.azure.com/subscriptions/${AZURE_SUBSCRIPTION_ID}/providers/Microsoft.CognitiveServices/locations/${LOCATION}/models?api-version=2024-10-01" \
  -H "Authorization: Bearer ${MANAGEMENT_TOKEN}" | jq '.value[] | {model: .model.name, version: .model.version, lifecycle: .model.lifecycleStatus, deprecation: .model.deprecation}'

Verify: Output lists available model versions and identifies whether the active version is marked Deprecating or has an upcoming deprecation date. Record dates in snippets/ops/model-inventory.yaml.

Quota and billing impact: creating a parallel deployment consumes TPM quota and incurs cost

Deploying ${CANDIDATE_DEPLOYMENT} allocates ${CAPACITY_UNITS} TPM quota from your regional allowance and bills usage while active. To roll back if provisioning fails or tests abort, delete the candidate with az cognitiveservices account deployment delete --name "${RESOURCE_NAME}" --resource-group "${RESOURCE_GROUP}" --deployment-name "${CANDIDATE_DEPLOYMENT}" (see Section 7).

Step 2. Create the candidate deployment in parallel.

Deploy the candidate model version with policy set to no_auto_upgrade (Opt out of automatic model version upgrades) to guarantee stability during evaluation:

az cognitiveservices account deployment create \
  --name "${RESOURCE_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --deployment-name "${CANDIDATE_DEPLOYMENT}" \
  --model-name "${MODEL_NAME}" \
  --model-version "${CANDIDATE_VERSION}" \
  --model-format OpenAI \
  --sku-capacity "${CAPACITY_UNITS}" \
  --sku-name Standard

Verify: Deployment succeeds and az cognitiveservices account deployment show returns provisioningState: Succeeded.

Stage 2: Parallel evaluation against the golden set

Step 3. Refresh the baseline and evaluate the candidate deployment.

Execute compare_deployments.py to generate a fresh baseline from ${CURRENT_DEPLOYMENT} and candidate outputs from ${CANDIDATE_DEPLOYMENT} in the same batch under identical system prompts:

python snippets/eval/compare_deployments.py \
  --dataset "<path-to-your-golden-dataset.jsonl>" \
  --current-deployment "${CURRENT_DEPLOYMENT}" \
  --candidate-deployment "${CANDIDATE_DEPLOYMENT}" \
  --system-prompt "<path-to-your-system-prompt>" \
  --output-dir eval

Verify: Two JSONL answer files exist (eval/answers_${CURRENT_DEPLOYMENT}.jsonl and eval/answers_${CANDIDATE_DEPLOYMENT}.jsonl) with identical query counts and fresh timestamps. Metrics summary file eval/deployment_comparison.json is generated.

Step 4. Measure groundedness and relevance drift.

Run the Azure AI Evaluation SDK (preview) evaluators across both refreshed answer sets:

python -c '
import sys
from azure.ai.evaluation import AzureOpenAIModelConfiguration, GroundednessEvaluator, evaluate
cfg = AzureOpenAIModelConfiguration(azure_endpoint=sys.argv[1], azure_deployment=sys.argv[2])
evaluators = {"groundedness": GroundednessEvaluator(cfg)}
res_curr = evaluate(data=f"eval/answers_{sys.argv[3]}.jsonl", evaluators=evaluators)
res_cand = evaluate(data=f"eval/answers_{sys.argv[4]}.jsonl", evaluators=evaluators)
print("Current Groundedness:", res_curr["metrics"])
print("Candidate Groundedness:", res_cand["metrics"])
' "${AZURE_OPENAI_ENDPOINT}" "${JUDGE_DEPLOYMENT}" "${CURRENT_DEPLOYMENT}" "${CANDIDATE_DEPLOYMENT}"

Verify: Output displays mean Groundedness scores for both deployments.

Stage 3: Evaluation gate and prompt adjustment

Step 5. Compare evaluation metrics against release thresholds.

Review metrics against the illustrative team default gate thresholds ():

1. Mean Groundedness drop must not exceed 0.2 points on a 1–5 scale (team default).
2. Refusal accuracy on out-of-scope test cases must not degrade (team default).
3. Average completion token count must not exceed 1.25x of the refreshed baseline (team default).

If the candidate fails any threshold, do not proceed with cutover. Adjust the prompt template on a branch and validate it using PB-RAG-07-01 before continuing.

Verify: Evaluation report confirms zero threshold breaches. Inspect at least 20 sampled answers manually.

Stage 4: Canary release behind configuration switch

Production traffic exposure: live users receive candidate responses

Routing traffic to a new model deployment can alter user responses. Confirm the rollback owner is actively monitoring telemetry before enabling canary traffic.

Step 6. Route canary traffic to the candidate deployment.

Update your application configuration switch (illustrative team key: RAG:ModelDeployment:CanaryShare, implemented in your app's routing layer) to send 10% of live traffic to ${CANDIDATE_DEPLOYMENT}, then query AzureOpenAIRequests split by ModelDeploymentName:

az appconfig kv set \
  --name "${APP_CONFIG_NAME}" \
  --key "RAG:ModelDeployment:CanaryShare" \
  --value "0.10" \
  --yes
az monitor metrics list \
  --resource "${AOAI_RESOURCE_ID}" \
  --metric "AzureOpenAIRequests" \
  --dimension "ModelDeploymentName" \
  --interval PT5M \
  --output table

Verify: Metric output split by ModelDeploymentName shows non-zero Total request counts arriving at ${CANDIDATE_DEPLOYMENT} alongside ${CURRENT_DEPLOYMENT}.

Step 7. Monitor canary telemetry for 24–48 hours.

Run KQL query snippets/ops/queries/aoai-errors.kql against Log Analytics to track error rates and latency:

az monitor log-analytics query \
  --workspace "${WORKSPACE_ID}" \
  --analytics-query @snippets/ops/queries/aoai-errors.kql \
  --timespan P1D \
  --output table

Verify: Candidate deployment error rate remains below 1%, and p95 latency matches or improves upon baseline.

Stage 5: Cut over traffic, update registry, and decommission old deployment

Full production cutover: 100% of live traffic moves to ${CANDIDATE_DEPLOYMENT}

Promoting ${CANDIDATE_DEPLOYMENT} routes all production users to the new model version immediately. Confirm the 24–48 hour canary watch window passed with zero rollback triggers before proceeding. To roll back after cutover, revert the active key to ${CURRENT_DEPLOYMENT} (see Section 7).

Step 8. Promote candidate deployment to 100% traffic.

Update your application configuration keys (illustrative team keys: RAG:ModelDeployment:Active and RAG:ModelDeployment:CanaryShare) to route 100% of requests to ${CANDIDATE_DEPLOYMENT}:

az appconfig kv set \
  --name "${APP_CONFIG_NAME}" \
  --key "RAG:ModelDeployment:Active" \
  --value "${CANDIDATE_DEPLOYMENT}" \
  --yes
az appconfig kv set \
  --name "${APP_CONFIG_NAME}" \
  --key "RAG:ModelDeployment:CanaryShare" \
  --value "0.00" \
  --yes

Verify: AzureOpenAIRequests split by ModelDeploymentName shows 100% of new traffic reaching ${CANDIDATE_DEPLOYMENT} and zero new requests on ${CURRENT_DEPLOYMENT}.

Step 9. Keep the old deployment until sign-off or retirement, and confirm upgrade policy.

Keep ${CURRENT_DEPLOYMENT} deployed and warm during the 48-hour post-cutover watch window until formal sign-off (or Microsoft's retirement cutoff) is reached. Confirm the upgrade policy configured on ${CANDIDATE_DEPLOYMENT}:

az cognitiveservices account deployment show \
  --name "${RESOURCE_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --deployment-name "${CANDIDATE_DEPLOYMENT}" \
  --query "properties"

Verify: ${CURRENT_DEPLOYMENT} remains provisioned as the rollback target, and ${CANDIDATE_DEPLOYMENT} shows the intended upgrade policy.

Step 10. Update the prompt registry (snippets/prompts/registry.yaml).

Update model, model_version, change_ticket, and last_updated in snippets/prompts/registry.yaml and snippets/ops/model-inventory.yaml, then commit the change:

git add snippets/prompts/registry.yaml snippets/ops/model-inventory.yaml
git commit -m "chore(rag-07): record ${CANDIDATE_DEPLOYMENT} cutover in registry and inventory"

Verify: snippets/prompts/registry.yaml reflects the active ${MODEL_NAME} and ${CANDIDATE_VERSION} values.

Permanent removal: old model deployment is deleted

Deleting ${CURRENT_DEPLOYMENT} destroys the immediate rollback target. Ensure formal watch-window sign-off is recorded (or the retirement date is reached) before deleting.

Step 11. Decommission the old deployment.

Delete the old model deployment once the watch window has elapsed or the retirement date is reached:

az cognitiveservices account deployment delete \
  --name "${RESOURCE_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --deployment-name "${CURRENT_DEPLOYMENT}"

Verify: az cognitiveservices account deployment list confirms ${CURRENT_DEPLOYMENT} is deleted. Update snippets/ops/model-inventory.yaml.

6. Monitoring and alerts

The thresholds below are illustrative team defaults; tune them against your measured production baseline.

Signal	Source	Alert when	Action
Days until retirement	Scheduled inventory check (snippets/ops/model-inventory.yaml)	Fewer than 60 days to retirement with no upgrade ticket open	Open an upgrade ticket and initiate this runbook.
AzureOpenAIRequests by StatusCode	Azure Monitor metric / Log Analytics (snippets/ops/queries/aoai-errors.kql)	Client (4xx) or server (5xx) errors exceed 1% of requests for 15 minutes	Roll back traffic switch (Section 7) and investigate payload failures.
AzureOpenAITTLTInMS (Latency)	Azure Monitor metric	Average response latency rises > 1.5x baseline for 10 minutes	Review completion token lengths; increase capacity units if throttled.
GeneratedTokens (Completion tokens)	Azure Monitor metric	Average completion tokens per request increase > 25% over baseline for 1 hour	Review prompt length instructions and enforce explicit max_tokens limits.
Groundedness score	Offline evaluation on sampled production queries	Mean score drops > 0.2 on 1–5 scale over 24 hours (team default)	Revert routing switch to previous deployment and refine system prompt.

7. Rollback

Situation	Rollback	Time
Provisioning or test abort in Stage 1–2	Delete ${CANDIDATE_DEPLOYMENT} via az cognitiveservices account deployment delete. Keep ${CURRENT_DEPLOYMENT} untouched.	Under 5 minutes
Evaluation failure in Stage 3	Cancel cutover. Keep traffic 100% on ${CURRENT_DEPLOYMENT}. Refine prompts on a branch.	Under 1 minute
Canary anomaly in Stage 4	Set canary traffic share to 0% in your app configuration switch (example key: RAG:ModelDeployment:CanaryShare = 0.00).	Under 10 minutes
Post-cutover regression in Stage 5 (watch window)	Revert active deployment back to ${CURRENT_DEPLOYMENT} (example key: RAG:ModelDeployment:Active = ${CURRENT_DEPLOYMENT}).	Under 10 minutes
Old deployment retired by Microsoft before cutover	Roll forward only: adjust prompt on candidate deployment. Rollback impossible once retired (410 Gone).	Variable

Rollback triggers (team defaults — ; any one triggers rollback):

1. Error rate on candidate deployment exceeds 1% of total requests over 15 minutes (team default).
2. Groundedness or relevance evaluation drops by more than 0.2 points on a 1–5 scale on production samples (team default).
3. User refusal rate increases by more than 1.5x baseline over 1 hour (team default).
4. p95 latency or completion token generation exceeds 1.5x baseline without business justification (team default).

8. Known issues and fixes

Symptom	Likely cause	Fix
Output answers get longer and cost per answer rises	Successor model snapshot exhibits greater verbosity under identical prompt instructions	Add concise length constraints in system instructions, set max_tokens or max_completion_tokens, and re-evaluate.
Model version changes unexpectedly without a planned release	Deployment was configured with Upgrade once new default version becomes available or Once current version expires	Set the deployment upgrade policy explicitly to NoAutoUpgrade (Opt out of automatic model version upgrades).
Requests fail with HTTP 400 parameter errors after switch	Candidate model rejects deprecated parameters (such as max_tokens versus max_completion_tokens)	Verify model parameter reference on Microsoft Learn and make API parameters version-specific in configuration.
HTTP 429 RateLimitReached on the new deployment	Insufficient TPM quota allocated to the candidate deployment in the subscription or region	Request additional quota via Azure portal before starting canary, or reduce concurrency.
Increased refusals or loss of citation format	Safety filter sensitivity or system prompt adherence differs on the candidate model	Include refusal cases in the golden set and refine prompt boundary rules following PB-RAG-07-01.
Requests return HTTP 410 Gone immediately after cutover	The target model version reached its scheduled retirement date and was decommissioned by Microsoft	Ensure migration occurs at least 30 days prior to retirement date; deploy the designated GA replacement model.

9. AWS and open-source equivalents

Platform	Equivalent	Key differences
Amazon Bedrock	Model lifecycle (Legacy and EOL phases) and Cross-Region Inference Profiles	Bedrock models move to Legacy status before end-of-life. Pin specific model ARNs in application configuration. Use Bedrock Model Evaluation or prompt management versions to validate prompts before switching production alias ARNs.
Self-hosted / Open Source (e.g. vLLM, Ollama, TGI)	Image tag and model weight revision pinning	Models do not auto-upgrade. Pin container image tags and Hugging Face commit SHAs (revision="<commit_sha>"). Spin up candidate container in parallel, run golden evaluation suite against the OpenAI-compatible /v1/chat/completions endpoint, then repoint service DNS or reverse proxy.

10. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A dry-run engineer provisions two parallel model deployments in a sandbox resource group (current model version and candidate successor version). The engineer runs snippets/eval/compare_deployments.py across the golden dataset, verifies that grounding and latency pass gate thresholds, enables a 10% canary routing switch, and exercises a simulated rollback within 10 minutes.

Pass criteria:

• Comparative evaluation executes across the golden set with zero unhandled errors.
• Gating thresholds detect artificial regressions (for example, simulated token inflation or prompt degradation).
• Rollback via configuration switch restores 100% traffic to the baseline deployment in under 10 minutes.

Record

Item	Detail
Playbook	PB-RAG-07-02 at commit [hash]
Dry-run by	[Name] (must not be author or reviewer)
Date	YYYY-MM-DD
Sandbox	[Resource group or tag, tier and region]
Scenario	[Scenario followed]
Time taken	[Hands-on time] hands-on, [waiting time] waiting
Result	[PASS / PASS WITH NOTES / FAIL]
Notes	1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by	[Reviewer name and date]

11. Sources

Facts checked against official Microsoft Learn documentation on 30 October 2026.

• Microsoft Learn, Azure OpenAI model deprecations and retirements: 18-month GA lifecycle, 12-month deprecation, 60-day notification, 410 Gone return code on retirement, Models management API.
• Microsoft Learn, Model versions in Microsoft Foundry Models: Update policies (Opt out of automatic model version upgrades, Upgrade once new default version becomes available, Once the current version expires), PTU manual migration requirement.
• Microsoft Learn, Monitoring data reference for Azure OpenAI: AzureOpenAIRequests metric, dimensions (ModelDeploymentName, ModelVersion, StatusCode), latency metrics (AzureOpenAITTLTInMS), token metrics (GeneratedTokens).
• Microsoft Learn, Evaluate your generative AI application locally with the Azure AI Evaluation SDK (preview): azure-ai-evaluation package, preview status without SLA, built-in evaluators (GroundednessEvaluator, RelevanceEvaluator).

Related playbooks

• PB-RAG-06-04: Investigate an ungrounded answer or wrong citation incident
• PB-RAG-06-05: Prompt or model change without grounding regressions checklist
• PB-RAG-07-01: Release a prompt change without regressions
• PB-RAG-07-03: Build the standard RAG templates: grounded QA, summary, comparison, refusal and multi-document

📄 Snippet CODE

📋 Copy ↗ Raw ×