Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 days
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (every answer changes at once; a loose or strict refusal rule can block valid questions or allow out-of-scope ones)

Release a prompt change without regressions

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off.

1. Summary

This runbook executes a safe, zero-downtime release of changes to system prompt templates, few-shot examples, or output schemas in a live RAG application. Path A (configuration-driven via Azure App Configuration variant feature flags) is the recommended default, enabling instant prompt switching with no container redeployment; Path B (Azure Container Apps revision traffic splitting) is the deployment-driven alternative when prompt templates are baked into images. It gates changes in CI/CD by comparing candidate evaluation scores against a fresh baseline run, deploys to a canary traffic split (5–10%), and monitors production telemetry for 24–48 hours before full rollout or fast rollback in under 10 minutes.

Risk: Medium (every answer changes at once; a loose or strict refusal rule can block valid questions or allow out-of-scope ones). Typical duration: 1-2 days hands-on, plus 24–48 hours canary observation.

Facts were checked against official sources on 2 November 2026. Azure OpenAI Service inference API stable release 2024-10-21 is used for prompt execution; the Azure AI Evaluation SDK is in preview (azure-ai-evaluation==1.1.0 ).

2. When to use and when not to

Use this runbook when:

  • Editing system prompts, prompt templates, few-shot examples, or output schema formatting of a live RAG assistant.
  • Tuning assistant tone, response verbosity, conciseness, or refusal behaviour based on user feedback or compliance reviews.
  • Rolling out prompt changes through canary traffic allocation and automated evaluation regression gates.

Do not use it when:

  • You are triaging an active ungrounded answer incident or citation defect without changing prompts. Use PB-RAG-06-04 instead.
  • You are completing the pre-merge prompt review checklist. Use PB-RAG-06-05 instead.
  • You are retiring or upgrading underlying foundation model deployments without editing prompt templates. Use PB-RAG-07-02 instead.
  • You are re-indexing an Azure AI Search index or re-chunking documents. Use PB-RAG-09-01 or PB-RAG-01-03 instead.

3. What you must know first

  1. Evaluator specificity. Groundedness and relevance evaluators catch distinct failure modes. GroundednessEvaluator detects ungrounded claims and hallucinations (measures adherence to context), whereas RelevanceEvaluator measures whether the answer addresses the user's intent. Both must be evaluated independently.
  2. Relative baseline comparison. AI-assisted evaluators exhibit inherent non-determinism. To control for LLM-as-a-judge scoring variance, evaluate candidate and baseline prompts within the same evaluation batch against the same golden dataset rather than relying on absolute historical score thresholds.
  3. Refusal boundary calibration. Tightening grounding directives suppresses hallucinations but increases false refusals on partially answerable queries. Evaluations must include dedicated test sets for out-of-scope, missing context, and adversarial queries alongside standard answerable questions.
  4. Evaluation SDK version pinning. The Azure AI Evaluation SDK package (azure-ai-evaluation) evolves across preview and stable milestones. Always pin the exact package version in CI/CD pipeline dependencies (azure-ai-evaluation==1.1.0 ) to avoid silent breaking changes in metric extraction.
  5. Telemetry correlation and version tagging. Emit the active prompt version (prompt_version), model deployment name, and template hash as custom dimensions in application telemetry (Application Insights / OpenTelemetry) with every inference request to isolate regressions during canary rollouts.
  6. Configuration-driven fast rollback. Rollback must be decoupled from code redeployments whenever possible. Routing to a prompt version should default to dynamic feature flags in Azure App Configuration (Path A), allowing reversion in under 10 minutes without restarting services or redeploying containers.

4. Prerequisites and access

Item Detail
Roles Cognitive Services OpenAI User on Azure OpenAI; App Configuration Data Owner (Path A) or Azure Container Apps Contributor (Path B) to adjust canary traffic routing. Use Microsoft Entra ID.
Tools & SDKs Python 3.10+, azure-ai-evaluation==1.1.0 , azure-identity==1.16.0 , Azure CLI (az), GitHub CLI (gh).
Golden dataset Approved evaluation dataset in JSONL format with 50–200 representative query/context pairs and expected refusals (team default; ).
Change approval Change ticket referencing the prompt pull request, agreed evaluation regression tolerance, canary observation window (24–48 hours), and designated rollback owner.

Set the environment variables:

export AZURE_OPENAI_ENDPOINT="https://${AZURE_OPENAI_SERVICE}.openai.azure.com"
export JUDGE_DEPLOYMENT="${JUDGE_MODEL_DEPLOYMENT}"
export CURRENT_PROMPT_VERSION="v1.2.0"
export CANDIDATE_PROMPT_VERSION="v1.3.0"
export APP_CONFIG_NAME="${APP_CONFIGURATION_NAME}"
export CONTAINER_APP_NAME="${CONTAINER_APP_NAME}"
export RESOURCE_GROUP="${RESOURCE_GROUP}"

5. Decide your approach

Approach Path Rollback time Requires image redeployment? When to choose
Azure App Configuration variant feature flag on PROMPT_VERSION Path A (Default) Under 10 minutes No (instant configuration switch) Standard production default. Application dynamically selects prompt templates at runtime based on feature flag allocation or key-value configuration.
Azure Container Apps revision traffic splitting Path B (Alternative) Under 10 minutes Yes (new container revision per prompt change) Use when prompt templates are compiled immutably into container images or bundled with backend code and dependency updates.

6. Steps

Placeholders: ${APP_CONFIG_NAME} (Azure App Configuration store name), ${CONTAINER_APP_NAME} (Container App name), ${RESOURCE_GROUP} (resource group name), ${CURRENT_REVISION} (active revision name), ${CANDIDATE_REVISION} (new canary revision name).

Stage 1: Prepare and baseline

Step 1. Register candidate prompt version in the registry.

Update the prompt registry entry in snippets/prompts/registry.yaml with the target version tag ${CANDIDATE_PROMPT_VERSION} and change description:

yq eval '.prompts.rag_qa.candidate_version = "'"${CANDIDATE_PROMPT_VERSION}"'"' \
  -i snippets/prompts/registry.yaml

Verify: snippets/prompts/registry.yaml contains ${CANDIDATE_PROMPT_VERSION} under active candidate entries.

Step 2. Open prompt pull request.

Create a Git feature branch, commit the prompt changes, and open a pull request targeting main:

git checkout -b "docs/RAG-07-T02-prompt-${CANDIDATE_PROMPT_VERSION}"
git add snippets/prompts/registry.yaml prompts/
git commit -m "docs(RAG-07): candidate prompt version ${CANDIDATE_PROMPT_VERSION} [RAG-07-T02]"
gh pr create \
  --title "feat(RAG-07): release prompt ${CANDIDATE_PROMPT_VERSION}" \
  --body "Prompt update candidate with automated regression gate."

Verify: PR diff touches only intended prompt template lines, instructions, or few-shot examples.

Step 3. Generate baseline evaluation dataset.

Execute current production prompt ${CURRENT_PROMPT_VERSION} against the golden query set (team default: 50–200 queries) and capture outputs into eval/answers_baseline.jsonl:

python snippets/eval/generate_answers.py \
  --prompt-version "${CURRENT_PROMPT_VERSION}" \
  --dataset "data/golden/qa_eval_set.jsonl" \
  --output "eval/answers_baseline.jsonl"

Verify: eval/answers_baseline.jsonl contains valid JSONL records with non-empty query, context, and response.

Stage 2: Evaluate and compare

Step 4. Generate candidate evaluation dataset.

Execute candidate prompt revision ${CANDIDATE_PROMPT_VERSION} against the identical golden query set using identical retrieval settings and temperature:

python snippets/eval/generate_answers.py \
  --prompt-version "${CANDIDATE_PROMPT_VERSION}" \
  --dataset "data/golden/qa_eval_set.jsonl" \
  --output "eval/answers_candidate.jsonl"

Verify: eval/answers_candidate.jsonl has matching query counts and structure to eval/answers_baseline.jsonl.

Step 5. Run automated regression comparison gate.

Execute the CI evaluation comparison gate script (snippets/eval/gate.py):

python snippets/eval/gate.py \
  --baseline eval/answers_baseline.jsonl \
  --candidate eval/answers_candidate.jsonl \
  --tolerance 0.2

Verify: The script outputs baseline vs candidate scores for groundedness and relevance. Exits with code 0 if all scores remain within the agreed tolerance (team default: a drop of 0.2 or less), or exit code 1 if regression exceeds tolerance.

Step 6. Conduct manual qualitative sample inspection.

Inspect at least 20 changed responses by hand (team default), including all newly introduced refusals and dropped citations.

Verify: Responses maintain domain tone, verbatim attribution accuracy, and standard refusal phrasing without conversational drift.

Stage 3: Canary rollout

Canary exposure impacts live users

Canary traffic sends production queries to the new prompt. Ensure telemetry version tagging is active and monitor error rates closely during initial routing. If quality or latency degrades, execute immediate rollback (Section 8).

Step 7. Route canary traffic to candidate prompt version.

Configure canary traffic allocation (team default: 5–10%) using your chosen path:

Path A (Default: App Configuration feature flag variant allocation):

Add a 10% percentage allocation filter to the PromptVersionCanary feature flag:

az appconfig feature filter add \
  --name "${APP_CONFIG_NAME}" \
  --feature "PromptVersionCanary" \
  --filter-name "Microsoft.Percentage" \
  --filter-parameters Value=10 \
  --yes

Verify: az appconfig feature show --name "${APP_CONFIG_NAME}" --feature "PromptVersionCanary" confirms the 10% filter is enabled.

Path B (Alternative: Container Apps revision traffic split):

Ensure multiple revision mode is enabled, then allocate 10% ingress traffic to the candidate revision:

az containerapp revision set-mode \
  --name "${CONTAINER_APP_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --mode multiple

az containerapp ingress traffic set \
  --name "${CONTAINER_APP_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --revision-weight "${CURRENT_REVISION}=90" "${CANDIDATE_REVISION}=10"

Verify: az containerapp ingress traffic show --name "${CONTAINER_APP_NAME}" --resource-group "${RESOURCE_GROUP}" displays 90 for ${CURRENT_REVISION} and 10 for ${CANDIDATE_REVISION}.

Step 8. Monitor canary telemetry over observation window.

Observe canary and control telemetry for 24–48 hours across groundedness alerts, refusal rates, and P95 latency (team default: 5000 ms SLA; ).

Verify: Canary telemetry exhibits no statistically significant increase in refusal rate or decrease in groundedness relative to control traffic (team default: refusal rate increase of 1.5x or less, groundedness score drop of 0.2 or less; ). Application telemetry records emit prompt_version = ${CANDIDATE_PROMPT_VERSION} for canary traffic.

Stage 4: Promote and complete

Step 9. Promote candidate prompt to 100% production traffic.

Update routing to direct 100% of live traffic to the candidate prompt version:

Path A (Default: App Configuration):

Set the production key PROMPT_VERSION to ${CANDIDATE_PROMPT_VERSION} and remove the canary filter:

az appconfig kv set \
  --name "${APP_CONFIG_NAME}" \
  --key "PROMPT_VERSION" \
  --value "${CANDIDATE_PROMPT_VERSION}" \
  --yes

az appconfig feature filter delete \
  --name "${APP_CONFIG_NAME}" \
  --feature "PromptVersionCanary" \
  --filter-name "Microsoft.Percentage" \
  --yes

Verify: az appconfig kv show --name "${APP_CONFIG_NAME}" --key "PROMPT_VERSION" returns ${CANDIDATE_PROMPT_VERSION}.

Path B (Alternative: Container Apps):

Set 100% traffic weight on candidate revision:

az containerapp ingress traffic set \
  --name "${CONTAINER_APP_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --revision-weight "${CANDIDATE_REVISION}=100"

Verify: az containerapp ingress traffic show --name "${CONTAINER_APP_NAME}" --resource-group "${RESOURCE_GROUP}" shows 100% traffic routed to ${CANDIDATE_REVISION}.

Step 10. Record release outcome and merge pull request.

Attach evaluation report artifacts, canary telemetry charts, and sign-off approval to the pull request and merge to main:

gh pr merge --merge --auto

Verify: PR is merged; Git release tag ${CANDIDATE_PROMPT_VERSION} is applied to the repository.

Snippets referenced:

7. Monitoring and alerts

The thresholds below are illustrative team defaults; tune them against your measured production baseline.

Signal Source Alert when Action
GroundednessScore Azure Monitor / Application Insights Mean score drops > 0.3 on 1–5 scale vs control baseline over 24 hours Revert canary traffic to 0%; investigate failed query traces (PB-RAG-06-04).
RefusalRate Application logs / Log Analytics Increases by > 1.5x baseline rate for 1 hour Review refused queries; roll back if valid queries are falsely rejected.
UserNegativeFeedbackRate Application telemetry Exceeds 50% above baseline over 24 hours Pause rollout; review negative user feedback comments with product owner.
P95InferenceDuration APM / Platform metrics P95 latency exceeds 5000 ms for 15 minutes (team default; ) Inspect prompt token inflation and schema complexity.

8. Rollback

Situation Rollback Time
Regression caught in CI gate (Stage 2) Block PR merge; author investigates evaluation regressions on failed rows Immediate
Canary degradation observed — Path A (Stage 3) Delete percentage filter or disable PromptVersionCanary in Azure App Configuration (az appconfig feature disable --name "${APP_CONFIG_NAME}" --feature "PromptVersionCanary" --yes) Under 10 minutes
Canary degradation observed — Path B (Stage 3) Reset traffic weight to 100% on current revision via Azure CLI (az containerapp ingress traffic set --name "${CONTAINER_APP_NAME}" --resource-group "${RESOURCE_GROUP}" --revision-weight "${CURRENT_REVISION}=100") Under 10 minutes
Post-promotion degradation — Path A (Stage 4) Revert PROMPT_VERSION key in App Configuration to ${CURRENT_PROMPT_VERSION} (az appconfig kv set --name "${APP_CONFIG_NAME}" --key "PROMPT_VERSION" --value "${CURRENT_PROMPT_VERSION}" --yes) Under 10 minutes
Post-promotion degradation — Path B (Stage 4) Revert ingress traffic weight to previous revision (az containerapp ingress traffic set --name "${CONTAINER_APP_NAME}" --resource-group "${RESOURCE_GROUP}" --revision-weight "${CURRENT_REVISION}=100") Under 10 minutes

Rollback triggers (any one):

  • GroundednessScore drops by > 0.3 on 1–5 scale against baseline over 24 hours (team default).
  • RefusalRate increases by > 1.5x baseline for 1 hour (team default).
  • UserNegativeFeedbackRate exceeds 50% above baseline over 24 hours (team default).
  • P95InferenceDuration exceeds 5000 ms SLA for 15 minutes (team default; ).

9. Known issues and fixes

Symptom Likely cause Fix
Evaluation scores swing between runs of the same prompt Judge model non-determinism and small test set Run baseline and candidate in the same evaluation batch; compare relative deltas over a larger golden set.
Refusals jump after wording change Scope constraint in system prompt is broader than intended Add explicit in-scope boundary examples and recalibrate using no-answer decision guide (PB-RAG-06-02).
Citations disappear or are malformed Format rules moved or conflict with few-shot examples Keep grounding and citation rules unified in system prompt; verify schema with Pydantic (PB-RAG-06-01).
Canary looks fine but full rollout degrades Canary traffic was not representative (one tenant or geography) Assign canary routing randomly by user ID / session hash rather than tenant or geography.
Prompt token inflation causes latency spikes and throttling Lengthy prompt instructions or excessive few-shot examples inflate token counts Measure tokens using tiktoken and keep prompt template within defined token budget.
Structured outputs returns 400 BadRequest after prompt edit Unsupported JSON schema properties or incompatible model deployment Verify target model deployment supports structured outputs and validate schema constraints (PB-RAG-06-01).

10. AWS and open-source equivalents

Platform Equivalent Key differences
AWS Bedrock Prompt Management & AppConfig / Route 53 Prompt Management provides immutable numbered versions (DRAFT, 1, 2, 3) referenced by ARN. Canary traffic splitting is implemented at the application or AWS AppConfig / API Gateway layer.
Open-source (Promptfoo / Langfuse) Prompt registry and evaluation gate Store prompt templates in Git; use promptfoo eval or Langfuse prompt management for versioning and regression testing.
Open-source (DeepEval / Ragas) FaithfulnessMetric / AnswerRelevancyMetric Open-source LLM-as-a-judge evaluators that compute groundedness and relevancy locally without cloud vendor dependencies.

11. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A second engineer makes two changes to a sandbox assistant: a harmless wording refinement and a harmful change that removes the negative constraint ("answer only from sources").

Pass criteria: The evaluation gate passes the first change and cleanly blocks the second; the canary routes version-tagged telemetry; configuration rollback restores the previous prompt version in under 10 minutes.

12. Sources

Checked on 2 November 2026.