Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-3 days per template including evaluation
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (silent behaviour change across every agent built on the template)

Upgrade the agent framework or model version without regressions

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-20). Author: James · dry-run: Sivaraj · sign-off: Prasanna. Work items: AGT-01-T09, AGT-01-T10, AGT-01-T11, AGT-01-T12.

1. Summary

When upgrading LangGraph, LangChain or Agent Framework, moving from Semantic Kernel, or switching a model deployment. Also when a security patch forces an upgrade.

Risk: High (silent behaviour change across every agent built on the template). Typical duration: 1-3 days per template including evaluation.

2. When to use and when not to

Use it when: When upgrading LangGraph, LangChain or Agent Framework, moving from Semantic Kernel, or switching a model deployment. Also when a security patch forces an upgrade.

Do not use it when: to be written by the author.

3. What you must know first

  • Framework upgrades can change parameter names and state handling
  • Existing checkpoints must still load after the upgrade
  • Offline evaluation on a fixed dataset is the regression gate
  • Model upgrades change tool-calling behaviour even with the same prompt
  • Rollback means previous image plus previous prompt and model deployment

4. Prerequisites and access

  • Pinned dependency lock file
  • Regression dataset and baseline evaluation results
  • Sandbox with a copy of production checkpoints
  • Release notes and migration guides read
  • Change approval from the agent lead

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Read the release notes and list breaking changes that touch the template
  2. Create a branch, bump versions in the lock file and fix compile and test failures
  3. Load a copy of production checkpoints and resume sample threads on the new version
  4. Run the offline evaluation and compare with the baseline
  5. Review trace diffs for tool-call count and token use per task
  6. Canary to one tenant or 10 percent of traffic for 48 hours
  7. Watch error rate, loop alerts and spend during the canary
  8. Promote or roll back to the previous image and deployment
  9. Update the baseline and the template's version table

Draft snippet: Azure Pipelines gate that blocks upgrades on quality drop

# azure-pipelines/agent-upgrade-gate.yml
trigger: none
pr:
  branches:
    include:
      - main
pool:
  vmImage: ubuntu-latest
variables:
  EVAL_DATASET: agent-regression-v3
steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: "3.12"
  - script: pip install -r requirements.lock
    displayName: Install pinned dependencies
  - script: pytest tests/agent_contract -q
    displayName: Contract tests (state schema, tools, termination)
  - script: python scripts/run_eval.py --dataset "$(EVAL_DATASET)" --out evals/candidate.json
    displayName: Run offline evaluation
  - script: python scripts/compare_eval.py --baseline evals/baseline.json --candidate evals/candidate.json --max-drop 0.02
    displayName: Fail the PR if quality drops

Snippets to add:

  • snippets/azure-pipelines/agent-upgrade-gate.yml: PR gate pipeline for agent upgrades (yaml)
  • snippets/scripts/compare_eval.py: Compare evaluation results (python)
  • snippets/docs/upgrades/TEMPLATE.md: Upgrade record template (markdown)

6. Monitoring and alerts

Signal Alert when Action
Evaluation score drop in CI More than 2 points below baseline Block merge and investigate
Error rate of canary versus control Canary error rate above control by 1 percentage point Stop the canary and roll back
Resume failures on existing threads Any Pause rollout and run PB-AGT-05-04

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Old threads fail to resume after upgrade Checkpoint format or state schema changed Drain or migrate open threads before cut-over; keep the old version for them
Tests pass but quality drops Model now calls tools in a different order Tighten tool descriptions and add the failing cases to the dataset
Import errors after upgrade Package split or renamed modules Follow the official migration guide and pin all sub-packages
Latency rises after model switch New model or deployment has different throughput limits Check deployment quotas and adjust concurrency

9. AWS and open-source equivalents

On AWS use CodePipeline or GitHub Actions with the same gate and AgentCore Runtime versions for canary. LangSmith experiments can replace the custom compare script for LangGraph agents.

10. Validation record

Pending. Planned dry-run scenario: A second engineer upgrades one template by a minor version in a sandbox, resumes ten copied threads, runs the gate and demonstrates both a pass and a forced fail; then performs a rollback in under 30 minutes.

11. Sources