Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-4 hours around the release
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (pending approvals cannot resume and users lose work)

Resume or recover paused threads after a deploy or graph change

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Checked and verified against official sources on 10 November 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-16). Author: Deeparani · dry-run: James · sign-off: Prasanna. Work items: AGT-05-T13, AGT-05-T14, AGT-05-T15, AGT-05-T16.

1. Summary

Use this runbook before and after any release that changes the nodes, the state or the interrupt payloads of an agent while approvals are pending. A paused thread is saved at a checkpoint and waits for a person. If the new release no longer matches what was saved, the person's answer arrives and the resume fails. This runbook finds those threads before the release, picks a safe path, and recovers the ones that break.

There are three paths. Path A, deploy a compatible release, is the default. Path B, drain first, resolves the pending threads before the deploy. Path C, migrate the threads, changes their saved state after the deploy and is the riskiest. The guide covers LangGraph (the pack default) and Microsoft Agent Framework workflows. Facts checked against official documentation on 10 November 2026.

Risk: High (pending approvals cannot resume and users lose work). Typical duration: 1-4 hours around the release.

2. When to use and when not to

Use this runbook when:

  • A release renames or removes a node, or changes the number or order of interrupt() calls in a node.
  • A release changes the state schema or the payload that an interrupt returns.
  • For Microsoft Agent Framework, a release changes the workflow topology or an executor identity, including an agent's Id or Name.
  • You ship any change while approvals are pending, and you want proof that they still resume. Run it before the release, and again after.
  • Resume errors appear after a release.

Do not use it when:

  • You are building the approve, edit, reject and timeout paths. Use PB-AGT-05-01.
  • You are delivering approvals through Teams or email. Use PB-AGT-05-02.
  • You need to clear an approval backlog or handle timeouts. Use PB-AGT-05-03.
  • An agent acted outside its permission envelope. Use PB-AGT-08-04.
  • The checkpoint store or a tool is down. Use PB-AGT-09-02.

3. What you must know first

  1. A saved thread records which node runs next. graph.get_state(config) returns a StateSnapshot. Its next field lists the "node names to execute next". To continue, you invoke the graph with Command(resume=...) and the same thread ID. The node then restarts from its beginning. Reusing a thread ID alone does not resume anything. Sources: LangGraph: Checkpointers; LangGraph: Interrupts.
  2. A renamed or removed node is the main risk. The saved next still names the old node. The LangGraph pages do not describe what happens when the graph code changes under a saved thread, so this is the team's reasoning from point 1, and the dry-run must prove it.
  3. Interrupts are matched by position. The docs say matching is "strictly index-based", so the order of interrupt() calls in a node matters. They also say not to skip a call conditionally, not to loop calls with non-deterministic logic, and not to wrap them in try/except. The payload must be JSON-serialisable. So a release must not add, remove or reorder interrupts before an existing one in the same node. Put a new interrupt in a new node (team guidance). Source: LangGraph: Interrupts.
  4. The checkpointer must be durable, and its serialiser has limits. InMemorySaver keeps checkpoints in RAM, so a restart loses them. PostgresSaver is described as ideal for production. The default serialiser (JsonPlusSerializer) "is not suitable for all types of objects". A release that changes state types may therefore break the loading of old checkpoints. Sources: LangGraph: Checkpointers; LangGraph: Durable execution.
  5. Recovery tools exist, and they add checkpoints. get_state_history lists a thread's snapshots, newest first. update_state "creates a new checkpoint with the updated values" and does not modify the original. Values pass through reducers. Invoking with an earlier checkpoint_id re-runs the nodes after that checkpoint, so side effects can run again. Source: LangGraph: Checkpointers.
  6. Agent Framework rehydration needs the same topology and identities. Checkpoints are created at the end of each superstep. They capture the executor states, the pending messages, the pending requests and responses, and the shared state. A rehydrated workflow "must preserve the topology and executor identities" of the one that made the checkpoint. In .NET, local agents need a stable ChatClientAgentOptions.Id, and a stable Name if one is set. Changing either makes resume fail. Stable values do not repair checkpoints made with different IDs. You must start a new session and checkpoint lineage. Source: Microsoft Agent Framework: Workflow checkpoints.
  7. Checkpoint storage is a trust boundary. The Python checkpoint stores use a restricted unpickler. A custom application type must be allowed (allowed_checkpoint_types, in module:qualname form), or loading fails with WorkflowCheckpointException. Never load checkpoints from an untrusted source. The page does not state a preview status for checkpoints, and the Cosmos DB provider package installs with --pre. Source: Microsoft Agent Framework: Workflow checkpoints.

4. Prerequisites and access

Item Detail
List of pending threads A list of the threads with pending interrupts. The LangGraph pages do not show an API that lists all threads, so the snippet uses the team's own registry.
Checkpoint store Read access to the production checkpoint database, and a protected staging copy. Checkpoints hold user data, so treat the copy as production data (team guidance).
New release The new release running in staging, built with the production checkpointer type.
Tools Python with langgraph and langgraph-checkpoint-postgres (pin the tested versions). For Agent Framework, the SDK and its checkpoint storage package.
Rollback plan The previous release kept deployable, and a way to route a paused thread to the release that created it (section 5, Path A).
Change approval A change ticket with the release window, the rollback owner and the sign-off owner.
People The approvers' contact list, so you can tell them if a pending approval must be repeated. 
export CHECKPOINT_DB_URL="postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:5432/${DB_NAME}?sslmode=require"
export PENDING_THREADS_QUERY="${REGISTRY_QUERY_FOR_PENDING_THREADS}"
export RELEASE_OLD="${CURRENT_RELEASE_TAG}"
export RELEASE_NEW="${NEW_RELEASE_TAG}"

The password form is for the sandbox only. Use Microsoft Entra ID token authentication in production, as in PB-AGT-08-02.

5. Decide your approach

Change in the release Path Notes
No change to node names, state types, interrupt order or payloads A: Deploy as it is Still run the check and resume a sample.
A node is renamed or removed A with an alias node, or B Keep the old name as an alias node that routes to the new one, or drain first.
An interrupt is added, removed or reordered in a node that has pending threads B Matching is by position, so existing pending threads would get the wrong value. Or move the new interrupt to a new node and use A.
The state schema or a state type changes B, or C A migration with update_state can map old state to new. Test it on the staging copy first.
Agent Framework: an executor ID, an agent Id or Name, or the topology changes B The checkpoint cannot be repaired. Drain, or start a new session.
The check lists blocked threads and you cannot wait C, with sign-off The riskiest path. Rollback is described in section 8.
Path A: Compatible release Path B: Drain first Path C: Migrate threads
Downtime for approvals None The wait while threads resolve None, but threads change under people
Rollback Redeploy the old release for old threads Redeploy the old release The earlier checkpoint remains, but a replay can repeat side effects
Effort Low Medium (timing and reminders) High (migration code and tests)
Risk Low if the check is clean Low High

6. Steps

Placeholders: ${PLACEHOLDER} (what it is, for example ${THREAD_ID} is one paused thread). Never paste connection strings or tokens into tickets or chat. Record the time of each step.

Stage 1: Before the release

Step 1. List the threads with pending interrupts.

psql "${CHECKPOINT_DB_URL}" -c "${PENDING_THREADS_QUERY}" -o pending-threads-before.txt

Verify: the file exists, and its count matches the number of pending approvals your approval queue shows.

Step 2. Back up the checkpoint store and restore it into staging.

Checkpoints contain user data

A copy has the same sensitivity as production. Put it only in a protected staging store with production-level access control, and delete it after the release. Undo: delete the staging copy.

Verify: the restored staging copy lists the same pending threads as step 1.

Step 3. Run the compatibility check against the new graph. Use snippets/ops/check_pending_threads.py. It loads each pending thread's state with the new release's graph and checks that every node in next still exists. Also read the release diff for any change to the number or order of interrupt() calls (point 3 in section 3).

Verify: the output lists the threads that would not resume, with the missing nodes. An empty list and an unchanged interrupt order mean Path A is safe.

Step 4. Choose the path and record the decision. Use section 5. If the check lists blocked threads, add an alias node, drain the threads, or approve Path C. Write the choice, the approver and the time in the change ticket.

Verify: the ticket names the path and the sign-off owner.

Stage 2: Deploy and test resume

Step 5. Deploy the release.

A bad release strands pending approvals

If threads cannot resume, people lose work. Deploy in the agreed window and keep the previous release ready. To undo, follow section 8.

Verify: the new release is healthy, and the pending count equals the count from step 1.

Step 6. Resume a sample of paused threads, first in staging and then in production.

from langgraph.types import Command

config = {"configurable": {"thread_id": "${THREAD_ID}"}}
result = graph.invoke(Command(resume="${DECISION}"), config=config)

Verify: each sample thread resumes with the same thread ID, the interrupt returns the expected value, and no unknown-node or checkpoint error appears.

Stage 3: Watch and recover

Step 7. Watch resume errors for 24 hours (team default). Check the alerts in section 7. Pending threads that are older than their expiry go to the sweeper in PB-AGT-05-03.

Verify: no resume errors, and no pending thread is older than its expiry.

Step 8. Recover any thread that fails. Choose one:

  1. Redeploy the previous release for that thread, if it still matches its checkpoint.
  2. For LangGraph, migrate the saved state with update_state on the staging copy first, then on production, then resume. This creates a new checkpoint. The original remains.
  3. For Agent Framework with changed IDs, start a new session. The old checkpoint cannot be repaired.

Migration changes saved state

update_state passes values through reducers, so values can accumulate instead of being replaced. Test on the staging copy first. Undo: resume from the earlier checkpoint with the old release. Nodes after it run again, so check for side effects.

Verify: every thread from step 1 has resumed, been drained or been closed with a reason. In the dry-run, all five sandbox threads resume.

Stage 4: Close

Step 9. Record the result and update the check. Add the HITL section to the release note (snippets/docs/releases/hitl-section.md). Add a case to the check for the change that caused the failure.

Verify: the release note is saved, and the new case fails on the old graph and passes on the fixed one.

Draft snippet: Check that paused threads can resume on the new graph

# pseudo
from agent_app.graph import build_graph  # the NEW release's builder
from agent_app.registry import threads_with_pending_interrupts  # your own helper

graph = build_graph()  # compiled with the production checkpointer
known_nodes = set(graph.get_graph().nodes)
blocked = []
for thread_id in threads_with_pending_interrupts():
    snap = graph.get_state({"configurable": {"thread_id": thread_id}})
    missing = [n for n in snap.next if n not in known_nodes]
    if missing:
        blocked.append((thread_id, missing))
print(f"{len(blocked)} threads would not resume on the new graph")
for thread_id, missing in blocked[:20]:
    print(thread_id, missing)

threads_with_pending_interrupts is the team's own helper, not a LangGraph function. The get_graph().nodes call is unverified.

Snippets to add:

  • snippets/ops/check_pending_threads.py: Pre-release compatibility check (python)
  • snippets/docs/releases/hitl-section.md: Release note section for HITL (markdown)

7. Monitoring and alerts

The thresholds are team defaults, not platform recommendations. Tune them. The signals marked custom are metrics or queries that your own code must provide.

Signal Alert when Action
Threads blocked by the pre-release check (custom) Any blocked thread before the release Do not deploy. Choose a path (section 5).
Resume errors (custom, from the application logs) Any after a release Pause the rollout and investigate.
Pending threads older than their expiry (custom) Any after the release Run the sweeper in PB-AGT-05-03.
Checkpoint load failures such as WorkflowCheckpointException (custom, Agent Framework) Any after a release Check the allowed types and the executor identities (section 3, points 6 and 7).

8. Rollback

Situation Rollback Time
Before the deploy Postpone the release. Nothing changed for the pending threads. Immediate
After the deploy, resume errors on the new release Redeploy the previous release for the affected threads. Route each paused thread to the release that created it. Store a release tag with each paused thread. Depends on the pipeline
After a thread was migrated with update_state Resume from the earlier checkpoint with the old release. Nodes after it run again, so check for side effects. Minutes per thread
Agent Framework checkpoint with changed IDs None. Stable IDs do not repair it. Start a new session and tell the approver Not applicable
A resumed thread already took its action No rollback. Compensate where possible (PB-AGT-09-03) Varies

Rollback trigger (any one): an unknown-node error, a wrong value returned to an interrupt, a checkpoint load failure, or resume errors above the team default in section 7.

9. Known issues and fixes

Symptom Likely cause Fix
Resume fails with an unknown node A node was renamed or removed Keep the old name as an alias node, or drain the threads first
Wrong value returned to interrupt() The order or number of interrupts changed in the node Keep the order stable. Add new interrupts in new nodes. Interrupts are matched by position
Nothing resumes although the thread ID is correct The graph was not invoked with Command(resume=...) Reusing a thread ID alone does not resume execution. Invoke with the resume command
Side effects run twice after a resume The node restarts from its beginning Put side effects after the interrupt and make them safe to repeat (PB-AGT-08-02)
Workflow fails to restore a checkpoint An executor ID, an agent Id or Name, or the topology changed Keep them stable across releases. If already changed, start a new session
WorkflowCheckpointException when loading (Python) A state type is not allowed by the restricted unpickler Add the type through allowed_checkpoint_types on the store before loading
Checkpoints disappear after a restart An in-memory checkpointer was used Use a durable checkpointer such as PostgresSaver
Saves fail for some threads with PostgresSaver The thread ID is too long The docs say to keep thread IDs under 255 characters

10. Cost and capacity notes

  • Staging copy. A copy of the checkpoint store needs storage and protection until it is deleted. Check the current price before approval.
  • Running two releases. Path A with routing, or Path B while draining, runs the old and new releases at once.
  • Waiting. Path B waits for people to answer, so plan the release window around the approvers.

11. AWS and open-source equivalents

Platform Equivalent Key differences
AWS Step Functions State machine versions and aliases A version is an immutable snapshot, so publish a version and start executions from a version or an alias. Executions started without a version use the most recent revision.
OpenAI Agents SDK RunState saved with to_json() or to_string() The docs advise storing a version marker with each long-pending approval, so you can route loading to matching code when models, prompts or tool definitions change. Load only from trusted storage.
Microsoft Agent Framework Workflow checkpoints The same idea as LangGraph, with the identity rules in section 3, point 6.

12. Checklist

Pre-flight

  • Pending threads listed and counted (step 1).
  • Checkpoint store backed up and restored into protected staging (step 2).
  • Compatibility check run on the new graph, and the interrupt order reviewed (step 3).
  • Path chosen and signed off in the change ticket (step 4).
  • Previous release ready, with a release tag stored on each paused thread (section 8).

Post-change

  • Sample threads resumed in staging, then in production (step 6).
  • No resume errors for 24 hours (step 7).
  • Every thread from step 1 resumed, drained or closed with a reason (step 8).
  • Release note and check updated (step 9).

13. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: a second engineer pauses five threads in the sandbox, deploys a release that renames a node, detects the break with the check, fixes it and resumes all five. The run includes one deliberate rollback to the previous release.

Pass criteria: the check lists the five threads before the deploy; after the fix, all five resume with the same thread IDs and the expected interrupt values; the rollback returns the threads to a resumable state.

Record

Item Detail
Playbook PB-AGT-05-04 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

14. Sources

Checked on 10 November 2026.

  • PB-AGT-05-01: Build approve, edit, reject and timeout paths (builds the paused threads this runbook protects).
  • PB-AGT-05-02: Deliver approvals in Teams or email and resume the graph.
  • PB-AGT-05-03: Clear an approval queue backlog and handle timeouts (the sweeper).
  • PB-AGT-05-05: HITL audit trail and compliance sign-off checklist.
  • PB-AGT-08-02: Enforce action allow-lists and approval gates (the same interrupt rules).
  • PB-AGT-09-03: Build compensating actions and checkpoint resume (reversing actions).