Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-3 hours
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (stale approvals act on outdated data; mass approval bypasses control)

Clear an approval queue backlog and handle timeouts

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Verified and completed against official documentation on 02 Nov 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-05). Author: Yasar · dry-run: Sanjay Sarvesh · sign-off: James. Work items: AGT-05-T09, AGT-05-T10, AGT-05-T11, AGT-05-T12.

1. Summary

This runbook guides engineers through safely clearing an approval queue backlog, escalating overdue items, and executing automated safe timeouts for paused human-in-the-loop agent workflows. It ensures that stale requests are terminated with a safe fallback rather than executed against mutated environmental state.

There are two resolution paths: Path A (automated sweeper escalation and safe timeout fallback) and Path B (targeted triage and re-routing to designated backup delegates). Path A is the default in production. Facts checked against official documentation on 02 November 2026.

2. When to use and when not to

Use this runbook when:

  • Pending human approval queues exceed agreed operational volume thresholds or SLAs.
  • Approvers are unavailable due to absence or emergency, causing critical agent workflows to stall.
  • The approval sweeper reports missed execution runs or approval notifications fail to deliver.
  • End-users complain that agent conversational sessions are paused indefinitely waiting for human input.
  • A sudden volume surge of agent tasks overwhelms human reviewer review capacity.

Do not use it when:

  • You are designing interrupt nodes and resume paths in graph code. Use PB-AGT-05-01 instead.
  • You are delivering approvals in Teams or email and handling webhook resume callbacks. Use PB-AGT-05-02 instead.
  • Paused agent threads are orphaned following container deployments or state schema changes. Use PB-AGT-05-04 instead.
  • You are auditing approval logs for regulatory compliance sign-off. Use PB-AGT-05-05 instead.

3. What you must know first

  1. Safe timeout fallback (fail-closed default). A timeout must always resolve to a safe terminal state (action: timeout resulting in no-op or cancellation), never implicit auto-approval. Never bulk-approve timed-out items without risk sign-off.
  2. State staleness hazard. An approval request paused for a long time can reference external data (such as inventory levels, pricing quotes, or permissions) that has mutated. Resuming with stale parameters causes data corruption.
  3. Prohibition of unverified bulk approval. Bulk approval of backlogged items is not allowed without risk sign-off. Bypassing human review violates organizational risk controls and creates regulatory compliance liabilities.
  4. LangGraph interrupt resume semantics. LangGraph interrupts (interrupt()) pause execution at a checkpoint. Resuming requires a pending interrupt on the thread. Resuming a thread with no pending interrupt does nothing useful, so check graph.get_state(cfg).interrupts first (it lists only interrupts still waiting for an answer).
  5. Two-tier escalation window. Production architectures enforce two distinct SLAs: a reminder age and an expiry age set by the timeout policy (agreed with risk) to re-route notifications to backup delegates and execute automatic safe timeouts.
  6. Idempotent queue sweeper design. Batch sweepers must verify that a thread's latest checkpoint state still holds active interrupts before dispatching a resume command. This prevents race conditions where an approver submits an action simultaneously with a timeout sweep.

4. Prerequisites and access

Item Detail
Roles Database/Queue Operator (read and update approval queue records) and Monitoring Reader or Log Analytics Reader (query notification and sweeper telemetry).
Tools Azure CLI (az), Python 3.10+, langgraph at the template lock-file version, no lower than 0.4.0 (first release exposing StateSnapshot.interrupts), curl, jq.
Queue access Read and write access to the shared approvals database table or queue storage service.
Checkpointer access Read and execute permissions on the LangGraph checkpointer store (e.g. Azure Database for PostgreSQL).
Approval dashboard Queue size and oldest age grouped by action and risk class.
Timeout policy Reminder age and expiry age agreed with risk and compliance.
Sweeper job Deployed and validated to execute safe timeout sweeps on demand.
Escalation matrix Approved roster of primary approvers, secondary delegates, and business escalation leads.
Change ticket Approved operational incident or maintenance ticket detailing queue clearance parameters.

Export the environment variables and acquire an authentication token for the agent service:

export APPROVALS_DB_HOST="${DB_HOST}"
export CHECKPOINTER_ENDPOINT="postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:5432/${DB_NAME}"
export AGENT_SERVICE_URL="https://${SERVICE_NAME}.azurewebsites.net"
export AGENT_SERVICE_APP_ID="api://${SERVICE_NAME}"
export TOKEN="$(az account get-access-token --resource "${AGENT_SERVICE_APP_ID}" --query accessToken --output tsv)"

5. Steps

Placeholders: ${APPROVALS_DB_HOST}, ${AGENT_SERVICE_URL}, ${EXPIRE_AFTER_HOURS}, ${REMINDER_AGE_HOURS}, ${BACKUP_APPROVER_ID}.

Stage 1: Queue assessment and root cause triage

Step 1. Inspect queue backlog volume and age distribution.

Query the approvals database to inspect pending items grouped by age bracket and risk category:

python -m scripts.check_queue \
  --db-host "${APPROVALS_DB_HOST}" \
  --group-by "risk_class,age_hours"

Verify: Output lists pending count categorized by the policy's reminder and expiry ages and identifies outlier requests.

Step 2. Identify the root cause of the approval backlog.

Audit telemetry and incident reports to determine whether the bottleneck is approver absence, delivery webhook failure, or an upstream task volume surge.

Verify: The incident log records the isolated root cause and target remediation path.

Stage 2: Escalation to backup approvers

Step 3. Re-route overdue approvals to backup delegates.

Trigger the escalation dispatcher to reassign approvals older than the agreed reminder threshold to secondary reviewer groups:

python -m scripts.escalate_queue \
  --min-age-hours "${REMINDER_AGE_HOURS}" \
  --backup-delegate "${BACKUP_APPROVER_ID}"

Verify: Queue table records updated escalated_to timestamps and secondary approvers receive notification cards.

Stage 3: Safe timeout execution via sweeper

Bulk state transition

Executing the timeout sweeper terminates paused threads with fallback outcomes. Verify that the timeout criteria comply with the agreed business policy before running.

Step 4. Execute the automated timeout sweeper.

Run the idempotent sweeper to resume expired threads with safe timeout parameters:

# Sweeper safely resumes threads past EXPIRE_AFTER with timeout
for item in pending_approvals(older_than=EXPIRE_AFTER):
    cfg = {"configurable": {"thread_id": item.thread_id}}
    state = graph.get_state(cfg)
    if state.interrupts:
        record_timeout(item.id, reason="approval_sla_expired")
        graph.invoke(Command(resume={"action": "timeout"}), cfg)

Verify: Sweeper log confirms expired threads transition to completed state with "action": "timeout".

Step 5. Notify affected end-users of expired requests.

Dispatch automated notifications to original task requesters explaining that their approval request timed out with resubmission instructions.

Verify: Notification log confirms delivery of expiry notices with corresponding correlation IDs.

Step 6. Fix notification or routing faults.

Correct the root cause identified in Step 2 (Power Automate flow failure, Teams bot delivery outage, routing rules, or missing backup approver assignments) and resend notification cards for items still pending.

Verify: New approval cards are delivered to approvers and acknowledged in flow logs.

Stage 4: Post-clearance verification and queue recovery

Step 7. Verify queue depth is restored below SLA thresholds.

Query the operational queue endpoint to confirm that all expired and escalated items have been cleared:

curl -sS -H "Authorization: Bearer ${TOKEN}" \
  "${AGENT_SERVICE_URL}/api/v1/approvals/queue/stats" | jq '.pending_count'

Verify: Response returns a pending count under the agreed operational target threshold.

Step 8. Complete the incident record and tune alert thresholds.

Document total items escalated, timed out, and processed, and update routing rules in the incident management system.

Verify: Incident ticket is closed with full resolution timestamps and audit sign-off.

6. Monitoring and alerts

The thresholds below are aligned with the catalog baseline; tune them against an agreed measured baseline in production.

Signal Source Alert when Action
Pending approvals backlog count Approval queue metric Above the agreed queue limit Initiate Stage 1 triage to identify approver bottleneck.
Age of oldest pending approval Approval queue metric Older than the reminder age Trigger Step 3 to notify designated backup delegates.
Notification delivery failure rate Webhook / Teams flow telemetry Above the agreed baseline (TO VERIFY) Inspect notification gateway and resend failed delivery cards.
Approval sweeper job failures Azure Function / Cron execution log Sweeper fails or skips scheduled execution Run sweeper manually and inspect database connection pool.
Daily timeout rate percentage Application audit log Above the agreed baseline (TO VERIFY) Review SLA windows and expand primary reviewer staffing.

7. Rollback

Situation Rollback Time
Premature timeout triggered by clock drift or sweeper misconfiguration Re-initialize the affected thread from the pre-interrupt checkpoint state and re-issue the approval request with fresh parameters. TO VERIFY in dry run
Erroneous escalation routing to wrong delegate group Revert approver assignment in the queue database table and reissue approval cards to the primary review group. TO VERIFY in dry run
Notification delivery loop spamming approvers Pause the notification dispatcher flow in Power Automate or clear transient retry queues in Service Bus. TO VERIFY in dry run
Sweeper process causes database connection exhaustion Kill the running sweeper worker process and throttle batch query chunk sizes. TO VERIFY in dry run

Rollback triggers (any one of the following):

  1. Any approval is timed out before its expiry age due to sweeper timestamp calculation errors.
  2. An approver's decision arrives for a thread that the sweeper has already timed out (detected by the approval service, not by a LangGraph error).
  3. Database query latency on the checkpointer store exceeds the agreed baseline during sweep execution (TO VERIFY in dry run).
  4. End-users receive multiple duplicated rejection notices for a single task.

8. Known issues and fixes

Symptom Likely cause Fix
Approvers report never receiving Adaptive Cards or notification emails Notification webhook failure or Power Automate flow run error Check flow run history, verify webhook endpoints, and resend cards from the admin dashboard.
Sweeper times out a request immediately after a human approver approved it Race condition between manual submission and automated sweeper run Enforce an atomic lock on the approval record and verify state.interrupts before invoking timeout.
A single approver absence stalls multiple organizational workflows Queue routing lacks round-robin or backup delegate coverage Add designated backup approvers and configure automatic delegation after the reminder age.
Approver's resume has no effect, or the approval service reports the request as already closed Thread was already resumed or timed out by the sweeper before the approver's action was processed Check graph.get_state(cfg).interrupts before submitting resume commands and enforce atomic status locking on approval records.
High database contention and locks during bulk sweeper execution Sweeper queries and updates large batches in a single unindexed transaction Paginate sweeper queries in chunks (size TO VERIFY) and index created_at and status columns.
Approved action fails on external system due to expired session token Thread remained in paused interrupt state beyond external token TTL Refresh external authorization tokens upon resuming before executing downstream tool calls.

9. AWS and open-source equivalents

Platform Equivalent Key differences
Microsoft Power Automate & Agent Framework Power Automate Approvals and Adaptive Card actions Approvals can remain pending up to 30 days before timing out automatically. Configure Timeout parameters on actions using ISO 8601 durations (e.g. P2D for 2 days; check current limits in official docs). Use flow cancellation triggers to clean up paused agent runs.
AWS Step Functions Task tokens (waitForTaskToken) and callback patterns Step Functions pauses execution until SendTaskSuccess or SendTaskFailure is called with the task token. Enforce TimeoutSeconds and HeartbeatSeconds directly on task state definitions to automate native cloud timeouts (check current limits in official docs).
LangGraph & Custom Open Source LangGraph interrupt() with PostgreSQL Checkpointer Checkpoint state is persisted in relational tables (e.g. PostgresSaver). Sweeper scripts run as scheduled cron containers, querying pending threads and dispatching Command(resume=...) payloads directly to the compiled graph.

10. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A dry-run engineer seeds a sandbox database with 20 mock approval requests across three age categories: 5 fresh items (<2 hours), 5 overdue items (>4 hours), and 10 expired items (>48 hours). The dry-runner executes the runbook: verifies backlog categorization, triggers escalation for the overdue items, executes the sweeper to expire the 10 stale items to a safe terminal state, and confirms zero unapproved items are bulk-authorized.

Pass criteria:

  • Queue assessment script categorizes all 20 seeded items into correct age brackets based on reminder and expiry policy.
  • 5 overdue items are successfully escalated to backup approvers with audit timestamps.
  • 10 expired items terminate safely with "action": "timeout" and zero exceptions.
  • Active pending backlog count drops from 20 to 10 within the maintenance window.

Record

Item Detail
Playbook PB-AGT-05-03 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

11. Sources

Checked against official documentation on 02 November 2026.