Skip to content
⚠️ Before You Start
Risk: High
Duration: 30-60 minutes hands-on to stabilise, then the length of the outage
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (blind retries multiply load on the failing dependency and duplicate side effects)

Handle a tool or model outage during agent runs

Draft: not validated

Drafted with AI from official sources on 16 Oct 2026. Outline corrected against official sources on 03 Nov 2026 (AGT-09-T05). Written (AGT-09-T06); snippets not yet run in the sandbox. Do not use on a live system until it is written, dry-run by someone other than the author, and signed off (planned 2026-11-30). Author: Sandeep Velu · dry-run: Yasar · sign-off: Sivaraj. Work items: AGT-09-T05, AGT-09-T06, AGT-09-T07, AGT-09-T08.

1. Summary

Use this runbook when a tool API, an MCP server or a model deployment fails, throttles or times out during live LangGraph agent runs. It isolates the failing dependency with a circuit breaker, parks affected runs on an Azure Service Bus queue and tells users the run is paused. When the dependency recovers, it resumes runs in small batches from their checkpoints without duplicating writes.

There are three paths (Section 5):

  • Path A, tool outage: open the tool breaker, park runs, resume from checkpoints after recovery. This is the default for tool and MCP outages.
  • Path B, gated model failover: let the APIM backend pool use a fallback deployment, only if it passed the regression gate and the agent's permission envelope allows it.
  • Path C, park all: when no fallback is allowed or every model region is down, open the model breaker, park runs and notify users.

2. When to use and when not to

Use it when:

  • A tool API or MCP server returns repeated 5xx errors, connection resets or timeouts.
  • A model deployment returns 503, or 429 from the model service itself, during live runs.
  • Runs fail mid-workflow and their state must be kept and parked, not dropped.

Do not use it when:

  • The 429 comes from the APIM llm-token-limit policy. That is a tenant budget limit, not an outage. Use PB-AGT-11-02.
  • An agent loops or burns tokens while its dependencies are healthy. Use PB-AGT-09-01.
  • You need to build checkpointing or compensating actions into the agent. Use PB-AGT-09-03.
  • The outage is over and messages sit in the dead-letter queue. Use PB-AGT-09-04.
  • You need to fix the tool side (quotas, backoff contract, client settings). Use PB-AGT-02-05. This runbook covers what happens to live runs.
  • Calls fail with 401 or 403 because of expired credentials or removed roles. Fix identity first.

3. What you must know first

  1. Retry only idempotent calls. Reads can be retried automatically. A write needs an idempotency key, or no retry at all. Use the template's key ${THREAD_ID}:${STEP_NUMBER} (PB-AGT-01-02), sent in the header or field the target API documents, and record it on the call's span as idempotency_key.
  2. Respect 429 and Retry-After. Back off for the time the response asks. An Azure OpenAI 429 can carry a very long Retry-After (the APIM docs give one day as an example). APIM backend circuit breakers can accept it with acceptRetryAfter.
  3. Classify the failure before acting. A timeout or 5xx is an access failure that needs a retry decision. A successful call that returns no results is a valid answer, not an error. A 429 from llm-token-limit is a budget limit (Section 2).
  4. Return a structured error, never a fake success. When a tool fails, return its failure type, the request, whether it can be retried, any partial results and possible alternatives to the model or supervisor (tool_error in snippets/agent/resilience.py; the tool error contract is in PB-AGT-02-01). Do not return an empty result marked as success.
  5. Recover at the lowest level that can. The tool client retries a transient fault on an idempotent read once, honouring Retry-After. Anything it cannot fix goes to the model or supervisor as a structured error, with the steps it already tried. Only an open breaker parks the run.
  6. What a resume repeats. LangGraph saves a checkpoint at each super-step. When you resume a thread, nodes that succeeded in the interrupted super-step are not re-run, but the node that failed runs again from its start. Writes inside that node repeat unless they carry an idempotency key.
  7. Circuit breaker states. Closed (calls pass), Open (calls fail fast and runs park) and Half-Open (one probe call is allowed). The breaker in snippets/agent/resilience.py keeps state per process, so an App Configuration feature flag, tool_<name>_force_open, is the shared control across replicas. Enabling the flag sets the breaker's own state to Open, so disabling it moves the breaker to Half-Open, not straight to Closed. This flag is not PB-AGT-02-05's tool_<name>_enabled flag: that one removes the tool from the agent and answers with a degraded message, while this one keeps the tool and parks runs. Do not use both for the same outage.
  8. APIM backend circuit breakers and pools. Circuit breaker rules belong to individual backends, one rule per backend. A pool holds backends with priorities, and APIM uses lower-priority backends only when every higher-priority backend has tripped. When a backend trips and no other is available, APIM returns 503. Gateway instances do not share breaker state, so tripping is approximate.
  9. Service Bus delivery. Each delivery increments DeliveryCount. After MaxDeliveryCount deliveries (default 10) the message moves to <queue>/$deadletterqueue. The default lock duration is 1 minute (maximum 5); renew the lock for longer steps.
  10. Tell the user. When a run parks, send a message such as "Run paused: waiting for an external service". Silent waiting causes repeat submissions.

4. Prerequisites and access

Item Detail
Roles Azure Service Bus Data Sender and Azure Service Bus Data Receiver on the parked-runs queue. API Management Service Contributor for backend changes. Monitoring Reader for Application Insights queries. Use Microsoft Entra ID.
Tools Azure CLI with the application-insights extension, Python 3.10 or later, curl, jq.
Agent code A CircuitBreaker and timeout per tool from snippets/agent/resilience.py, with settings per tool in snippets/agent/tool-resilience.json. A feature flag tool_<name>_force_open per tool in App Configuration (created disabled with az appconfig feature set), read through the App Configuration provider with feature-flag refresh enabled. Writes record idempotency_key on their spans. configure_azure_monitor() at start-up and otel_state_listener() on each breaker, so the agent.circuit_breaker.open metric reaches Application Insights.
Infrastructure Service Bus queue ${PARKED_QUEUE_NAME} with dead-lettering. A durable LangGraph checkpointer, for example PostgresSaver on Azure Database for PostgreSQL.
Fallback model A fallback deployment that passed the regression gate in PB-AGT-07-02 and is allowed by the agent's envelope in PB-AGT-08-01. Without both, there is no fallback: runs park.
Contacts Status page or owner contact for each tool.
Change approval An incident ticket with an incident lead and a rollback owner.

Placeholders: ${RESOURCE_GROUP}, ${SUBSCRIPTION_ID}, ${APPINSIGHTS_NAME}, ${APP_CONFIG_NAME}, ${SERVICEBUS_NAMESPACE}, ${PARKED_QUEUE_NAME}, ${APIM_NAME}, ${PRIMARY_BACKEND_ID}, ${POOL_BACKEND_ID}, ${FAILING_TOOL}, ${OUTAGE_START_ISO}, ${PG_CONN_STRING}, ${SERVICEBUS_FQDN}. Never paste credentials or keys into scripts, tickets or chat.

5. Decide your approach

Outage Path Key mechanism
Tool API or MCP server down Path A: break and park Open the tool breaker, park runs, resume after recovery.
One model deployment throttled or down, gated fallback available Path B: gated failover Confirm the gate and envelope, then let the APIM pool use the fallback backend.
No allowed fallback, or every model region down Path C: park all Open the model breaker (model_chat in tool-resilience.json), park all runs that need the model, notify users.

If only some sources fail and the task can still be answered, do not park the run. Let it finish and add coverage notes to the answer that name the missing sources and the conclusions they weaken.

6. Steps

Stage 1: Detect and isolate

Step 1. Identify the failing dependency.

az monitor app-insights query --app "${APPINSIGHTS_NAME}" -g "${RESOURCE_GROUP}" --offset 15m \
  --analytics-query "dependencies | where success == false | summarize Failures=count() by target, resultCode | order by Failures desc" \
  -o table

Verify: the output names the failing target and its result codes. Check the tool's status page. If the code is 429, check whether it came from llm-token-limit (Section 2) before going on.

Step 2. Force the tool's circuit breaker open.

The breaker opens by itself after its failure threshold (team default: 5 failures, 60-second cool-off, the same as PB-AGT-02-05; set per tool in tool-resilience.json). Force it open now so every replica stops calling the tool:

az appconfig feature enable --name "${APP_CONFIG_NAME}" \
  --feature "tool_${FAILING_TOOL}_force_open" --yes

Verify: az appconfig feature show --name "${APP_CONFIG_NAME}" --feature "tool_${FAILING_TOOL}_force_open" shows the flag enabled. After the provider's refresh interval, agent.circuit_breaker.open is 1 for the tool and the dependency receives no calls from the agent.

Stage 2: Park and inform

Step 3. Confirm affected runs are parked and users are told.

az servicebus queue show -g "${RESOURCE_GROUP}" --namespace-name "${SERVICEBUS_NAMESPACE}" \
  --name "${PARKED_QUEUE_NAME}" \
  --query "countDetails.{Active:activeMessageCount,DeadLetter:deadLetterMessageCount}" -o table

Verify: Active grows as runs park and DeadLetter is 0. Open one parked run in the chat channel and confirm the user saw the pause message. In its trace, confirm the last tool result is a structured error with is_error set and an error_type, not an empty success.

Stage 3: Model outages only

Step 4. Check the primary backend's circuit breaker and the pool's priorities.

SVC="https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.ApiManagement/service/${APIM_NAME}"
az rest --method get --url "${SVC}/backends/${PRIMARY_BACKEND_ID}?api-version=2024-05-01" --query "properties.circuitBreaker"
az rest --method get --url "${SVC}/backends/${POOL_BACKEND_ID}?api-version=2024-05-01" --query "properties.pool.services"

Verify: circuitBreaker.rules on the primary backend has a failureCondition whose statusCodeRanges cover 429 and 5xx, with acceptRetryAfter set to true. In the pool's services, the fallback backend has a lower priority than the primary.

Changes cost and agent behaviour for every tenant

Failover sends live traffic to a different model deployment. It can change tool-calling behaviour and cost. Allow it only if the fallback passed the regression gate (PB-AGT-07-02) and the permission envelope allows it (PB-AGT-08-01). To undo, see Section 8.

Step 5. Allow failover only for a gated fallback.

Check two records:

  • Gate: the latest run of the agent regression gate CI job from PB-AGT-07-02 for the fallback deployment ended with Regression gate passed, and any expected change has agent lead sign-off in the change ticket.
  • Envelope: the agent's signed-off policy file from PB-AGT-08-01 is the version running in production, and its spend limits (maxTokensPerRun, maxStepsPerRun) still hold at the fallback deployment's token use.

If both checks pass, let the pool fail over. If either fails, follow Path C: run Step 2 with FAILING_TOOL=model_chat to force the model breaker open, and park runs.

Verify: with failover, requests succeed through the fallback backend and tool-call errors stay at baseline. With Path C, runs that need the model are parked and users see the pause message.

Stage 4: Watch

Step 6. Track queue depth and dead-letter count.

Run the Step 3 query every 10 minutes during the outage.

Verify: DeadLetter stays 0. If it rises, stop the resume worker (Section 8) and check the reason with PB-AGT-09-04.

Stage 5: Probe recovery

Step 7. Send one probe call to the tool.

When the tool owner reports recovery, call the tool's read-only probe_url from tool-resilience.json once, outside the agent:

PROBE_URL="$(jq -r --arg t "${FAILING_TOOL}" '.tools[$t].probe_url' snippets/agent/tool-resilience.json)"
curl -sS -o /dev/null -w "%{http_code} %{time_total}\n" --max-time 10 "${PROBE_URL}"

Verify: the probe returns 200 within the tool's timeout_seconds.

Step 8. Remove the override.

az appconfig feature disable --name "${APP_CONFIG_NAME}" \
  --feature "tool_${FAILING_TOOL}_force_open" --yes

Verify: after the provider's refresh interval, the breaker moves to Half-Open, lets one call through, and closes after that call succeeds. agent.circuit_breaker.open returns to 0 for the tool.

Stage 6: Resume parked runs

Resuming too fast can bring the tool down again

Resuming all parked runs at once can overload a recovering tool and repeat writes. Resume in small batches. If errors rise, follow Section 8.

Step 9. Resume parked runs in small batches.

Run resume_parked_runs from snippets/agent/resilience.py. It resumes each parked thread_id from its checkpoint, completes the message only after the run resumes, and stops at the first failure. Run one batch first:

from langgraph.checkpoint.postgres import PostgresSaver

from agent_app.graph import builder  # the template's uncompiled StateGraph, as in PB-AGT-09-01
from resilience import resume_parked_runs

with PostgresSaver.from_conn_string("${PG_CONN_STRING}") as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)
    resumed = resume_parked_runs(graph, "${SERVICEBUS_FQDN}", "${PARKED_QUEUE_NAME}",
                                 batch_size=5, delay_seconds=2.0, max_batches=1)
    print(f"resumed={resumed}")

Verify: the first batch prints resumed=5 (or fewer if fewer were parked), the tool's error rate stays at baseline, and Active falls. Then run again without max_batches to drain the rest.

Step 10. Confirm the queue is drained.

Verify: the Step 3 query shows Active 0 and DeadLetter 0, and the agent's error rate is back to its baseline.

Step 11. Check for duplicate writes.

Find idempotency keys that were sent successfully more than once since the outage began:

dependencies
| where timestamp > datetime(${OUTAGE_START_ISO})
| extend key = tostring(customDimensions["idempotency_key"])
| where isnotempty(key) and success == true
| summarize calls = count() by key, target
| where calls > 1

Verify: the query returns no rows. For each row it does return, look up the key in the target system. If the target honoured the key, the second call returned the first result and there is no duplicate. Record any real duplicates and their fix in the incident ticket.

7. Monitoring and alerts

The thresholds below are team defaults, not platform recommendations. Tune them to measured baselines.

Signal Source Alert when Action
dependencies/failed, timeouts and 5xx only Application Insights Above 5% for 5 minutes on any tool Start this runbook at Step 1
ActiveMessages Service Bus platform metric Growing on ${PARKED_QUEUE_NAME} for 30 minutes Check the resume worker and dependency status
DeadletteredMessages Service Bus platform metric Above 0 on ${PARKED_QUEUE_NAME} Stop the resume worker; use PB-AGT-09-04
ServerErrors Service Bus platform metric Above 5 in 5 minutes Check Service Bus namespace health
agent.circuit_breaker.open Custom metric (OpenTelemetry UpDownCounter, tool attribute) Above 0 for one tool for more than 15 minutes Notify the tool owner and update users

Count only access failures (timeouts, 5xx, connection errors) in the tool error alert. A successful call with no results is not a failure.

otel_state_listener() in snippets/agent/resilience.py adds 1 when a breaker opens and subtracts 1 when it leaves Open. Each replica reports its own breaker, so the sum per tool is the number of replicas with that breaker open. Azure Monitor shows an UpDownCounter with the Sum aggregation.

8. Rollback

Situation Rollback
Probe in Step 7 fails Leave the tool_<name>_force_open flag enabled. Keep runs parked and probe again later.
Breaker reopens after Step 8 Enable the flag again (Step 2). Stop the resume worker.
Resumed batches raise errors or 429s The worker stops at the first failure. Enable the flag again (Step 2). Restart later with a smaller batch_size and a longer delay_seconds.
Fallback model gives wrong tool calls Remove the fallback backend from the pool through the pipeline, or force the model breaker open (Path C). Park runs.
Messages near MaxDeliveryCount Stop the resume worker, fix the cause, then continue. Dead-lettered messages go to PB-AGT-09-04.

Roll back when any of these happen:

  • The probe fails or times out.
  • The tool returns 429 or 5xx on more than 2% of resumed calls (team default).
  • DeadletteredMessages rises above 0.

Verify after rollback: the breaker refuses calls to the tool, new runs park, and no new dead-lettered messages appear.

9. Known issues and fixes

Symptom Likely cause Fix
Runs hang for minutes, then fail No timeout on the tool call Set a per-tool timeout below the user-facing timeout (snippets/agent/tool-resilience.json)
Duplicate records or emails after resume Write retried, or failed node re-run, without an idempotency key Add idempotency keys; remove automatic retries on writes
Resumed run starts from the beginning Graph invoked with new input instead of None on the same thread_id Resume with resume_run on the existing thread
Resume raises EmptyInputError Thread was never checkpointed durably Check the checkpointer and durability mode; replay the request as a new run
Messages dead-lettered with MaxDeliveryCountExceeded Worker crashed or the lock expired before settlement Settle within the lock duration; renew locks for long steps
Fallback model breaks tool calling Fallback never ran the regression gate Run PB-AGT-07-02 on the fallback before adding it to the pool
Supervisor keeps retrying a source that returned no results Empty result reported as an error Return an empty success; keep errors for access failures
Failover never happens Pool backends share one priority, or the primary has no breaker rule Set priorities and a breaker rule per backend
Long outage after a 429 Retry-After was very long and the breaker accepted it Expected behaviour; park runs and tell users

10. Cost and capacity notes

  • Parking and resuming runs adds Service Bus operations. Check the current price before approving high-frequency polling.
  • Failover to another deployment or region changes token cost. Check the current price before approving failover for long outages.

11. AWS and open-source equivalents

Platform Isolate and park Resume and model fallback
AWS Amazon SQS with a dead-letter queue (redrive policy, maxReceiveCount). The force-open flag as an AWS AppConfig feature flag, read through AWS AppConfig Agent. Step Functions Retry and Catch for orchestrated tool calls. Workers resume from the checkpointer. Amazon Bedrock cross-region inference for model routing.
Open source RabbitMQ or Kafka for parking. pybreaker for the breaker (fail_max, reset_timeout, a CircuitBreakerListener for state changes, and CircuitRedisStorage to share state across replicas). tenacity for retries with backoff on idempotent calls. LangGraph with PostgresSaver. A gateway such as LiteLLM for model fallback.

12. Checklist

Pre-flight

  • Failure classified: access failure, model throttling, or budget limit (Step 1).
  • Tool breaker forced open (Step 2).
  • Runs parked, dead-letter count 0, users see the pause message (Step 3).
  • For model outages, failover only to a gated fallback (Steps 4 and 5).

Post-recovery

  • Probe succeeded (Step 7) and override removed (Step 8).
  • Runs resumed in batches (Step 9) and queue drained (Step 10).
  • Duplicate writes checked (Step 11).

13. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: the dry-runner deploys a LangGraph agent wired to a mock tool and a Service Bus queue. The mock tool returns HTTP 500 for 10 minutes during a 50-run scripted load. The dry-runner follows Path A, including one deliberate rollback when the first probe is sent before the mock recovers.

Pass criteria:

  • The breaker opens and runs park with a user-facing message.
  • No duplicate writes occur.
  • No messages are dead-lettered.
  • All 50 parked runs complete within 15 minutes of the tool recovering.
  • The early probe fails and the rollback keeps runs parked.

Record

Item Detail
Playbook PB-AGT-09-02 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

14. Sources

  • PB-AGT-09-01: Stop and triage an agent stuck in a loop or burning tokens.
  • PB-AGT-09-03: Build compensating actions and checkpoint resume into a LangGraph agent.
  • PB-AGT-09-04: Drain and replay the agent dead-letter queue.
  • PB-AGT-01-02: Build a task agent from the template (idempotency keys).
  • PB-AGT-02-01: Define a tool with Pydantic schemas and an error contract.
  • PB-AGT-02-05: Handle a tool dependency outage or throttling (tool side).
  • PB-AGT-07-02: Run the agent regression gate.
  • PB-AGT-08-01: Define an agent's permission envelope.
  • PB-AGT-11-02: Roll out or change API Management token limits per tenant.