⚠️ Before You Start

Risk: Medium

Duration: 30 minutes to contain; duration of the outage

Last Validated: Not yet validated (draft)

Kind: Runbook

Risk Note: Medium (users get failed answers; retries can make throttling worse)

Handle a tool dependency outage or throttling¶

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026; verified on 21 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-11). Author: Sanjay Sarvesh · dry-run: Yasar · sign-off: James. Work items: AGT-02-T17, AGT-02-T18, AGT-02-T19, AGT-02-T20.

1. Summary¶

This runbook isolates, contains and recovers from a downstream API dependency outage or throttling condition affecting an agent tool. It stops agent looping, sheds load with circuit breaking and rate limiting, returns deterministic degraded responses to the model, and safely restores normal tool calling once the upstream dependency recovers.

Risk: Medium (users receive degraded answers while the tool is disabled; uncontrolled retries worsen upstream throttling). Typical duration: 30 minutes to contain; duration of the downstream outage to resolve.

2. When to use and when not to¶

Use this runbook when:

A downstream API behind an agent tool or MCP server returns HTTP 429 Too Many Requests, 502 Bad Gateway, 503 Service Unavailable or 504 Gateway Timeout.
Outbound tool calls exceed their latency budget or client timeout, causing agent execution steps to stall.
Agents enter recursive tool-calling loops because a downstream tool failure returns an unhandled or vague retryable error.
Downstream service teams announce an active incident or emergency maintenance affecting dependencies integrated with tools.

Do not use it when:

The foundation model itself is throttled (HTTP 429 TPM/RPM on Azure OpenAI or Amazon Bedrock). Use model-gateway rate limits, Provisioned Throughput (PTU), or cross-region model routing instead.
The tool failure is caused by an expired secret, rotated key or authentication error (HTTP 401/403). Use PB-AGT-02-02 instead.
The tool fails due to a parameter schema mismatch or breaking API change (HTTP 400). Use PB-AGT-02-03 instead.
The tool was blocked by an authorization guardrail or security review. Use PB-AGT-02-04 instead.

3. What you must know first¶

Tool timeouts must be shorter than the agent step timeout. Orchestrator engines impose strict step deadlines. If a tool client has no timeout or a timeout longer than the agent step budget, the orchestrator forcibly terminates the step before the tool wrapper can record metrics, trip a circuit breaker, or format a clean error.
HTTP 429 requires exponential backoff with full jitter. Rapid retries against a throttled service exhaust token-bucket quotas immediately. Tools must respect Retry-After headers and never run fast retry loops inside an agent execution turn.
MCP protocol error contract vs tool execution errors. The Model Context Protocol (2026-07-28) reserves JSON-RPC error objects for protocol failures (unknown tool, malformed request). Dependency failures, like input-validation errors, are returned inside result with isError: true and a text content block so the language model can read the explanation and reason about next steps.
A clear, non-retryable error message breaks agent loops. When a dependency is unavailable, returning an ambiguous error or marking the error as transient causes the model to guess new arguments or repeatedly invoke the failing tool. The tool result must explicitly instruct the model: "The downstream service is currently unavailable. Do not retry this tool; inform the user directly."
Azure API Management policies apply at the MCP server endpoint level. In Azure API Management, policies such as rate limiting and quotas apply to the entire MCP server API rather than per-tool operations. For fine-grained per-tool controls, enforce rate limits via application-level tool wrappers or inspect the JSON-RPC tool name in gateway routing policies.

4. Prerequisites and access¶

Permissions: Contributor access to Azure API Management (or AWS API Gateway Administrator), and Operator access to the agent feature flag configuration (Azure App Configuration or LaunchDarkly).
Observability: Access to Application Insights / Log Analytics workspace (or Amazon CloudWatch Logs Insights) with permissions to run KQL queries for tool execution traces and error codes.
Dependency Contacts: Confirmed status page URL and on-call escalation channel (Slack/Teams) for the downstream dependency team.
Configuration: Ability to deploy or update tool registry configuration (tools.yaml) and circuit breaker parameters.

5. Steps¶

Follow these steps sequentially to diagnose, contain, and resolve tool dependency outages.

Step 1: Confirm the failing dependency from tool error codes and traces¶

Inspect telemetry to identify the exact failing tool, downstream endpoint, and HTTP response code.

// Query tool invocation failures by dependency over the last 30 minutes
traces
| where timestamp > ago(30m)
| where customDimensions["type"] == "tool_execution"
| extend tool_name = tostring(customDimensions["tool_name"]),
         dependency = tostring(customDimensions["dependency"]),
         status_code = toint(customDimensions["status_code"]),
         duration_ms = todouble(customDimensions["duration_ms"])
| summarize total_calls = count(),
            failure_count = countif(status_code >= 400 or isempty(status_code)),
            p95_duration = percentile(duration_ms, 95)
            by tool_name, dependency, status_code
| extend failure_rate = round(100.0 * failure_count / total_calls, 2)
| order by failure_count desc

Check: The query output shows which specific downstream dependency is returning 429, 5xx, or timing out.

Step 2: Check downstream dependency status and contact the owner¶

Open the dependency's internal or external status page.
Check for active incident declarations, planned maintenance windows, or degraded performance notices.
Message the dependency owner on the team's operations channel with the error rate and p95 latency.

Check: Dependency owner confirms whether the issue is a transient rate-limit spike or an extended outage.

Step 3: Disable the tool by feature flag or configure backend circuit-breaker rules¶

Disabling a tool changes agent runtime behaviour

Removing or disabling a tool removes its capability from the agent's schema. While the tool is disabled, user prompts requiring this tool will be answered with a degraded fallback message.

If the dependency owner cannot confirm recovery within the agreed containment window (TO VERIFY with the SRE team), disable the tool flag to prevent cascading timeouts.

Via Azure CLI (Azure App Configuration):

az appconfig feature disable \
  --name "${APP_CONFIG_NAME}" \
  --feature "tool_${TOOL_NAME}_enabled" \
  --yes

If the dependency sits behind API Management, confirm the backend has a circuit-breaker rule so the gateway stops forwarding once failures meet the rule (failureCondition, tripDuration, acceptRetryAfter). A rule trips on observed failures; it cannot be opened on demand, so use the feature flag for an immediate stop. Backend circuit breakers are not supported in the Consumption tier, and tripping is approximate.

Via Azure API Management native backend circuit breaker update:

az rest --method patch \
  --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.ApiManagement/service/${APIM_NAME}/backends/${BACKEND_NAME}?api-version=2024-05-01" \
  --body '{
    "properties": {
      "circuitBreaker": {
        "rules": [
          {
            "name": "OutageContainmentRule",
            "failureCondition": {
              "count": 3,
              "interval": "PT1M",
              "statusCodeRanges": [{"min": 500, "max": 599}, {"min": 429, "max": 429}]
            },
            "tripDuration": "PT15M",
            "acceptRetryAfter": true
          }
        ]
      }
    }
  }'

Check: az appconfig feature show --name "${APP_CONFIG_NAME}" --feature "tool_${TOOL_NAME}_enabled" shows the flag disabled, and agent traces stop showing calls to the tool after the provider's refresh interval (30 seconds by default; feature-flag refresh must be enabled in the agent's App Configuration provider).

Step 4: Ensure tool returns a non-retryable degraded MCP response¶

Verify the tool wrapper returns a compliant MCP CallToolResult with isError: true containing explicit instructions for the model.

"""
Tool wrapper with client timeout, thread-safe circuit breaker, and MCP-compliant error contract.
"""
import enum
import threading
import time
from typing import Any, Dict
import httpx

class BreakerState(enum.Enum):
    CLOSED = "CLOSED"
    OPEN = "OPEN"
    HALF_OPEN = "HALF_OPEN"

class CircuitBreaker:
    def __init__(self, failure_limit: int = 5, recovery_timeout: float = 60.0):
        self.failure_limit = failure_limit
        self.recovery_timeout = recovery_timeout
        self.failure_count = 0
        self.state = BreakerState.CLOSED
        self.last_state_change = time.time()
        self._lock = threading.Lock()

    def allow_request(self) -> bool:
        with self._lock:
            now = time.time()
            if self.state == BreakerState.OPEN:
                if now - self.last_state_change > self.recovery_timeout:
                    self.state = BreakerState.HALF_OPEN
                    self.last_state_change = now
                    return True
                return False
            return True

    def record_success(self) -> None:
        with self._lock:
            self.failure_count = 0
            self.state = BreakerState.CLOSED
            self.last_state_change = time.time()

    def record_failure(self) -> None:
        with self._lock:
            self.failure_count += 1
            if self.failure_count >= self.failure_limit or self.state == BreakerState.HALF_OPEN:
                self.state = BreakerState.OPEN
                self.last_state_change = time.time()

breaker = CircuitBreaker(failure_limit=5, recovery_timeout=60.0)

def call_crm_tool(endpoint: str, payload: Dict[str, Any]) -> Dict[str, Any]:
    """Invokes CRM downstream dependency with circuit breaking and MCP error contract."""
    if not breaker.allow_request():
        return {
            "content": [
                {
                    "type": "text",
                    "text": "CRM service is temporarily unavailable (circuit breaker open). "
                            "Do NOT retry this tool call. Inform the user that CRM records cannot be accessed right now."
                }
            ],
            "isError": True
        }

    try:
        with httpx.Client(base_url="${CRM_BASE_URL}", timeout=httpx.Timeout(10.0)) as client:
            response = client.post(endpoint, json=payload)
            response.raise_for_status()
            breaker.record_success()
            return {
                "content": [{"type": "text", "text": response.text}],
                "isError": False
            }
    except (httpx.TimeoutException, httpx.HTTPStatusError) as exc:
        breaker.record_failure()
        status = getattr(getattr(exc, "response", None), "status_code", "timeout")
        return {
            "content": [
                {
                    "type": "text",
                    "text": f"CRM dependency error (status: {status}). "
                            "Service is degraded. Do NOT retry this tool call; summarize the failure for the user."
                }
            ],
            "isError": True
        }

Check: Execute a synthetic tool call while the backend is down; verify the return payload has "isError": true and agents complete their turn without recursive retries.

Step 5: Reduce concurrency or apply rate limits at the gateway if throttled¶

If the root cause is HTTP 429 throttling due to excessive concurrent agent workers, apply a strict rate-limit-by-key policy in Azure API Management to prevent token exhaustion.

<!-- In Azure API Management policy for the tool or MCP backend -->
<in-bound>
    <base />
    <rate-limit-by-key calls="50" renewal-period="60"
        counter-key="@(context.Subscription.Id)"
        increment-condition="@(context.Response.StatusCode != 429)" />
</in-bound>

Check: Run load test probes and confirm that requests exceeding the limit receive HTTP 429 with appropriate Retry-After headers before hitting the upstream API.

Step 6: Monitor error and loop alerts until stable¶

Watch agent orchestration metrics in Application Insights to ensure agents do not loop or stall.

// Monitor agent execution step count and loop terminations
traces
| where timestamp > ago(15m)
| where customDimensions["type"] == "agent_run"
| extend step_count = toint(customDimensions["step_count"]),
         outcome = tostring(customDimensions["outcome"])
| summarize avg_steps = avg(step_count),
            p99_steps = percentile(step_count, 99),
            failed_runs = countif(outcome == "loop_limit_exceeded")
            by bin(timestamp, 1m)
| order by timestamp desc

Check: failed_runs for loop_limit_exceeded drops to zero and avg_steps returns to normal baseline.

Step 7: Close the circuit gradually when dependency recovers¶

Once the downstream service reports healthy:

Allow the circuit breaker to enter Half-Open state.
Send single synthetic probe requests through the tool connector.
If probe requests succeed, re-enable the tool feature flag in Azure App Configuration:

az appconfig feature enable \
  --name "${APP_CONFIG_NAME}" \
  --feature "tool_${TOOL_NAME}_enabled" \
  --yes

Check: Probe calls return HTTP 200 OK with normal latency. The circuit breaker resets failure_count to 0.

Step 8: Replay or notify users of failed requests if required¶

Extract agent conversations that encountered the degraded fallback message during the outage window.
If the agent supports asynchronous replay or offline task continuation, enqueue pending user requests into the processing queue.
If requests were real-time chat sessions, publish an operational status update confirming service restoration.

Check: Telemetry confirms live agent queries are successfully invoking the tool with 0% error rate.

6. Monitoring and alerts¶

Signal	Alert when	Action
Tool error rate per dependency	Above 10 percent for 5 minutes	Verify downstream health and trip circuit breaker if failing
Downstream HTTP 429 responses	Above baseline for 5 minutes	Reduce agent concurrency; enforce gateway rate limits
p95 tool execution latency	Above the tool's timeout budget	Open circuit breaker; investigate network or backend bottleneck
Agent loop limit terminations	Any occurrence in 5 minutes	Inspect tool error responses; confirm `isError: true` non-retryable directive is present

7. Rollback¶

If containment steps cause unexpected agent behaviour or need to be reversed:

Re-enable disabled tool flag: Run az appconfig feature enable --name "${APP_CONFIG_NAME}" --feature "tool_${TOOL_NAME}_enabled" --yes. Rollback takes effect immediately upon client cache expiration (typically 30 seconds).
Reset circuit breaker: Restore the backend circuit-breaker rule recorded before the incident (keep a copy of the backend JSON before Step 3). In application code, reset the breaker to CLOSED.
Revert gateway rate limits: Remove the temporary strict rate-limit-by-key XML policy from APIM to restore full production throughput quotas.
Safety check: Run one end-to-end synthetic agent interaction before declaring the rollback complete.

8. Known issues and fixes¶

Symptom	Likely cause	Fix
Agents loop continuously on a failing tool	Tool returns vague string or marks error as retryable	During an open circuit, return the registry error `dependency_unavailable` with `retryable: false`, serialised in the content text with `isError: true`, and an explicit instruction not to retry.
Throttling cascades across all agents	All agents share a single API key or unmetered quota	Apply `rate-limit-by-key` by agent ID or subscription key at the API gateway.
Agent steps hang until overall timeout	Missing or unbounded HTTP client timeout	Set an explicit client timeout on every outbound call, shorter than the agent step timeout.
Circuit breaker flaps rapidly between Open and Closed	Recovery timeout too short or no half-open concurrency cap	Lengthen the recovery timeout (value TO VERIFY) and require several consecutive successful probes before closing.
Agent hallucinates alternative actions when tool is disabled	Model prompt does not handle missing tool schemas	Provide fallback prompt guidance explaining that specific capabilities may be temporarily offline.

9. AWS and open-source equivalents¶

AWS: AgentCore Gateway and API Gateway throttling provide the same controls on AWS. For Lambda-backed Bedrock action groups, monitor Lambda Errors and Throttles; for API Gateway, monitor 5XXError and 4XXError.
Open Source / Python: Use tenacity with wait_random_exponential and stop_after_attempt for transient retry loops, pybreaker for redis-backed or in-memory circuit breakers, and LangChain's handle_tool_error (ToolException) to catch errors and return structured ToolMessage payloads with status="error".

10. Validation record¶

Pending. Planned dry-run scenario: In a sandbox, a second engineer points a tool at a fault-injection endpoint returning 503 and 429, and passes when agents return the degraded message without looping and recover automatically after the fault is removed.