Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 2-6 hours
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (cost overrun and poor answers until fixed)

Investigate bad plans, replan loops and planner cost spikes

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Verified and completed against official documentation on 02 Nov 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-11). Author: Yasar · dry-run: Sanjay Sarvesh · sign-off: James. Work items: AGT-03-T09, AGT-03-T10, AGT-03-T11, AGT-03-T12.

1. Summary

This runbook guides engineers through diagnosing and mitigating planner failure modes in autonomous agents, including unbounded replan loops, degraded task plans, and sudden token cost spikes. It isolates root causes across agent prompts, underlying tool errors, context window bloat, and foundation model behavioral drift.

There are two investigation paths: Path A (telemetry-driven diagnosis via Application Insights KQL queries and distributed traces) and Path B (sandbox thread replay and bounded replanning configuration tuning). Path A is the default in production. Facts checked against official documentation on 02 November 2026.

2. When to use and when not to

Use this runbook when:

  • Production telemetry alerts on token consumption per task rising > 1.5x above the 14-day baseline.
  • Users report that an agent is trapped in repetitive action cycles or failing to terminate multi-step tasks.
  • Evaluation metrics show task completion rates falling > 5 percentage points below release thresholds.
  • Average response latency spikes due to repeated planning and replanning graph iterations.
  • An underlying foundation model version upgrade causes reasoning drift or unhandled schema formatting errors.

Do not use it when:

  • You are selecting an agent planner architecture (workflow, ReAct, plan-and-execute) from scratch. Use PB-AGT-03-01 instead.
  • You are implementing bounded replanning graphs in LangGraph during development. Use PB-AGT-03-02 instead.
  • You are executing a pre-deployment regression verification checklist for prompt updates. Use PB-AGT-03-04 instead.
  • The failure is isolated to a third-party tool API outage or HTTP 429 rate limit. Use PB-AGT-02-05 instead.

3. What you must know first

  1. OpenTelemetry GenAI conventions. OpenTelemetry defines standard span attributes for generative AI (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.operation.name; see OpenTelemetry Semantic Conventions for GenAI spans). In Azure Foundry and Application Insights, these attributes record model calls and tool executions within unified trace graphs.
  2. Anatomy of a replan loop. An agent enters a replan loop when an execution error or unexpected tool response triggers a replan step, but the replan prompt fails to update its hypothesis. Unconstrained graphs will loop indefinitely until halted by external timeouts or token budget caps.
  3. Quadratic context expansion. In multi-step planners, every replanning cycle re-submits previous step histories, thoughts, and verbose tool responses. Cumulative input tokens grow quadratically ($O(N^2)$), multiplying overall task costs even when model call counts appear modest.
  4. Tool failures disguised as planner defects. Planners frequently fail because an underlying tool returns a formatted error or schema mismatch that the model misinterprets as an environmental change. Always audit tool execution spans before altering planner system prompts.
  5. Recursion limits in agent frameworks. In LangGraph and custom state graphs, an explicit recursion limit (recursion_limit) must be set on every execution invoke (default varies by version). When reached, the graph raises GraphRecursionError, preventing unbounded billing spikes.
  6. Decoupled routing and configuration switches. Planner model deployments, prompt templates, and recursion limits must be decoupled into runtime configuration (AZURE_OPENAI_DEPLOYMENT_NAME, Agent:Planner:MaxReplanCount mapped to max_replan_iterations in code). This enables emergency rollbacks without a code deployment.

4. Prerequisites and access

Item Detail
Roles Monitoring Reader or Log Analytics Reader (query traces in Log Analytics / Application Insights) and Cognitive Services OpenAI User (execute sandbox re-runs). Use Microsoft Entra ID rather than shared API keys.
Tools Azure CLI (az), Python 3.10+, curl, jq, opentelemetry-sdk, langgraph (>= 0.2.0).
Trace access Access to the Azure Application Insights workspace receiving distributed spans from Azure Foundry or LangGraph instrumentation.
Release history Prompt, model, and tool releases for the period under investigation.
Failing tasks A sample of failing tasks or thread/run IDs showing replan loops or token expansion.
Sandbox Dedicated test environment configured to replay threads with identical inputs and tooling.
Test dataset The regression evaluation dataset (PB-AGT-03-04) containing representative complex multi-step queries with known step sequences.
Change ticket Approved operational change or incident response ticket specifying the investigation scope and rollback authority.

Export the environment variables and acquire an authentication token for the Application Insights API:

export AZURE_SUBSCRIPTION_ID="${SUBSCRIPTION_ID}"
export RESOURCE_GROUP="${RESOURCE_GROUP_NAME}"
export APPLICATION_INSIGHTS_APP_ID="${APP_INSIGHTS_ID}"
export AZURE_OPENAI_ENDPOINT="https://${RESOURCE_NAME}.openai.azure.com"
export TOKEN="$(az account get-access-token --resource https://api.applicationinsights.io --query accessToken --output tsv)"

5. Steps

Placeholders: ${APPLICATION_INSIGHTS_APP_ID}, ${TARGET_RUN_ID}, ${AZURE_OPENAI_ENDPOINT}, ${APP_CONFIG_NAME}.

Stage 1: Triage and identify runaway agent runs

Step 1. Identify high-cost runaway runs in Application Insights.

Query the top 20 agent executions by token volume and model call counts over the past 24 hours:

dependencies
| where timestamp > ago(24h) and name startswith "chat"
| extend run_id = tostring(customDimensions["app.run_id"]),
         in_tok = todouble(customDimensions["gen_ai.usage.input_tokens"]),
         out_tok = todouble(customDimensions["gen_ai.usage.output_tokens"])
| where isnotempty(run_id)
| summarize calls = count(), total_tokens = sum(in_tok + out_tok) by run_id
| top 20 by total_tokens desc

Verify: Query outputs the top 20 outlier runs sorted by total token volume and model call counts.

Step 2. Compare with the previous week.

Run the Step 1 summary for the current period and for the same period a week earlier, then compare input tokens, output tokens and model calls per task to find which grew:

let current_period = dependencies
| where timestamp between (ago(24h) .. now()) and isnotempty(customDimensions["app.run_id"])
| extend in_tok = todouble(customDimensions["gen_ai.usage.input_tokens"]),
         out_tok = todouble(customDimensions["gen_ai.usage.output_tokens"])
| summarize period = "current_24h", avg_in = avg(in_tok), avg_out = avg(out_tok), p95_tok = percentile(in_tok + out_tok, 95);
let previous_week = dependencies
| where timestamp between (ago(8d) .. ago(7d)) and isnotempty(customDimensions["app.run_id"])
| extend in_tok = todouble(customDimensions["gen_ai.usage.input_tokens"]),
         out_tok = todouble(customDimensions["gen_ai.usage.output_tokens"])
| summarize period = "prior_week_same_day", avg_in = avg(in_tok), avg_out = avg(out_tok), p95_tok = percentile(in_tok + out_tok, 95);
union current_period, previous_week

Verify: Output compares token distributions against the previous week to isolate whether cost expansion is driven by ballooning input tokens (context accumulation) or excessive model call counts (replan looping).

Stage 2: Trace inspection and failure classification

Step 3. Extract execution span trees for outlier runs.

Extract the span trees for at least three of the worst runs identified in Step 1 and inspect the plan, replans, and tool results:

dependencies
| where timestamp > ago(24h) and customDimensions["app.run_id"] == "${TARGET_RUN_ID}"
| project timestamp, name, type, duration, resultCode, customDimensions
| order by timestamp asc

Verify: Chronological output displays the sequence of planner reasoning steps, tool calls, and replan triggers for each analyzed run.

Step 4. Classify the root cause of the planning breakdown.

Examine the trace spans and classify the failure into one of the recognized archetypes:

  1. Unbounded replan loop: Model repeats identical tool calls with unchanged arguments due to failure to register previous tool output.
  2. Flaky tool cascade: An external tool returned a 5xx error or unexpected payload, prompting the planner to invent alternatives.
  3. Context window bloat: Accumulated message history exceeds the model's effective attention span, degrading plan coherence.
  4. Model behavioral drift: A model snapshot change altered formatting compliance, causing structured output validation retries.
  5. Prompt change: A planner or replanner prompt was edited in the recent release.
  6. Data change: Retrieved context or tool data changed shape, schema, or content.

Verify: Root cause category is recorded in the incident ticket with supporting span IDs.

Stage 3: Sandbox reproduction and parameter tuning

Step 5. Replay the failing thread in a sandbox environment.

Pass the recorded initial state and user query into the sandbox test harness:

python -m scripts.replay_thread \
  --run-id "${TARGET_RUN_ID}" \
  --endpoint "${AZURE_OPENAI_ENDPOINT}" \
  --output "sandbox_repro.json"

Verify: The sandbox execution reproduces the failure mode (looping or budget breach) with identical trace signatures.

Step 6. Enforce recursion limits and replan bounds in configuration.

Configure explicit guardrails on the planner state graph using runtime configuration parameters:

config = {
    "recursion_limit": 25,  # Enforce explicit invocation limit
    "configurable": {
        "max_replan_iterations": 3,  # Loaded from Agent:Planner:MaxReplanCount
        "summarize_intermediate_tool_outputs": True,
    },
}
app = workflow.compile()
response = app.invoke(initial_state, config=config)

Verify: The agent terminates cleanly when the replan threshold is reached, returning a fallback response rather than looping.

Stage 4: Quality validation and deployment

Production traffic impact

Updating planner prompts or recursion limits alters agent responses across all active user sessions. Ensure the canary telemetry watch is active before enabling changes.

Step 7. Add cases to regression set and validate against benchmark.

Add the reproduced failing cases to the regression dataset (PB-AGT-03-04) so they are tested on every subsequent change:

python -m eval.add_regression_cases \
  --input "sandbox_repro.json" \
  --dataset "eval/regression_dataset.jsonl"

Execute the offline evaluation suite across the regression benchmark:

python -m eval.run_eval \
  --dataset "eval/regression_dataset.jsonl" \
  --output "eval/results_candidate.json"

Verify: Task success and cost return to baseline, and no run exceeds the token cap.

Step 8. Deploy the configuration change to production.

Update the application configuration switch in Azure App Configuration:

az appconfig kv set \
  --name "${APP_CONFIG_NAME}" \
  --key "Agent:Planner:MaxReplanCount" \
  --value "3" \
  --yes

Verify: Telemetry over the agreed observation window (TO VERIFY in dry run) confirms average tokens per task return to baseline levels.

6. Monitoring and alerts

The thresholds below are aligned with the catalog baseline; tune them against an agreed measured baseline in production.

Signal Source Alert when Action
Tokens per task (Daily average) Application Insights custom metric More than 1.5x the 14-day average Initiate this runbook (Stage 1) to isolate runaway run IDs.
GraphRecursionError count Application Insights traces Any occurrence Inspect trace spans to identify failing nodes; verify tool health.
Replan count per thread Application telemetry span count Above the configured replan limit Investigate prompt clarity or tool output schemas.
Task completion rate Online evaluation pipeline Drops > 5 percentage points below the agreed baseline Execute rollback (Section 7) and test regression dataset in sandbox.
AzureOpenAIRequests 429 Azure Monitor metric Above the agreed baseline (TO VERIFY) Scale deployment provisioned throughput (PTU) or throttle agent concurrency.

7. Rollback

If the fix is not ready within a day, roll back the prompt or model version (rows 2–3) and continue the fix in the sandbox.

Situation Rollback Time
Replan storm causing runaway cost in production Set Agent:Planner:MaxReplanCount to 0 in App Configuration to stop replanning and force a single-pass answer (or 1 if single replan fallback is preferred; confirm against planner code). TO VERIFY in dry run
Prompt or configuration release degrades task accuracy Revert Agent:Planner:PromptVersion to the previous stable Git commit tag in App Configuration. TO VERIFY in dry run
Foundation model upgrade induces plan regressions Repoint AZURE_OPENAI_DEPLOYMENT_NAME to the previous stable model deployment snapshot. TO VERIFY in dry run
Tool failure triggers cascading replan cycles Mark the failing tool as disabled in the tool registry; agent returns a graceful capability refusal. TO VERIFY in dry run

Rollback triggers (any one of the following):

  1. Daily average tokens per task exceed 1.5x the 14-day average post-deployment (observation window TO VERIFY in dry run).
  2. Task completion rate on production sample drops > 5 percentage points below baseline.
  3. Any occurrence of unhandled GraphRecursionError in production.
  4. p95 request latency exceeds agreed baseline threshold (TO VERIFY in dry run).

8. Known issues and fixes

Symptom Likely cause Fix
Replans triggered by a single flaky tool Tool timeouts or transient 5xx errors Fix the tool and treat transient errors as a retry (see PB-AGT-02-05), not a replan.
Spend per task doubled immediately following a model snapshot switch Successor model generates more verbose reasoning steps and plans extra sub-tasks Tune the planner prompt (for example a step budget), or revert the model deployment.
Agent repeatedly invokes the same tool with identical arguments The planner prompt does not instruct the model to inspect previous tool failure messages Update planner prompt to require explicit reflection on previous tool output before selecting the next action.
Context length exceeded (400 context_length_exceeded) during replanning Full raw JSON payloads from multiple previous tool calls are retained in state history Implement tool response pruning or summarization to strip non-essential fields before appending to the message graph.
Planner hallucinates tools not in the registered catalog Tool catalog description is ambiguous, or system prompt lacks strict tool boundaries Constrain tool selection using JSON schema enums and add an explicit negative instruction against unlisted tools.
Planner omits mandatory verification or compliance steps Replan logic bypasses non-critical intermediate milestones under user query variations Enforce mandatory verification nodes deterministically in the graph workflow rather than leaving them to LLM discretion.
Extreme p95 latency on complex multi-step queries Planner generates deep sequential chains rather than executing independent steps in parallel Refactor planner to identify non-dependent sub-tasks and execute them concurrently via asyncio.gather.

9. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon Bedrock AgentCore Amazon CloudWatch and Bedrock AgentCore Observability AgentCore Observability shows token use, latency and errors in CloudWatch; configure CloudWatch alarms on those metrics to detect replan storms.
LangSmith & LangGraph LangSmith Tracing and Run Trees LangSmith automatically tracks full graph execution DAGs, token costs, latency, and step-level input/output states. Filter runs by status=error or order by total_tokens to pinpoint loops. Set alert webhooks on high-cost run thresholds.
Self-Hosted / Open Source (vLLM / Phoenix) OpenTelemetry Collector and Arize Phoenix Export standard OpenTelemetry GenAI spans to an OpenTelemetry collector and visualize in Arize Phoenix or Jaeger. Phoenix automatically computes token counts, tool call sequences, and evaluation drift across self-hosted open-source planners.

10. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A dry-run engineer deploys a simulated planner regression in a sandbox environment (injecting a mock tool error that triggers an unbounded replan loop). The dry-runner executes this runbook: queries Application Insights to isolate the runaway run ID, diagnoses the failure mode, applies a recursion limit and replan counter in configuration, and verifies that the agent terminates cleanly within token budget.

Pass criteria:

  • The second engineer finds the injected cause from traces and restores baseline cost within four hours.
  • Application Insights KQL query isolates runaway run IDs and identifies the token expansion driver.
  • Task success and cost return to baseline, and no run exceeds the token cap.

Record

Item Detail
Playbook PB-AGT-03-03 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

11. Sources

Checked against official documentation on 02 November 2026.