Stop and triage an agent stuck in a loop or burning tokens¶
Draft outline: not validated
Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-25). Author: Sivaraj · dry-run: Sandeep Velu · sign-off: Deeparani. Work items: AGT-09-T01, AGT-09-T02, AGT-09-T03, AGT-09-T04.
1. Summary¶
When an alert shows a run with abnormal steps, tool calls or token spend, or a user reports an agent that never answers. Use it during the incident, before any code change.
Risk: High (live spend and possible repeated side effects on customer systems). Typical duration: 30-90 minutes to contain; root cause within a day.
2. When to use and when not to¶
Use it when: When an alert shows a run with abnormal steps, tool calls or token spend, or a user reports an agent that never answers. Use it during the incident, before any code change.
Do not use it when: to be written by the author.
3. What you must know first¶
- get_state_history lists checkpoints for a thread, newest first
- Repeated node scheduling with no state change is the loop signature
- recursion_limit is the hard stop; budgets in state are the soft stop
- Side effects before an interrupt or in a retried node may have run more than once
- Model deployment rate limits in Foundry are set per deployment
4. Prerequisites and access¶
- Read access to Application Insights and the checkpointer database
- Permission to scale down or disable the agent deployment
- Kill switch flag in app configuration
- On-call contact for the agent owner
5. Steps¶
Outline of the steps; the author adds the commands and a check after each one.
- Confirm the alert: find the run and thread_id in traces and note start time and spend
- Contain: set the agent kill switch or scale the deployment to zero for the affected tenant
- Inspect the thread with get_state_history and count which nodes repeat
- Check whether a tool returned the same error each time and the model kept retrying
- List side effects executed by the looping run and notify owners of affected systems
- Apply the fix in sandbox: stop rule, tool error contract or prompt; replay the thread
- Release the fix through the normal pipeline and re-enable the agent
- Verify no new loop alerts for 24 hours and record the incident
- Rollback: redeploy the previous image and prompt version if the fix raises errors
Draft snippet: Find the looping node in a thread's checkpoint history
from collections import Counter
from langgraph.checkpoint.postgres import PostgresSaver
from agent_app.graph import builder # the template's uncompiled StateGraph
DB_URI = "${PG_CONN_STRING}"
THREAD_ID = "${THREAD_ID}"
with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
graph = builder.compile(checkpointer=checkpointer)
config = {"configurable": {"thread_id": THREAD_ID}}
visits = Counter()
checkpoints = 0
for snapshot in graph.get_state_history(config):
checkpoints += 1
for node in snapshot.next:
visits[node] += 1
latest = graph.get_state(config)
print(f"checkpoints={checkpoints} latest_next={latest.next}")
print("most scheduled nodes:", visits.most_common(5))
# A node that dominates this list with no state change is the loop.
Snippets to add:
snippets/ops/inspect_thread.py: Inspect a thread's checkpoint history (python)snippets/ops/queries/loop_detection.kql: Runs with abnormal model calls (kql)snippets/ops/kill_switch.sh: Flip the kill switch in App Configuration (bash)
6. Monitoring and alerts¶
| Signal | Alert when | Action |
|---|---|---|
| Model calls per run (custom metric from spans) | More than 30 or 3x the agent's p95 | Page on-call and start this runbook |
| Token spend per hour per agent | More than 2x the 7-day hourly baseline | Check for loops and recent releases |
| GraphRecursionError count | Any in 15 minutes | Open the affected threads and contain |
7. Rollback¶
To be written by the author: how to undo each risky step.
8. Known issues and fixes¶
| Symptom | Likely cause | Fix |
|---|---|---|
| Same tool called with identical arguments again and again | Tool error message is vague so the model retries | Return isError with a clear, non-retryable code and a message the model can act on |
| Supervisor alternates between two workers | No hop counter or finish criterion | Add a hop limit and require a FINISH decision with structured output |
| Token spend climbs but steps look normal | Conversation history not trimmed; each step resends everything | Summarise or trim history and cap context size |
| Loop restarts after the pod restarts | Thread resumes from the last checkpoint with the same bad state | Mark the thread as closed in the registry or update state to a terminal value before re-enabling |
| Same tool called with the same arguments again and again | Tool returns an error the model cannot interpret | Return structured, plain-language errors and add a repeated-call guard |
| Run ends with GraphRecursionError on valid long tasks | Limit too low for the task | Split the task into sub-graphs or raise the limit only for that graph after review |
9. AWS and open-source equivalents¶
On AgentCore, use Observability in CloudWatch for traces and token metrics and stop the runtime endpoint to contain. LangSmith tracing gives the same per-run view for LangGraph on any cloud.
10. Validation record¶
Pending. Planned dry-run scenario: In a sandbox, a second engineer deploys a deliberately looping agent, lets the alert fire, contains it within 15 minutes using the kill switch, identifies the looping node from history and confirms no further spend after containment.