Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-4 hours to contain and diagnose
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (live spend and wrong results sent to users or systems)

Investigate a multi-agent incident: ping-pong, conflicting writes or cost spike

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-24). Author: James · dry-run: Yogeshwaran · sign-off: Deeparani. Work items: AGT-06-T13, AGT-06-T14, AGT-06-T15, AGT-06-T16.

1. Summary

When a multi-agent run never ends, produces contradictory results, or spend jumps after a release.

Risk: High (live spend and wrong results sent to users or systems). Typical duration: 1-4 hours to contain and diagnose.

2. When to use and when not to

Use it when: When a multi-agent run never ends, produces contradictory results, or spend jumps after a release.

Do not use it when: to be written by the author.

3. What you must know first

  • Foundry traces use invoke_agent, invoke_workflow and execute_tool spans
  • operation_Id groups spans of one run in Application Insights
  • Ping-pong shows as agents alternating with no new results
  • Conflicting writes come from more than one writer of a shared field
  • Hop and round limits are the first containment lever

4. Prerequisites and access

  • Access to traces in Application Insights
  • Kill switch per workflow
  • Release history
  • Sandbox for replay

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Find the affected runs and their operation IDs
  2. Contain: lower hop or round limits by config or switch off the workflow
  3. Read the agent sequence and spot ping-pong or repeated work
  4. Check which agents wrote shared fields and in what order
  5. Check recent changes to instructions, models or tools
  6. Reproduce in sandbox and fix routing, ownership or termination
  7. Release, re-enable and watch for 24 hours
  8. Rollback the release if hop counts stay high

Draft snippet: Detect high hop counts and handoff ping-pong

// Multi-agent runs with unusual hop counts or handoff ping-pong (Application Insights).
dependencies
| where timestamp > ago(6h)
| where name startswith "invoke_agent"
| extend agent = tostring(customDimensions["gen_ai.agent.name"])
| order by operation_Id asc, timestamp asc
| extend prev1 = prev(agent, 1), prev2 = prev(agent, 2), same_op = prev(operation_Id, 2) == operation_Id
| extend bounce = same_op and agent == prev2 and agent != prev1
| summarize hops = count(), agents = dcount(agent), back_and_forth = countif(bounce)
            by operation_Id
| where hops > 12 or back_and_forth > 3
| order by hops desc

Snippets to add:

  • snippets/ops/queries/multi_agent_pingpong.kql: Find runs with high hops or ping-pong (kql)
  • snippets/ops/set_hop_limit.sh: Lower hop limit by config (bash)

6. Monitoring and alerts

Signal Alert when Action
Agent invocations per run More than 12 or 3x the p95 Start this runbook
Back-and-forth handoffs per run More than 3 Start this runbook
Token spend per workflow per hour More than 2x baseline Check for loops and recent releases

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Two agents hand off to each other repeatedly Overlapping responsibilities in instructions Make responsibilities disjoint and restrict handoff targets
Final answer contradicts a worker result Two writers to the same field Give one agent ownership; others return proposals
Cost tripled after adding an agent New agent receives full history Filter inputs to the new agent
No spans for some agents Agent not instrumented Enable tracing integration for that framework

9. AWS and open-source equivalents

AgentCore Observability provides traces and metrics in CloudWatch. LangSmith and OpenAI Agents SDK tracing show agent and handoff spans for open-source stacks.

10. Validation record

Pending. Planned dry-run scenario: In a sandbox, a second engineer triggers a seeded ping-pong between two agents, contains it within 15 minutes and identifies the overlapping instruction from traces.

11. Sources