Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-3 hours depending on volume
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (replaying blindly repeats the failure or duplicates actions)

Drain and replay the agent dead-letter queue

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-14). Author: Sandeep Velu · dry-run: Yasar · sign-off: Sivaraj. Work items: AGT-09-T13, AGT-09-T14, AGT-09-T15, AGT-09-T16.

1. Summary

When the dead-letter count on the agent run queue is above zero, after an outage, or before a release that changes the run message format.

Risk: Medium (replaying blindly repeats the failure or duplicates actions). Typical duration: 1-3 hours depending on volume.

2. When to use and when not to

Use it when: When the dead-letter count on the agent run queue is above zero, after an outage, or before a release that changes the run message format.

Do not use it when: to be written by the author.

3. What you must know first

  • Each queue has a dead-letter subqueue at /$deadletterqueue that is never cleaned up automatically
  • Common reasons: MaxDeliveryCountExceeded, TTLExpiredException, HeaderSizeExceeded
  • Time-to-live is not enforced on the dead-letter queue
  • Peek first; receive and complete only what you replay
  • Service Bus Explorer in the portal can peek, edit and resend in small batches

4. Prerequisites and access

  • Azure Service Bus Data Owner or equivalent on the queue
  • Root cause of the dead-lettering fixed and deployed
  • Idempotent run handling (PB-AGT-09-03)

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Peek the dead-letter queue and group messages by dead-letter reason.
  2. Confirm the fix for each reason is deployed.
  3. Decide per group: replay, discard with a record, or hand to a human.
  4. Replay a batch of 5 messages and watch their runs complete.
  5. Replay the rest in batches, completing each dead-lettered message only after its resend succeeds.
  6. Record discarded messages with run ids in the incident or change ticket.
  7. Verify the dead-letter count is zero or only holds messages awaiting a human.
  8. Rollback: stop replay if the error rate rises; replayed messages that fail return to the dead-letter queue.

Draft snippet: Peek in dry-run mode, otherwise replay and complete

import os

from azure.identity import DefaultAzureCredential
from azure.servicebus import ServiceBusClient, ServiceBusMessage, ServiceBusSubQueue

QUEUE = "agent-runs"
DRY_RUN = os.environ.get("DRY_RUN", "true") == "true"
SKIP_REASONS = {"TTLExpiredException"}  # decide these with the business owner

with ServiceBusClient(os.environ["SERVICEBUS_FQDN"], DefaultAzureCredential()) as client:
    receiver = client.get_queue_receiver(QUEUE, sub_queue=ServiceBusSubQueue.DEAD_LETTER)
    sender = client.get_queue_sender(QUEUE)
    with receiver, sender:
        if DRY_RUN:
            for msg in receiver.peek_messages(max_message_count=50):
                print(msg.message_id, msg.dead_letter_reason, msg.dead_letter_error_description)
        else:
            for msg in receiver.receive_messages(max_message_count=5, max_wait_time=10):
                if msg.dead_letter_reason in SKIP_REASONS:
                    continue  # lock expires and the message stays for review
                sender.send_messages(ServiceBusMessage(str(msg), application_properties={"replayed": True}))
                receiver.complete_message(msg)

Snippets to add:

  • snippets/ops/dlq_replay.py: Peek, replay and complete dead-lettered runs (python)
  • snippets/ops/queue-counts.sh: CLI query for queue and dead-letter counts (bash)

6. Monitoring and alerts

Signal Alert when Action
Dead-lettered message count on agent queues Above 0 for 15 minutes Open a ticket and start this runbook
Dead-letter growth rate More than 50 messages in 10 minutes Page on-call; likely a live outage or bad deploy

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Replayed messages dead-letter again Root cause not fixed or message format changed Stop, fix the handler and retry a batch of 5
Messages with TTLExpiredException Runs waited longer than their time-to-live Decide with the business owner whether stale runs still make sense
Replay duplicates actions Handler not idempotent Stop replay and add idempotency keys first
Dead-letter count keeps rising Workers closing receivers before settling messages Settle messages before closing the receiver

9. AWS and open-source equivalents

On AWS, SQS dead-letter queues support redrive back to the source queue. RabbitMQ dead-letter exchanges and Kafka retry topics are the open-source equivalents.

10. Validation record

Pending. Planned dry-run scenario: In the sandbox, a second engineer forces 20 run messages into the dead-letter queue with a failing handler, fixes it and replays. Pass: dry run lists all 20 with reasons, replay completes them in batches and no run executes twice.

11. Sources