Build compensating actions and checkpoint resume into a LangGraph agent¶
Draft outline: not validated
Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-08). Author: Sivaraj · dry-run: Prasanna · sign-off: Yogeshwaran. Work items: AGT-09-T09, AGT-09-T10, AGT-09-T11, AGT-09-T12.
1. Summary¶
When an agent performs more than one write in a run, so a failure halfway leaves systems inconsistent. Use it during build, before the first multi-step write goes live.
Risk: High (partial completion leaves orders, tickets or records half done). Typical duration: 2-3 days per agent.
2. When to use and when not to¶
Use it when: When an agent performs more than one write in a run, so a failure halfway leaves systems inconsistent. Use it during build, before the first multi-step write goes live.
Do not use it when: to be written by the author.
3. What you must know first¶
- Checkpointers save graph state per thread so a run can resume after a crash
- Resuming replays from the last checkpoint, so writes must be idempotent
- A compensating action undoes a completed step; not every step can be undone
- Steps that cannot be undone need approval before they run
- Compensation itself can fail and must alert a human
4. Prerequisites and access¶
- Durable checkpointer such as PostgresSaver configured
- List of write tools with their undo operation or manual fallback
- Idempotency keys supported by the target systems
5. Steps¶
Outline of the steps; the author adds the commands and a check after each one.
- List each write tool and its undo action, or mark it irreversible.
- Move irreversible actions to the end of the graph behind an approval gate.
- Pass an idempotency key derived from run id and step to every write.
- Register a compensation after each successful write.
- On failure, run compensations in reverse order and record each outcome.
- Alert a human with the list of compensations that failed.
- Compile the graph with a durable checkpointer and stable thread ids.
- Test a crash between two writes and resume from the checkpoint.
- Test a failure after two writes and confirm both are undone.
Draft snippet: Saga helper with reverse-order compensation
from dataclasses import dataclass, field
from typing import Callable
@dataclass
class Saga:
run_id: str
compensations: list[tuple[str, Callable[[], None]]] = field(default_factory=list)
def step(self, name: str, action: Callable[[], object], undo: Callable[[], None]):
result = action() # action must use an idempotency key based on run_id and name
self.compensations.append((name, undo))
return result
def compensate(self, log) -> list[str]:
failed = []
for name, undo in reversed(self.compensations):
try:
undo()
log.info("compensated %s for run %s", name, self.run_id)
except Exception:
log.exception("compensation failed: %s", name)
failed.append(name)
return failed # alert a human if not empty
Snippets to add:
snippets/agent/saga.py: Saga helper with reverse-order compensation (python)snippets/tests/test_resume.py: Crash-and-resume test with a durable checkpointer (python)
6. Verify it works¶
A second engineer injects a failure after the second of three writes in the sandbox agent, then separately kills the process between writes. Pass: the first case undoes both writes and the second resumes without duplicating any write.
7. Known issues and fixes¶
| Symptom | Likely cause | Fix |
|---|---|---|
| Resume repeats a write | Write not idempotent | Use an idempotency key per run and step |
| Compensation fails silently | Exceptions swallowed in the undo loop | Log, alert and return the failed list for manual follow-up |
| Cannot resume after deploy | State schema changed between versions | Version the state schema and drain in-flight runs before breaking changes |
8. Monitoring and alerts¶
| Signal | Alert when | Action |
|---|---|---|
| Compensation failures | Any failed compensation | Page the agent owner with the run id and failed steps |
| Runs in partial state | Older than 1 hour without resume or compensation | Review and decide resume or undo |
9. AWS and open-source equivalents¶
On AWS, Step Functions supports saga-style compensation with catch states, and Bedrock AgentCore Runtime hosts LangGraph agents. Open-source alternatives include Temporal for durable workflows.
10. Validation record¶
Pending. Planned dry-run scenario: A second engineer injects a failure after the second of three writes in the sandbox agent, then separately kills the process between writes. Pass: the first case undoes both writes and the second resumes without duplicating any write.