Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 days for the first 20 scenarios
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Medium (recorded traces may contain personal or client data)

Turn recorded agent traces into replayable scenarios

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-30). Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: AGT-12-T01, AGT-12-T02, AGT-12-T03, AGT-12-T04.

1. Summary

When an agent has production or UAT traces and the team wants regression cases from real behaviour, especially after an incident or a user complaint.

Risk: Medium (recorded traces may contain personal or client data). Typical duration: 1-2 days for the first 20 scenarios.

2. When to use and when not to

Use it when: When an agent has production or UAT traces and the team wants regression cases from real behaviour, especially after an incident or a user complaint.

Do not use it when: to be written by the author.

3. What you must know first

  • A scenario is input, recorded tool results and expected behaviour, not a full transcript
  • Expected behaviour should be the tool sequence and key facts, not exact wording
  • Trace content can include user inputs and tool arguments; redact before it enters the repository
  • Store scenarios as plain JSON so they diff well in review

4. Prerequisites and access

  • Traces with tool names, arguments and results captured (Foundry tracing or OpenTelemetry)
  • Data handling approval for using trace content in tests
  • Redaction script for personal data

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Pick traces covering normal tasks, edge cases and every past incident.
  2. Export the user input, each tool call with arguments and each tool result.
  3. Redact personal and client data and replace with realistic fake values.
  4. Write the expected tool sequence, step ceiling and facts the answer must mention.
  5. Add a seed and the model deployment used at recording time.
  6. Review each scenario with the agent lead for correctness.
  7. Commit scenarios under tests/scenarios with one file per case.

Draft snippet: Scenario recorded from a trace

{
  "id": "refund-over-limit",
  "source": "trace from UAT, redacted",
  "recordedWith": {"deployment": "${AGENT_DEPLOYMENT}", "agentVersion": "1.4.0"},
  "seed": 7,
  "input": "Customer C-1001 wants a refund of 900 GBP on invoice INV-2001",
  "tool_calls": [
    {"tool": "lookup_invoice", "args": {"invoice_id": "INV-2001"}, "result": {"amount": 900, "status": "paid"}},
    {"tool": "get_refund_limit", "args": {"customer_id": "C-1001"}, "result": {"limit": 500}}
  ],
  "expected": {
    "tool_sequence": ["lookup_invoice", "get_refund_limit"],
    "max_steps": 6,
    "must_mention": ["approval", "500"],
    "must_not_call": ["issue_refund"]
  }
}

Snippets to add:

  • snippets/tests/scenarios/refund-over-limit.json: Scenario file format (json)
  • snippets/tools/export_scenario.py: Trace-to-scenario export script (python)

6. Verify it works

A second engineer converts 5 sandbox traces into scenarios using the playbook. Pass: the redaction scan finds no personal data, each scenario replays with the harness and the agent lead agrees with every expected block.

7. Known issues and fixes

Symptom Likely cause Fix
Scenarios fail after harmless wording changes Expected output checked word for word Check tool sequence and required facts only
Personal data found in the repository Redaction missed a field Remove history per policy and extend the redaction rules
Scenario cannot be replayed Tool result missing from the trace Enable capture of tool results in tracing or record them in the harness

8. Monitoring and alerts

Signal Alert when Action
Scenario files failing the redaction scan in CI Any Block the merge until redacted

9. AWS and open-source equivalents

On AWS, AgentCore observability traces can be exported the same way. Open-source tools such as Langfuse and LangSmith datasets store traces as replayable examples.

10. Validation record

Pending. Planned dry-run scenario: A second engineer converts 5 sandbox traces into scenarios using the playbook. Pass: the redaction scan finds no personal data, each scenario replays with the harness and the agent lead agrees with every expected block.

11. Sources