AGT-12 · Agent simulation & scenario testing¶
Agent behaviour is tested by hand in a chat window, so nobody can show that a change kept the agent safe on last month's tricky cases. Without recorded scenarios and mocked tools, raising an agent's autonomy level is a leap of faith.
- Who reads it: QE engineers and agent engineers building regression suites for agents (emerging practice, Level 1)
- Outcome: Recorded scenarios replayed on every change, with flaky results triaged
- Pack owner: Prasanna (catalog owner role: QE Lead)
Master catalog entry¶
- Catalog name: Agent Simulation & Test Harness (P3)
- Why it matters: Enables confident scaling of autonomy levels over time
- What it removes: Regression-tests agent behaviour against recorded scenarios
- Stack: Python, mocked tools, recorded traces, pytest
- Maturity target: Level 1 — emerging practice; expect iteration
Playbooks¶
| ID | Story | Title | Kind | Complexity | Urgency | Status | Owner | Planned publish |
|---|---|---|---|---|---|---|---|---|
| PB-AGT-12-01 | AGT-12-S1 | Turn recorded agent traces into replayable scenarios | How-to | Medium | 2 | draft | Devadharshini Raja | 2026-12-30 |
| PB-AGT-12-02 | AGT-12-S2 | Run scenario regression tests with mocked tools in pytest | How-to | Medium | 3 | draft | Devadharshini Raja | 2026-12-24 |
| PB-AGT-12-03 | AGT-12-S3 | Triage a failing or flaky scenario run in CI | Runbook | Medium | 4 | draft | Devadharshini Raja | 2026-12-14 |
| PB-AGT-12-04 | AGT-12-S4 | Autonomy level promotion checklist using simulation evidence | Checklist | Low | 2 | draft | Devadharshini Raja | 2026-12-28 |