Skip to content
⚠️ Before You Start
Risk: Low
Duration: 1-2 days to set up, minutes per run
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Low (tests run against mocks; the risk is mocks drifting from real tools)

Run scenario regression tests with mocked tools in pytest

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-24). Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: AGT-12-T05, AGT-12-T06, AGT-12-T07, AGT-12-T08.

1. Summary

When scenarios exist and the team wants them to run on every pull request, without calling real systems.

Risk: Low (tests run against mocks; the risk is mocks drifting from real tools). Typical duration: 1-2 days to set up, minutes per run.

2. When to use and when not to

Use it when: When scenarios exist and the team wants them to run on every pull request, without calling real systems.

Do not use it when: to be written by the author.

3. What you must know first

  • pytest monkeypatch replaces an attribute for one test and undoes it afterwards
  • Patch the name where the agent looks it up, not where it is defined
  • unittest.mock autospec catches calls with the wrong signature
  • The model is still live, so results vary; assert on behaviour, not wording
  • An unexpected tool call should fail the test, not return a default

4. Prerequisites and access

  • Scenario files from PB-AGT-12-01
  • Agent code with a single tool-call entry point that can be patched
  • Model deployment for tests with its own quota

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Expose one function in the agent that all tool calls go through.
  2. Write a fake that returns recorded results keyed by tool and arguments.
  3. Fail the test on any call the scenario did not record.
  4. Parametrise one test over all scenario files.
  5. Assert tool sequence, step ceiling, forbidden tools and required facts.
  6. Run locally, then add the suite to the pull request pipeline.
  7. Refresh recorded tool results when real tool contracts change.

Draft snippet: Replay scenarios against mocked tools

import json
from pathlib import Path

import pytest

from agent_app import tools
from agent_app.runner import run_agent  # returns {"final": str, "tool_calls": [(name, args), ...]}

SCENARIOS = sorted(Path("tests/scenarios").glob("*.json"))


@pytest.mark.parametrize("path", SCENARIOS, ids=lambda p: p.stem)
def test_scenario(path, monkeypatch):
    scenario = json.loads(path.read_text())
    recorded = {(c["tool"], json.dumps(c["args"], sort_keys=True)): c["result"] for c in scenario["tool_calls"]}

    def fake_call(name, args):
        key = (name, json.dumps(args, sort_keys=True))
        assert key in recorded, f"unexpected tool call {key}"
        return recorded[key]

    monkeypatch.setattr(tools, "call_tool", fake_call)
    result = run_agent(scenario["input"], seed=scenario.get("seed", 0))

    called, expected = [name for name, _ in result["tool_calls"]], scenario["expected"]
    assert called == expected["tool_sequence"]
    assert len(called) <= expected["max_steps"]
    assert not set(called) & set(expected.get("must_not_call", []))
    for phrase in expected["must_mention"]:
        assert phrase.lower() in result["final"].lower()

Snippets to add:

  • snippets/tests/test_scenarios.py: Parametrised scenario test with a recorded-tool fake (python)
  • snippets/pipelines/agent-scenarios.yml: Pipeline job for smoke and nightly suites (yaml)

6. Verify it works

A second engineer wires the harness into the sandbox agent and runs it on 10 scenarios, then introduces a prompt change that skips a tool. Pass: the clean run passes, the changed run fails on the right scenario, and no real tool is called.

7. Known issues and fixes

Symptom Likely cause Fix
Mock is never used and real tools are called Patched the definition module instead of the lookup location Patch the attribute on the module the agent imports from
Tests pass but production fails Recorded tool results out of date Refresh scenarios from recent traces and add contract tests on real tools
Suite too slow for pull requests Every scenario calls the model several times Run a smoke subset on pull requests and the full suite nightly

8. Monitoring and alerts

Signal Alert when Action
Nightly full scenario suite Pass rate drops below the last release value Triage with PB-AGT-12-03 before the next release

9. AWS and open-source equivalents

The same pytest harness runs on any cloud. On AWS, teams pair it with AgentCore on-demand evaluations; DeepEval integrates with pytest for metric assertions.

10. Validation record

Pending. Planned dry-run scenario: A second engineer wires the harness into the sandbox agent and runs it on 10 scenarios, then introduces a prompt change that skips a tool. Pass: the clean run passes, the changed run fails on the right scenario, and no real tool is called.

11. Sources