Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 30 minutes to 3 hours
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (ignoring real failures ships regressions; chasing noise wastes days)

Triage a failing or flaky scenario run in CI

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-14). Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: AGT-12-T09, AGT-12-T10, AGT-12-T11, AGT-12-T12.

1. Summary

When the scenario suite fails on a pull request or the nightly run, or a scenario passes and fails without code changes.

Risk: Medium (ignoring real failures ships regressions; chasing noise wastes days). Typical duration: 30 minutes to 3 hours.

2. When to use and when not to

Use it when: When the scenario suite fails on a pull request or the nightly run, or a scenario passes and fails without code changes.

Do not use it when: to be written by the author.

3. What you must know first

  • A failure is deterministic, flaky, or caused by stale recordings
  • Model version changes on the test deployment cause sudden suite-wide shifts
  • Quarantine is temporary and every quarantined scenario has an owner
  • Never edit the expected block to make a test pass without agent lead review

4. Prerequisites and access

  • Access to CI logs and the test model deployment
  • Quarantine list and ticket queue for flaky scenarios
  • Recent changes to prompts, tools and model deployments

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Note which scenarios failed and on which assertion.
  2. Rerun each failed scenario 5 times to classify it as stable fail, flaky or pass.
  3. For stable failures, find the change that caused them and treat them as regressions.
  4. For flaky scenarios, check whether the assertion depends on wording or order the agent may vary.
  5. For unexpected tool call failures, check whether the tool contract changed and the recording is stale.
  6. Check the test model deployment version and quota errors.
  7. Fix, re-record or quarantine with a ticket and owner.
  8. Rerun the full suite and confirm the pass rate.
  9. Rollback: revert the pull request if a stable regression cannot be fixed the same day.

Draft snippet: Classify a scenario as stable, flaky or failing

#!/usr/bin/env bash
set -uo pipefail
SCENARIO="${1:?scenario name}"
RUNS="${RUNS:-5}"
pass=0
for i in $(seq 1 "$RUNS"); do
  if pytest "tests/test_scenarios.py::test_scenario[${SCENARIO}]" -q -p no:cacheprovider \
      > "triage-${SCENARIO}-${i}.log" 2>&1; then
    pass=$((pass + 1))
  fi
done
echo "${SCENARIO}: ${pass}/${RUNS} passed"
if [ "$pass" -eq "$RUNS" ]; then
  echo "stable pass"
elif [ "$pass" -eq 0 ]; then
  echo "stable failure: treat as a regression"
else
  echo "flaky: fix the assertion or quarantine with a ticket"
fi

Snippets to add:

  • snippets/ops/triage-scenario.sh: Rerun one scenario N times and classify it (bash)
  • snippets/docs/quarantine-ticket.md: Quarantine ticket template (markdown)

6. Monitoring and alerts

Signal Alert when Action
Scenario suite pass rate on main Below the last release value Assign triage the same day
Quarantined scenarios More than 10% of the suite or any older than 14 days Escalate to the QE lead

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Many scenarios fail at once overnight Test model deployment version changed Pin the deployment version and re-baseline with agent lead sign-off
Scenario passes 3 of 5 runs Assertion depends on optional steps Use superset or unordered expectations for that step
Unexpected tool call errors after a tool update Recorded arguments no longer match Re-record the scenario from a fresh trace
Suite fails with throttling errors Shared quota with other jobs Use a dedicated test deployment or lower parallelism

9. AWS and open-source equivalents

The triage steps are cloud-neutral. On AWS, pin Bedrock model versions for test runs; pytest plugins such as pytest-rerunfailures can help classify flakiness.

10. Validation record

Pending. Planned dry-run scenario: In the sandbox, a second engineer is given a suite with one real regression, one flaky scenario and one stale recording. Pass: all three are classified correctly within 2 hours and each gets the right action.

11. Sources