Run the agent regression gate before a prompt, model or tool change¶
Draft outline: not validated
Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-05). Author: Sivaraj · dry-run: Yasar · sign-off: Prasanna. Work items: AGT-07-T05, AGT-07-T06, AGT-07-T07, AGT-07-T08.
1. Summary¶
Before any release that changes a system prompt, model deployment or version, tool schema or orchestration logic. Also after a provider retires a model version and forces an upgrade.
Risk: High (a missed regression ships wrong tool calls or cost blow-ups to every user of the agent). Typical duration: 1-3 hours plus evaluation run time.
2. When to use and when not to¶
Use it when: Before any release that changes a system prompt, model deployment or version, tool schema or orchestration logic. Also after a provider retires a model version and forces an upgrade.
Do not use it when: to be written by the author.
3. What you must know first¶
- Compare against the baseline of the version in production, not the last green run on a branch
- Model upgrades change step counts and token use even when answers look the same
- Tolerances must exceed judge noise, or the gate fails at random
- Cost per task is a release metric, not a finance afterthought
- A tool schema change can break tool selection for unrelated tasks
4. Prerequisites and access¶
- Evaluation suite and committed baseline from PB-AGT-07-01
- CI pipeline with access to the judge deployment and a staging copy of the agent
- Change ticket naming what changed (prompt, model, tool or graph)
- Mocked or sandboxed tools for any action that writes to real systems
5. Steps¶
Outline of the steps; the author adds the commands and a check after each one.
- Confirm the baseline file matches the production agent version tag.
- Deploy the candidate to staging with tools pointed at sandboxes or mocks.
- Run the full evaluation dataset against the candidate.
- Run the gate script to compare task success, tool accuracy, step efficiency and cost per task with the baseline.
- If the gate fails, open the per-case results and list the cases that moved from pass to fail.
- Classify each regression as a real defect, an expected behaviour change or judge noise.
- For expected changes, get agent lead sign-off and record the reason in the change ticket.
- Re-run the gate after fixes; never edit tolerances to make a failing gate pass without sign-off.
- On pass, promote the candidate and save its metrics as the new baseline with the version tag.
- Verify in production: compare the first day of online metrics with the gate results.
- Rollback: redeploy the previous agent version and restore the previous baseline file.
Draft snippet: Gate a candidate against the production baseline
import json
import sys
from pathlib import Path
# Tolerances are team defaults; set them above measured judge noise.
GATES = {"task_success": 0.02, "tool_accuracy": 0.03, "step_efficiency": 0.05}
MAX_COST_INCREASE = 0.10
baseline = json.loads(Path("eval/baseline_metrics.json").read_text())
current = json.loads(Path("eval/current_metrics.json").read_text())
failures = []
for metric, tolerance in GATES.items():
if current[metric] < baseline[metric] - tolerance:
failures.append(f"{metric}: {current[metric]:.3f} vs baseline {baseline[metric]:.3f}")
cost_ratio = current["cost_per_task_usd"] / baseline["cost_per_task_usd"]
if cost_ratio > 1 + MAX_COST_INCREASE:
failures.append(f"cost_per_task up {cost_ratio - 1:.0%}")
if failures:
print("REGRESSION GATE FAILED")
print("\n".join(failures))
sys.exit(1)
print("Regression gate passed")
Snippets to add:
snippets/eval/gate.py: Baseline comparison gate script (python)snippets/pipelines/agent-eval-gate.yml: CI job that runs the suite and the gate (yaml)snippets/docs/regression-triage.md: Regression triage template for the change ticket (markdown)
6. Monitoring and alerts¶
| Signal | Alert when | Action |
|---|---|---|
| Regression gate result in CI | Any gate failure on a release branch | Block the release and notify the agent lead and QE lead |
| Online task success after release | Falls below the gate baseline for 2 consecutive hours | Start rollback and attach the failing traces to the incident |
| Cost per task after release | More than the agreed tolerance above baseline for 1 hour | Check step counts and model version; roll back if not explained |
7. Rollback¶
To be written by the author: how to undo each risky step.
8. Known issues and fixes¶
| Symptom | Likely cause | Fix |
|---|---|---|
| Gate fails on every run with tiny metric drops | Tolerance smaller than judge variance | Measure run-to-run variance on an unchanged agent and set tolerances above it |
| Cost per task doubles after a model upgrade | New model takes more steps or produces longer reasoning | Check step counts per case; tighten the prompt or the step budget before promotion |
| Gate passes but production tool accuracy drops | Staging tools return different data shapes from production | Refresh mocks from recent production traces and add schema contract tests |
| Baseline missing or from the wrong version | Baseline not saved on the last promotion | Recreate it by running the suite against the production version before testing the candidate |
| Judge deployment throttled mid-run | Shared quota with production traffic | Use a dedicated judge deployment or run the gate outside peak hours |
9. AWS and open-source equivalents¶
On AWS, run AgentCore on-demand evaluations in CodePipeline or GitHub Actions and compare scores the same way. Open-source teams use DeepEval in pytest or LangSmith experiments to compare candidate and baseline runs.
10. Validation record¶
Pending. Planned dry-run scenario: In the sandbox, a second engineer runs the gate twice: once with an unchanged agent (must pass) and once with a prompt edit that removes a tool instruction (must fail on tool accuracy). Pass: both outcomes correct, triage notes completed and rollback restores the old baseline.