Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 30-60 minutes per change
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Silent regressions in planning accuracy, excessive replanning, or runaway LLM token costs reach production undetected

Planner and prompt change regression checklist

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Verified and expanded under AGT-03-T13 on 23 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-21). Author: Sanjay Sarvesh · dry-run: Yasar · sign-off: James. Work items: AGT-03-T13, AGT-03-T14, AGT-03-T15, AGT-03-T16.

1. Summary

This checklist verifies that any pull request modifying an agent's planner prompt, planning pattern, underlying LLM, or tool registry undergoes rigorous regression testing prior to production deployment. It mandates running an automated offline evaluation experiment against a golden regression dataset, benchmarking task success rate, model invocation count, replan count, and latency against the production baseline. The outcome is the prevention of silent reasoning degradations, unhandled tool failures, and runaway token costs before code merge.

2. When to use and when not to

Use this checklist when:

  • A pull request changes any planner, executor, or replanner system prompt, few-shot examples, or output instructions.
  • A pull request switches the planning topology (e.g. from single-loop ReAct to Plan-and-Execute, or introducing a supervisor).
  • A pull request upgrades, downgrades, or swaps the underlying LLM model deployment (e.g. gpt-4o-2024-05-13 to gpt-4o-2024-08-06, or switching between Azure OpenAI and Anthropic models).
  • A tool is added, modified, deprecated, or removed in the agent's tool registry.
  • A production incident occurs involving planner loops or bad decisions, requiring a reproducing test case addition and bugfix validation.

Do not use it when:

  • The pull request contains only non-functional documentation, code comments, or styling changes that do not alter prompts, schemas, or agent code.
  • Routine infrastructure or dependencies updates occur (e.g. upgrading database client libraries) without changes to model calls or graph topology.
  • In-flight live incident response: follow PB-AGT-03-03 (Investigate bad plans, replan loops and planner cost spikes) to mitigate live outages first; apply this checklist to the subsequent post-mortem bugfix PR.

3. What you must know first

  1. Offline evaluation against a Golden Regression Dataset. Changes to non-deterministic LLM reasoning prompts cannot be validated by traditional unit tests alone. Engineers must execute the candidate branch against a versioned regression dataset (size agreed with the pack owner; TO VERIFY) and benchmark output quality, tool calls, and completion metrics against the baseline experiment using LangSmith or Azure AI Evaluation SDK.
  2. Dual-metric evaluation: Quality AND Cost. High task accuracy can conceal massive cost regressions. Evaluation runs must capture both task completion rate and operational cost metrics: median and p95 LLM calls per task, total token consumption, and replan count.
  3. Trajectory and tool-calling validation. Evaluating only the final answer is insufficient for agent workflows. Evaluators must inspect the execution trajectory: did the planner invoke the correct tools in the right sequence, pass valid Pydantic arguments, and avoid circular loops?
  4. Failure case ingestion ("Incident to Test"). Every production failure, infinite loop, or prompt deviation must generate a sanitised, reproducible test case added to the golden dataset before closing the incident ticket.
  5. Auditable PR evidence gate. Checklists must not be self-certified on trust. The pull request description must include a direct link to the comparative evaluation experiment in LangSmith or Microsoft Foundry showing candidate metrics alongside the production baseline.

Prerequisites

  • Versioned regression dataset
  • Baseline experiment for the current production version
  • PR template that includes this checklist section

4. Checklist

Phase 1: Pre-Submission Regression Experiment (Author)

  • Golden dataset execution: Run the candidate branch against the versioned golden regression dataset using the evaluation runner (evaluate() in LangSmith, Microsoft Foundry evaluation or open-source evaluators).
  • Task completion rate gate: Candidate task success rate is at or above baseline, or within the tolerance agreed with the pack owner (TO VERIFY).
  • Tool trajectory check: Verify tool selection precision and argument schemas match registered tool contracts (PB-AGT-02-01).
  • Replan & step limit audit: Replan count per task stays within the configured MAX_REPLANS, and no GraphRecursionError is raised.
  • Token budget & cost check: Verify that median and p95 token consumption and model invocation counts remain within the agent's cost budget relative to baseline (threshold TO VERIFY).
  • Incident regression case added: If addressing a prior bug or incident, verify the reproducing trace is added to the golden dataset and passes on the candidate branch.
  • Client expectation & scope update: If the prompt change restricts capabilities or adds constraints, verify release notes state what the agent will not do autonomously.

Phase 2: Code & Configuration Review (Reviewer)

  • Evaluation experiment link verified: Click and inspect the comparative evaluation experiment run linked in the pull request description.
  • Pydantic schema validation: Planning and replanning output schemas use strict Pydantic models with Literal bounds and structured output mode.
  • Tool registry authorization: Any newly added or updated tool has a registered entry in the Tool Registry (PB-AGT-02-01) with least-privilege scoping.
  • Reversible rollback plan: The PR includes instructions to revert the model/prompt deployment without requiring database schema rollbacks.
  • Merge decision: Merge only when every item passes, or when an exception is approved by the agent lead and recorded in the PR with the reason.

Phase 3: Post-Merge Verification & Canary Deployment (Deployer / On-Call)

  • Canary deployment or shadow evaluation: Deploy candidate prompt to a canary slice (size TO VERIFY with the deployer) or shadow traffic pipeline.
  • Real-time error rate check: Monitor online trace logs for the agreed observation window; verify zero BadRequestError (HTTP 400), Pydantic ValidationError or unhandled exceptions.
  • Sign-off ticket updated: Record evaluation experiment ID and PR link in the deployment change ticket.

Draft Snippet: PR Review Template Section

# Planner change review (PR template section)

- Change type: [ ] prompt  [ ] planner pattern  [ ] model  [ ] tool set
- Linked evaluation experiment: ${EXPERIMENT_URL}

## Evaluation Gates (Must pass before merge)
- [ ] Golden regression dataset: ${DATASET_NAME} (version: ${DATASET_VERSION})
- [ ] Task success rate on regression dataset: Candidate ${CANDIDATE_ACCURACY}% >= Baseline ${BASELINE_ACCURACY}%
- [ ] Median and p95 model calls per task: ${MEDIAN_CALLS} / ${P95_CALLS} (within budget)
- [ ] Replan count per task within configured maximum (${MAX_REPLANS})
- [ ] No new tool is reachable without a Tool Registry entry and scope (PB-AGT-02-01)
- [ ] Failure cases from the last incident are in the dataset and pass

## Merge Decision
- [ ] Merge only when every item passes, or when an exception is approved by the agent lead and recorded in the PR with the reason

## Client Expectation & Scope
- [ ] Release note states what the agent will not do on its own

Snippets to add

  • snippets/.github/pull_request_template.md: Standardised PR checklist section for planner and prompt changes (markdown)
  • snippets/eval/run_regression_eval.py: Automated evaluation script executing candidate vs baseline comparative runs (python)

5. Known issues and fixes

Symptom Likely cause Fix
Checklist ticked without real experimental evidence Reviewers approve PRs without verifying links Add automated GitHub Action status check that fails PR if EXPERIMENT_URL is empty; require reviewer to confirm baseline comparison link.
Golden dataset drifts and no longer reflects production edge cases Dataset is static and not updated after releases Implement monthly dataset grooming; automatically export flagged low-scoring production traces from LangSmith into the evaluation dataset.
Evaluation passes on small dataset but costs explode in production Golden dataset only contains simple short queries, omitting multi-turn edge cases Include long-context and multi-step complex tasks in the regression suite; evaluate p95 token consumption.
Flaky evaluation scores due to LLM non-determinism High model temperature during evaluation runs Fix temperature=0 for evaluation runs, or run each test case several times (experiment repetitions) and evaluate majority voting / average score.
Tool calling breaks silently after prompt edit Prompt wording inadvertently suppressed tool invocation syntax Inspect trajectory evaluation scores; verify tool calling frequency and argument validity across all test cases.

6. Monitoring and alerts

Signal Alert when Action
Merged planner changes without evaluation experiment link Any occurrence (>0) in weekly audit Agent Lead conducts audit with author; block subsequent merges until verified.
Online task success rate drop post-merge Drops 5 points within 48 hours Immediately revert prompt/model change to previous Git commit and rollback canary.
Average replans per task post-release Above the agreed baseline after release (threshold TO VERIFY) Review production traces in LangSmith; investigate prompt regression or tool latency.
Cost per agent task post-release Above the agent's cost budget after release (threshold TO VERIFY) Audit token usage; verify prompt compression and step granularity.

7. AWS and open-source equivalents

The checklist is tool-neutral; use LangSmith, Foundry evaluation or open-source evaluators to produce the evidence.

Azure

  • Microsoft Foundry & Azure AI Evaluation SDK: azure-ai-evaluation package provides evaluators for groundedness, relevance, coherence, and tool calling across Microsoft Foundry and local runs (Azure AI Evaluation SDK (Foundry classic)).

AWS Equivalents

  • Amazon Bedrock Model & Agent Evaluation: Run automated evaluation jobs evaluating task accuracy, robustness, and custom metrics using Amazon Bedrock evaluation datasets (Amazon Bedrock model evaluation).
  • CloudWatch Metrics & Bedrock Traces: Track invocation latency, error counts, and token utilization per agent version.

Open-Source Equivalents

  • LangSmith Offline & Online Evaluation: Comprehensive test evaluation runner (evaluate()) with dataset versioning, custom LLM judges, and side-by-side run comparisons.
  • DeepEval / Ragas / Promptfoo: Open-source evaluation frameworks running automated CI/CD assertion tests against LLM prompts and agent trajectories.

8. Validation record

Pending. Planned dry-run scenario: A second engineer reviews two sandbox PRs—one clean PR and one containing a hidden cost regression (an updated prompt causing redundant replan cycles). The engineer must successfully identify and block the regressing PR using only the checklist criteria and experiment evidence.

9. Sources