Skip to content
⚠️ Before You Start
Risk: High
Duration: 2-4 hours
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: High (promotion without evidence lets the agent act unsupervised on untested paths)

Autonomy level promotion checklist using simulation evidence

Draft outline: not validated

Drafted with AI on 16 Oct 2026 and outline-checked against official sources on 11 Nov 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-28). Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: AGT-12-T13, AGT-12-T14, AGT-12-T15, AGT-12-T16.

1. Summary

Use this checklist to decide whether an agent has enough simulation pass history and human-approval evidence to move from suggest-only to act-with-approval, or from act-with-approval to act-and-report. The Agent lead, QE lead and Responsible AI officer run and sign off the gate together; partial or conditional approvals keep high-risk actions gated while lower-risk actions promote for a canary cohort first.

Note on Foundry evaluators: IntentResolution (preview), TaskAdherence (preview) and TaskCompletion (preview) in Microsoft Foundry are marked preview on the documentation page alongside ToolCallAccuracy.

Risk: High (promotion without evidence lets the agent act unsupervised on untested paths). Typical duration: 2-4 hours.

2. When to use and when not to

Use it when:

  • An agent owner asks to move from suggest-only to act-with-approval, or from act-with-approval to act-and-report for one or more tools or task types.
  • You are expanding an already-promoted action from a canary tenant cohort to wider production traffic.
  • You are re-promoting an action after an emergency downgrade (see Respond to an agent acting outside its permission envelope).

Do not use it when:

3. What you must know first

  • Three autonomy tiers govern action authority: Define an agent's permission envelope defines suggest-only, act-with-approval and act-and-report, mapped per action (allow, requireApproval, deny) rather than all-or-nothing.
  • Human rejection, edit and timeout rates measure real-world risk: In act-with-approval, human reviewers can approve, edit arguments or reject before a tool runs (via LangGraph interrupt() and Command(resume=...) backed by a durable checkpointer such as PostgresSaver, or LangChain HumanInTheLoopMiddleware / interrupt_on policies). A low rejection rate paired with frequent argument edits still signals the agent cannot run safely without review.
  • Simulations must test refusal and adversarial paths, not just happy paths: Every action losing a human gate needs passing scenario fixtures (see Turn recorded agent traces into replayable scenarios), negative/refusal cases and prompt-injection tests covering OWASP LLM06:2025 Excessive Agency and OWASP Top 10 for Agentic Applications for 2026 (ASI01 Agent Goal Hijack, ASI02 Tool Misuse & Exploitation, ASI09 Human-Agent Trust Exploitation).
  • Mocks can drift from live tools: Passing mocked scenario runs only predict safe behaviour when mock request/response schemas and side-effect assertions match the current production tool contract (see Run scenario regression tests with mocked tools).
  • Promotion changes metric gates and governance rules: Raising the autonomy tier triggers a threshold re-evaluation under Choose agent evaluation metrics, thresholds and judge models and requires passing the regression gate before rollout.
  • Reversibility comes before elevation: Verified step, loop and cost budgets plus a tested kill switch returning the agent to suggest-only (Azure App Configuration feature flag + Entra/RBAC scope alignment) must be ready before any tier changes.

4. Checklist

Phase A: Scope and scenario evidence

Phase B: Human-in-the-loop (HITL) and safety control evidence

Phase C: Reversibility and phased rollout readiness

Verify rollback before raising autonomy

Do not update the live policy or feature flag until you have tested the kill switch in staging and verified it immediately downgrades the agent to suggest-only (see Respond to an agent acting outside its permission envelope).

  • 7. Test the autonomy kill switch back to the previous tier and suggest-only in staging (Agent lead — toggle the Azure App Configuration feature flag or policy override per Respond to an agent acting outside its permission envelope). Confirm in-flight tool calls pause or abort within the flag cache TTL and do not require redeploying application code.
  • 8. Re-baseline evaluation metrics and alerts for the new autonomy tier (QE lead — update target tolerances and online evaluation sampling per Choose agent evaluation metrics, thresholds and judge models and Online evaluation drift and judge model upgrade). Set the regression-alert trigger so any post-promotion drop triggers immediate downgrade.
  • 9. Restrict rollout to an approved canary cohort first (Agent lead — configure tenant IDs, task categories or traffic percentage in the policy config).

Phase D: Sign-off and post-promotion review

  • 10. Record partial exclusions, conditions and three-role sign-off in the governance log (Agent lead, QE lead, Responsible AI officer — record approved/excluded actions in promotion-record.json and review Agent governance go-live review checklist governance items).
  • 11. Complete a scheduled post-promotion review after 2 weeks (team default) before wider rollout (QE lead & Agent lead — compare live task success, error rates, budget consumption and user override tickets against pre-promotion baselines before removing canary scoping).

Draft snippet: Autonomy promotion checklist (excerpt)

# Autonomy promotion: ${AGENT_NAME}

From: act-with-approval  To: act-and-report  Canary scope: ${CANARY_SCOPE}
- [ ] Actions changing tier listed (irreversible actions excluded)
- [ ] Scenarios cover happy, refusal and ASI01/ASI02 injection cases (link)
- [ ] Suite green 30 days (team default); zero quarantined skips (link)
- [ ] 30-day approval rejection, edit and timeout rates within limit (link)
- [ ] Step/budget guards, audit logs and kill switch verified (link)
- [ ] Post-promotion review booked for 2 weeks (team default)
Decision: approve / approve with exclusions / reject
Signed: Agent lead, QE lead, Responsible AI officer

Snippets to add when writing this playbook:

  • snippets/governance/autonomy-promotion.md: Promotion checklist template with evidence link fields (markdown)
  • snippets/governance/promotion-record.json: Schema-aligned promotion decision and partial-exclusion log record (json)

5. Known issues and fixes

Symptom Likely cause Fix
Not enough approval history for a specific action Low live volume or agent rarely proposes the tool Extend the observation period, target simulated runs for that tool, or exclude that action from promotion while promoting high-volume actions
Low rejection rate hides frequent manual fixes Reviewers edit tool arguments before approving instead of clicking reject Measure argument-edit rate alongside rejection rate; treat high edit rates as unreliable parameter extraction and keep the action gated
High rejection rate on one action only Ambiguous tool description, missing validation or poor edge-case handling Use partial approval (approve with exclusions) to keep that action on requireApproval while promoting the remaining actions
CI passes for 30 days (team default) but failures appear immediately after promotion One or more failing scenarios were quarantined (pytest.mark.skip) or mocks drifted from real downstream APIs Audit skipped scenarios under PB-AGT-12-03 and re-run recorded traces against current tool schemas before sign-off
No negative or injection scenarios in the suite Test suite generated only from successful production transcripts Write refusal fixtures (must_not_call) and indirect prompt-injection scenarios covering ASI01 and ASI02 before promoting
High approval timeout rate during observation window Approvers overwhelmed or notification routing broken (PB-AGT-05-03) rather than bad model output Fix queue routing and SLA alerts first; do not treat unreviewed timeouts as implicit approvals or clean history
Kill switch toggled but agent continues autonomous calls Feature flag cached in-process or read only at container startup Configure short cache expiry and read the autonomy flag dynamically at every tool gate before execution

6. Monitoring and alerts

The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Source Alert when Action
Post-promotion task success / tool accuracy (TaskAdherence preview, ToolCallAccuracy) Online evaluation pipeline (Microsoft Foundry / LangSmith / AgentCore Evaluations) Drops below the agent's regression tolerance (PB-AGT-07-02, PB-AGT-07-03) for 1 day (team default) Flip the kill switch (PB-AGT-08-04) to revert the promoted actions to act-with-approval or suggest-only
Unattended tool executions (autonomous_tool_calls_count, custom metric) OpenTelemetry spans / immutable audit log Exceeds expected canary cohort volume or baseline rate by team tolerance (team default) Pause promotion, inspect recent traces for loop or unexpected trigger conditions (ASI10)
Downstream tool 4xx/5xx or idempotency conflict rate Tool API telemetry / OpenTelemetry spans Spikes above pre-promotion baseline after autonomy upgrade Inspect rejected parameters and compensating-action logs (PB-AGT-09-03); revert offending tool gate
Per-run step or token budget breaches (maxSteps / maxTokens) Budget limiter telemetry (PB-AGT-11-01, PB-AGT-11-04) Above agreed post-promotion warning rate (team default) Triage loop patterns (PB-AGT-09-01) and return action to approval
User override, undo or incident tickets referencing autonomous actions Support / HITL feedback queue Any confirmed unintended external mutation (zero tolerance for high-risk actions) Trigger kill switch immediately (PB-AGT-08-04), capture trace and turn it into a permanent regression scenario (PB-AGT-12-01)

7. AWS and open-source equivalents

Platform Equivalent Key differences
Azure (Microsoft Foundry) Foundry Agent evaluation (IntentResolution preview, TaskAdherence preview, TaskCompletion preview, ToolCallAccuracy), OpenTelemetry trace logs and Azure App Configuration kill switches Foundry evaluators run inside Foundry workflows alongside Azure Monitor telemetry and Entra/App Config governance; confirm preview/GA status of evaluators before hard CI blocking
AWS (Bedrock Agents & AgentCore) Amazon Bedrock Agents RETURN_CONTROL invocation logs (invocationId, invocationInputs), CloudWatch invocation telemetry and Amazon Bedrock AgentCore Evaluations (batch, on-demand and online evaluation) RETURN_CONTROL delegates approval and execution back to the caller before resuming via sessionState; Bedrock AgentCore adds AgentCore Policy (Cedar) and AgentCore Evaluations for production trace scoring. Confirm whether new builds target AgentCore vs Agents Classic
Open-source (LangGraph / Pytest) LangGraph interrupt() and Command(resume=...) backed by a durable checkpointer (PostgresSaver), LangChain HumanInTheLoopMiddleware (interrupt_on rules) and pytest mocked scenario CI logs Full control of the approval decision store (approve, edit, reject, timeout) in your own database and CI runner; you must operate audit storage, RBAC alignment and feature-flag rollback yourself

8. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A second engineer runs this checklist against the sandbox agent (act-with-approval -> act-and-report) with seeded 30-day simulation CI results and approval logs where one action shows a high rejection/argument-edit rate and one scenario carries a skipped negative test.

Pass criteria:

  1. The dry-runner follows Phases A–D and identifies both the high rejection/edit action and the skipped negative test.
  2. The resulting promotion decision excludes the failing action (approve with exclusions) while testing the staged kill switch rollback cleanly in sandbox.
  3. No production data, live secrets or unverified metrics are recorded in the decision log.

Record

Item Result
Date Pending
Dry-runner Pending (planned: Yasar)
Pack owner sign-off Pending (planned: Prasanna)
Outcome Pending
Notes Outline verified; full prose, snippets (autonomy-promotion.md, promotion-record.json) and sandbox dry-run remain in AGT-12-T14 and AGT-12-T15.

9. Sources

Checked on 11 November 2026: