Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 hours
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Medium (skipped checks surface as incidents after release)

Agent go-live checklist

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-18). Author: Sanjay Sarvesh · dry-run: Yasar · sign-off: James. Work items: AGT-01-T13, AGT-01-T14, AGT-01-T15, AGT-01-T16.

1. Summary

This checklist validates that an agent is reliable, observable, secure, and recoverable before releasing it to production. It applies to any agent built from the four library templates (task, planner, supervisor, or tool agent) across supported runtimes (Microsoft Foundry Agent Service prompt agents, LangGraph, or Microsoft Agent Framework) and hosting models (Foundry hosted agent, Azure Container Apps, or AKS).

It confirms the approved decision record from PB-AGT-01-01, durable state persistence, explicit stop rules and execution bounds, tool limits (128 tools maximum), identity and security, telemetry and tracing, quota headroom, evaluation thresholds, and tested rollback procedures. Run this checklist during the pre-release operational review meeting with the agent owner, DevOps engineer, and Site Reliability Engineer (SRE).

Facts checked against Microsoft Learn, LangChain, and AWS documentation on 6 November 2026. Voice-based agents and Agent Framework for Go are preview. The docs name no API version for either.

Risk: Medium (skipped checks surface as incidents after release). Typical duration: 1-2 hours.

2. When to use and when not to

Use this checklist when:

  • Before the first production release of any agent built from the library templates.
  • Before releasing a major update (for example, runtime upgrade, graph topology changes, prompt or model updates, or new tool integrations).
  • In the pre-release operational review meeting with the agent owner, DevOps engineer, and SRE.

Do not use it when:

  • You choose the agent template, runtime, or hosting model: use PB-AGT-01-01.
  • You build a task agent from the template: use PB-AGT-01-02.
  • You upgrade the framework or model version of a running agent: use PB-AGT-01-04.
  • A plain function or fixed workflow can do the job without an agent loop: use PB-AGT-03-01.
  • Shipping a prompt-only change that does not alter tool schemas, state keys, or graph topology: use PB-AGT-03-04.
  • Triaging an active production incident. To stop or triage a looping or failing agent, follow PB-AGT-09-01.
  • Setting up the checkpointer and store on Azure: use PB-AGT-04-02.
  • Reviewing per-tool permissions and security: use PB-AGT-02-04.

3. What you must know first

  1. A signed-off decision record is mandatory. Every production agent must have a committed decision record (docs/decisions/agent-template-choice.yaml) that specifies the template, runtime, hosting, and checked limits, signed off by the Agent Lead per PB-AGT-01-01. The library provides four templates (task, planner, supervisor, and tool agent). Who owns the control loop, state, and stop rule must be unambiguous before release.
  2. In-memory checkpointers lose state on restart. Classes such as InMemorySaver or MemorySaver in LangGraph, and InMemoryCheckpointStorage in Microsoft Agent Framework, store checkpoints in volatile RAM. They are lost whenever the process restarts or the container redeploys. Production agents must configure a durable checkpointer (such as PostgresSaver for LangGraph or CosmosCheckpointStorage for Agent Framework in Python) and run database table setup or migrations before receiving live traffic. Note that Agent Framework documentation currently shows in-memory checkpoint storage only for .NET. (LangGraph: Persistence, Agent Framework checkpoints)
  3. Hosted agent session lifecycle and the 30-day deletion rule. Microsoft Foundry permanently deletes a hosted agent session after 30 days of inactivity. An agent's idle timeout is configurable per agent version from 2 to 240 minutes (default 15 minutes), after which Foundry deallocates compute while retaining session state. Long-running human approvals (PB-AGT-05-01) or multi-day workflows must persist approval state and thread tokens in a durable database checkpointer, not solely in ephemeral session compute. (Hosted agents in Foundry Agent Service, LangGraph: Interrupts)
  4. Every agent requires explicit stop rules and execution limits. Default execution limits vary across framework versions and must never be trusted. LangGraph agents require an explicit recursion_limit in the run config. Each template requires concrete stopping criteria:
  5. Task agent: done flag, step budget, and token budget.
  6. Planner agent: plan complete check and bounded replans (max_replans).
  7. Supervisor agent: hop limit (max_hops across specialist workers) to prevent circular delegation.
  8. Tool agent: single-turn execution contract.
  9. Fixed workflows / evaluator-optimizer: bounded loop iterations. (LangGraph: Graph API overview, PB-AGT-01-01)
  10. Tool quotas and registry contracts. A Foundry agent can register at most 128 tools; this is a fixed platform limit. Every tool must have an entry in the tool registry with versioning, Pydantic schemas, an error contract, and least-privilege RBAC scopes. (Foundry limits and quotas, PB-AGT-02-01, PB-AGT-02-04)
  11. Managed identity is required for Azure resources. Microsoft recommends Microsoft Entra managed identities over long-lived API keys or secrets stored in application settings. Foundry hosted agents use a dedicated agent identity; Container Apps and AKS workloads use a managed identity (system-assigned or user-assigned; user-assigned is an Orion default). The Application Insights connection string is not a secret: its instrumentation key is not a security token (Connection strings). (Managed identities overview, Hosted agents)
  12. Tracing captures inputs and completions. Treat trace data as production telemetry and apply the same access controls and retention as logs. Foundry hosted agents send traces to Application Insights by default only when using official protocol libraries (Hosted agents); custom code requires OpenTelemetry and the Azure Monitor exporter. To restrict sensitive content, route it to the protected table and grant the Privileged Monitoring Data Reader role only to people who must read it (Restrict access to sensitive content). Configure trace sampling in production to avoid high telemetry ingestion costs. (Microsoft Foundry: Agent tracing overview, OpenTelemetry in Application Insights)
  13. Concurrent session quotas and model rate limits. Hosted agent calls fail with session_quota_exceeded or regional_session_quota_exceeded (HTTP 429) when concurrent session quotas for the subscription and region are exhausted. Ensure Azure OpenAI TPM/RPM and hosted agent session quotas comfortably accommodate anticipated peak concurrency. (Foundry limits and quotas, Azure OpenAI quotas and limits)
  14. Preview features require an agreed fallback. Voice-based agents and Agent Framework for Go are preview. The documentation names no API version for either. Any production agent using preview capabilities must have an agreed fallback and Agent Lead sign-off recorded before release.

Prerequisites

  • Committed decision record (PB-AGT-01-01) approved by the Agent Lead
  • Release candidate container image built, tagged immutably, and deployed to staging
  • Trajectory evaluation report meeting accuracy, groundedness, and safety thresholds (PB-AGT-07-01)
  • Metric alert rules provisioned in the monitoring repository and tested
  • Tested rollback plan with previous image, prompts, and database migration scripts

4. Checklist

Walk through the operational checklist with the agent owner, DevOps engineer, and SRE. Attach verifiable evidence (links to repository files, configuration values, automated test runs, dashboards, or traces) to the release ticket.

Do not release without durable checkpointers and execution limits

Deploying with an in-memory checkpointer (MemorySaver, InMemorySaver, or InMemoryCheckpointStorage) loses conversation state across container restarts or idle deallocations. Deploying without strict execution bounds (recursion_limit, token budgets, hop limits) risks infinite loops that exhaust model quotas and incur substantial API costs.

Review and sign-off procedure

  • Step 1: Walk the checklist. Convene the release review meeting with the agent owner, DevOps engineer, and SRE.
  • Verify: All three required roles attend or provide recorded written reviews.
  • Step 2: Verify the decision record. Confirm that the decision record exists, names the template, runtime, and hosting, and has Agent Lead approval.
  • Verify: docs/decisions/agent-template-choice.yaml is committed and approved_by is set.
  • Step 3: Attach evidence. Record configuration links, test run results, and dashboard URLs against each checklist item in the change record.
  • Verify: No item is marked complete without a verifiable artefact link.
  • Step 4: Enforce blockers. Raise an immediate release blocker if any item under Architecture, State, Bounds, Tools, Identity, or Rollback fails validation.
  • Verify: Change ticket status is marked "Blocked" pending remediation.
  • Step 5: Execute staging trace and concurrency validation. Dispatch representative user requests and concurrent sessions under separate thread_id values in staging and inspect resulting traces in Application Insights.
  • Verify: Span attributes contain thread_id, agent version, tool invocations, no leaked secrets or raw reasoning, and no state cross-contamination.
  • Step 6: Validate alert delivery. Fire a synthetic test alert to confirm notification delivery to the on-call channel.
  • Verify: Alert webhook or pager notification is delivered to the on-call channel (target time TO VERIFY with the SRE team).
  • Step 7: Record sign-off and authorise release. Obtain formal sign-off in the change ticket before cutting over production traffic.
  • Verify: Sign-off timestamp and approver role recorded in the deployment log.

Go-live verification checklist

The full itemised checklist with assigned checker roles and required verification evidence is maintained in the pack's snippets directory:

# Agent go-live verification summary (template: ${TEMPLATE_NAME}, runtime: ${RUNTIME_NAME})
# Full verification criteria with roles and evidence: snippets/checklists/agent-go-live.md
1. Decision record and architecture: Agent Lead verifies approved template and runtime.
2. State and persistence: DevOps/GenAI Engineer verifies durable DB checkpointer setup.
3. Execution bounds and stop rules: GenAI/QE Engineer verifies recursion and token limits.
4. Tools, identity, and security: Security/DevOps verifies Entra identity and least privilege.
5. Observability and quotas: SRE verifies OpenTelemetry traces, alerts, and quota headroom.
6. Release, evaluation, and recovery: QE/Agent Lead verifies golden baseline and rollback SLA.

Snippet:

  • snippets/checklists/agent-go-live.md: Full go-live verification checklist with checker roles and evidence

5. Known issues and fixes

Symptom Likely cause Fix
Agent loses conversation context after pod or container restart In-memory checkpointer (InMemorySaver, MemorySaver, or InMemoryCheckpointStorage) left enabled from development Switch to PostgresSaver or CosmosCheckpointStorage and run table setup() or migrations before restarting the service.
A hosted agent loses a waiting approval Foundry deleted the session after 30 days of inactivity Keep approval state in a durable checkpointer (PostgresSaver or Cosmos DB), not in ephemeral session compute.
Cannot register more tools on a Foundry agent 128 tools per agent is a fixed platform limit Split tools across agents using a supervisor topology (PB-AGT-06-02) or narrow the registered toolset.
Hosted agent calls fail with session_quota_exceeded or regional_session_quota_exceeded (HTTP 429) The concurrent session quota for the subscription and region is exhausted Spread sessions across regions, request a quota increase, or reduce peak concurrency. Check current limits in the docs.
Azure OpenAI returns HTTP 429 errors or agent stalls under load Azure OpenAI TPM (tokens per minute) or RPM (requests per minute) quota exceeded Request a quota increase, or configure exponential back-off retries.
No traces appear in Application Insights Default traces come only from agents using protocol libraries, or connection string is missing Set APPLICATIONINSIGHTS_CONNECTION_STRING and verify the exporter. On a Foundry hosted agent, use the protocol libraries or add the Azure Monitor OpenTelemetry exporter.
Alerts never fire Rules reference a metric name the app does not emit Trigger a test event and fix the rule
Application Insights ingestion cost spikes dramatically Verbose tracing enabled capturing all prompt payloads, completions, and intermediate states in production Configure trace sampling or mask high-cardinality payloads, retaining full payload capture only in staging.
A .NET workflow cannot resume after a restart The official documentation shows in-memory checkpoint storage only for .NET Plan custom durable storage implementation, or use Python with CosmosCheckpointStorage.
A Go agent cannot use declarative agents, CodeAct or functional workflows These features are not implemented in Agent Framework for Go (preview) Use Python or .NET for those capabilities, or record an agreed fallback.
Agent loops indefinitely or burns tokens without progressing Missing recursion_limit, absent supervisor hop limit, or faulty done flag Set recursion_limit explicitly in run config, enforce max_hops or bounded replans, and apply emergency triage via PB-AGT-09-01.
Agent runs but ignores registered tools Tool definitions missing from model invocation parameters or schema mismatch between prompt and code Verify tool definitions and Pydantic schemas match the deployed runtime and prompt specification.
Agent outputs internal reasoning or tool call details to users Output formatting node missing or structured output schema not filtering system properties Add a final formatting node or output guard that strips private keys, reasoning blocks, and raw tool payloads.
Client security or platform team rejects the deployment Hosting model and identity constraints were not aligned with client rules before build Revisit the decision record from PB-AGT-01-01, record identity and network constraints, and obtain Agent Lead sign-off.

6. Monitoring and alerts

The thresholds below are Orion team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Alert when Action
Release without completed checklist in change record Any production deployment Release manager or automated deployment gate blocks the release pipeline.
Agent error rate in first 24 hours Above agreed SLO (Orion starting baseline: 2% of total runs) Agent owner reviews failure traces in Application Insights and evaluates rollback.
Recursion limit reached (recursion_limit breaches) Any occurrence (Orion team default) SRE investigates looping graph or tool failure cycles; apply PB-AGT-09-01.
Hosted agent session quota errors (session_quota_exceeded or regional_session_quota_exceeded HTTP 429) Any occurrence (threshold TO VERIFY with the agent lead) Plan capacity, spread sessions across regions, or throttle upstream concurrency. Check current limits in the docs.
Azure OpenAI 429 rate limit errors Sustained 429 responses (window TO VERIFY against quota headroom) Review concurrency and TPM consumption, or throttle upstream requests.
Production agents using a preview feature without recorded fallback Any occurrence in production inventory Record the fallback immediately, or roll back off the preview feature.
Decision records past review_by Any record overdue (threshold TO VERIFY with the agent lead) Agent lead re-confirms the choice or reopens it with PB-AGT-01-01.
Agent end-to-end latency (p95) Above the agreed latency SLO (value and window TO VERIFY) Review tool response times, model completion delays, or network latency to external dependencies.
Daily trace ingestion volume Above the agreed baseline (threshold TO VERIFY) Audit telemetry sampling rules and reduce trace verbosity in production.

7. AWS and open-source equivalents

  • AWS: On AWS the same items map to AgentCore Runtime, Memory and Observability in CloudWatch (What is Amazon Bedrock AgentCore?). AgentCore Harness provides a managed agent loop invoked with a single API call. Amazon Bedrock Agents Classic is closed to new sign-ups; do not choose it for new work (Amazon Bedrock Agents).
  • Open source: For open source, LangSmith or an OpenTelemetry collector replaces Application Insights. Self-hosted PostgreSQL (PostgresSaver) provides durable persistence.

8. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A second engineer applies the checklist to a staging agent seeded with three faults (in-memory checkpointer, missing alert, no budget) and must find all three.

Pass criteria: every planted fault is caught by an item.

Record

Item Detail
Playbook PB-AGT-01-05 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

9. Sources

Checked on 6 November 2026.