Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-2 hours
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (a failed rotation breaks every agent that uses the server)

Rotate MCP server credentials without breaking agents

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-05). Author: Sivaraj · dry-run: Sandeep Velu · sign-off: Prasanna. Work items: AGT-10-T09, AGT-10-T10, AGT-10-T11, AGT-10-T12.

1. Summary

When a downstream API key or certificate used by an MCP server nears expiry, after a suspected leak, or when Key Vault raises a SecretNearExpiry event.

Risk: High (a failed rotation breaks every agent that uses the server). Typical duration: 1-2 hours.

2. When to use and when not to

Use it when: When a downstream API key or certificate used by an MCP server nears expiry, after a suspected leak, or when Key Vault raises a SecretNearExpiry event.

Do not use it when: to be written by the author.

3. What you must know first

  • Key Vault publishes SecretNearExpiry 30 days before expiry, which can trigger automated rotation through Event Grid
  • Overlap old and new credentials so running agents do not fail mid-run
  • Prefer managed identity to downstream services so there is nothing to rotate
  • Agent-to-server OAuth tokens are short-lived; rotating the server's client secret or certificate is a separate task
  • Never paste secrets into tickets, logs or shell history

4. Prerequisites and access

  • Key Vault Secrets Officer on the vault, or a rotation function with managed identity
  • Ability to issue a new credential in the downstream system while the old one still works
  • Container Apps contributor rights for the server
  • Smoke test and contract tests for the server

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Issue a new credential in the downstream system and keep the old one active.
  2. Store the new value as a new Key Vault secret version with an expiry date.
  3. Restart the server revision so it reads the new version.
  4. Run the smoke test and contract tests against the server.
  5. Watch tool error rates for 30 minutes.
  6. Revoke the old credential in the downstream system.
  7. Confirm error rates stay flat after revocation.
  8. Record the rotation date and next expiry in the catalog entry.
  9. Rollback: before revocation, disable the new secret version so the previous one is used, and restart the revision.

Draft snippet: Store the new credential and restart the active revision

#!/usr/bin/env bash
set -euo pipefail
# NEW_KEY is produced by your downstream system; never echo it
NEW_KEY="$(./scripts/issue-downstream-key.sh)"
EXPIRES="$(date -u -d '+90 days' +%Y-%m-%dT%H:%M:%SZ)"

az keyvault secret set --vault-name "${KV_NAME}" --name "${SECRET_NAME}" \
  --value "$NEW_KEY" --expires "$EXPIRES" --output none
unset NEW_KEY

REVISION="$(az containerapp revision list -n "${APP_NAME}" -g "${RG}" \
  --query "[?properties.active].name | [0]" -o tsv)"
az containerapp revision restart -n "${APP_NAME}" -g "${RG}" --revision "$REVISION"

# Smoke test: your health route plus the contract tests
curl -fsS "${MCP_HEALTH_URL}" > /dev/null && echo "health ok"

Snippets to add:

  • snippets/ops/rotate-mcp-secret.sh: Set new secret version and restart the server revision (bash)
  • snippets/ops/kv-near-expiry-subscription.json: Event Grid subscription filter for SecretNearExpiry (json)

6. Monitoring and alerts

Signal Alert when Action
Key Vault SecretNearExpiry event Any MCP server secret Start this runbook or check the automated rotation ran
Tool 401 or 403 rate after rotation Above baseline for 10 minutes Roll back to the previous secret version
Secrets with no expiry date Any MCP server secret Set an expiry and add it to the rotation schedule

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Server still uses the old credential Secret value cached at start-up or pinned to a version Restart the revision and reference the secret without a fixed version
Tool calls fail with 401 after revocation One replica or a second server still had the old value Restart all revisions and replicas, then revoke
Rotation function failed silently Event Grid retries exhausted or missing role assignment Check function logs and give its identity the secrets role
Secret visible in pipeline logs Value echoed by a script Revoke immediately, rotate again and mask variables

9. AWS and open-source equivalents

On AWS, Secrets Manager rotation with Lambda does the same job, and AgentCore Identity can hold outbound credentials for tools. Open-source stacks use HashiCorp Vault dynamic secrets.

10. Validation record

Pending. Planned dry-run scenario: In the sandbox, a second engineer rotates the demo server's downstream key while a scripted agent calls tools every 5 seconds. Pass: no tool call fails during rotation, the old key is revoked and the rollback path is shown to work before revocation.

11. Sources