Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-2 hours per credential
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (a failed rotation breaks every agent that uses the tool)

Rotate credentials for MCP servers and tool connectors without downtime

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-10-30). Author: James · dry-run: Sanjay Sarvesh · sign-off: Sivaraj. Work items: AGT-02-T05, AGT-02-T06, AGT-02-T07, AGT-02-T08.

1. Summary

On a scheduled rotation, when Key Vault raises SecretNearExpiry, after a suspected leak, or when a downstream provider forces a key change.

Risk: High (a failed rotation breaks every agent that uses the tool). Typical duration: 1-2 hours per credential.

2. When to use and when not to

Use it when: On a scheduled rotation, when Key Vault raises SecretNearExpiry, after a suspected leak, or when a downstream provider forces a key change.

Do not use it when: to be written by the author.

3. What you must know first

  • Prefer Entra managed identity; rotate keys only where the downstream needs them
  • Two-key systems allow overlap: rotate the alternate key, then switch
  • Key Vault publishes SecretNearExpiry to Event Grid before expiry
  • MCP servers must not pass through tokens they receive from clients
  • Restarting the server is what makes it read the new secret version unless it reloads on its own

4. Prerequisites and access

  • Key Vault Secrets Officer on the vault
  • Contributor on the MCP server hosting resource
  • Access to the downstream provider to create and revoke keys
  • Change ticket approved
  • Smoke test script for the server

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Identify every MCP server and connector that uses the credential from the registry
  2. Create the new credential at the provider while the old one is still valid
  3. Store it as a new Key Vault secret version with expiry and tags
  4. Restart or reload the MCP server revision and confirm it read the new version
  5. Run the smoke test: list tools and call one read-only tool
  6. Watch tool error rates for 30 minutes
  7. Revoke the old credential at the provider
  8. Rollback: re-enable the previous secret version and restart if errors rise before revocation
  9. Update the rotation log and next due date

Draft snippet: Rotate an MCP server credential in Key Vault and restart

#!/usr/bin/env bash
# Rotate the API key an MCP server uses for its downstream system.
set -euo pipefail
VAULT="${KEY_VAULT_NAME}"
SECRET="${SECRET_NAME}"
RG="${RESOURCE_GROUP}"
APP="${MCP_CONTAINER_APP}"

# 1. Store the new credential as a new secret version with an expiry date.
az keyvault secret set --vault-name "$VAULT" --name "$SECRET" \
  --value "$NEW_CREDENTIAL" \
  --expires "$(date -u -d '+60 days' +%Y-%m-%dT%H:%M:%SZ)" \
  --tags rotatedBy="$USER" ticket="${CHANGE_TICKET}"

# 2. Restart the active revision so the server reloads the secret.
REV=$(az containerapp revision list -n "$APP" -g "$RG" --query "[?properties.active].name | [0]" -o tsv)
az containerapp revision restart -n "$APP" -g "$RG" --revision "$REV"

# 3. Smoke test: list tools and call one read-only tool through the gateway.
python scripts/mcp_smoke.py --server "${MCP_URL}" --tool "${READ_ONLY_TOOL}"

Snippets to add:

  • snippets/ops/rotate_mcp_secret.sh: Rotate secret and restart MCP server (bash)
  • snippets/scripts/mcp_smoke.py: MCP smoke test (python)
  • snippets/infra/eventgrid-secret-near-expiry.json: Event Grid subscription filter for SecretNearExpiry (json)

6. Monitoring and alerts

Signal Alert when Action
Key Vault SecretNearExpiry event Event received Schedule rotation within the next working day
Tool 401 or 403 responses per server More than 5 in 5 minutes Check recent rotations and roll back the secret version if needed
Secrets with no expiry date Any in agent vaults (policy scan) Set expiry and add to the rotation calendar

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
401 from the downstream API after rotation Server still caches the old key Restart the active revision and check it reads the latest secret version
Other agents fail although only one was rotated Credential was shared across servers Split credentials per server and record ownership in the registry
Rotation function did not run Event Grid subscription missing or failing Check subscription delivery logs; rotate manually with this runbook
Account-level SAS tokens stop working Storage key regenerated Move consumers to Entra auth or re-issue SAS from the new key

9. AWS and open-source equivalents

On AWS store credentials in Secrets Manager with rotation, or use AgentCore Identity credential providers for OAuth and API keys. Self-hosted servers can use HashiCorp Vault dynamic secrets.

10. Validation record

Pending. Planned dry-run scenario: In a sandbox, a second engineer rotates a dummy API key for a sample MCP server with live agent traffic and passes when no tool call fails during the switch and the old key is revoked within the window.

11. Sources