Skip to content
⚠️ Before You Start
Risk: High
Duration: 1-2 hours plus a 24-hour watch
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: High (a wrong counter key or limit throttles every tenant or none)

Roll out or change API Management token limits per tenant

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Outline checked against the listed sources on 03 Nov 2026 (AGT-11-T05). Claims marked verify still need confirming by the author. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-01). Author: Sandeep Velu · dry-run: Yasar · sign-off: Sivaraj. Work items: AGT-11-T05, AGT-11-T06, AGT-11-T07, AGT-11-T08.

1. Summary

Use this runbook to roll out, or change, per-tenant token limits in Azure API Management (APIM). It uses the llm-token-limit policy for a tokens-per-minute rate and a monthly token quota, and the llm-emit-token-metric policy for per-tenant token metrics.

The gateway limit is a backstop. The agent's own budgets in code (PB-AGT-11-01) stop a run first and show the user a clear message. Set gateway limits slightly above the code limits. All changes go through source control and the deployment pipeline.

Risk: High (a wrong counter key or limit throttles every tenant or none). Typical duration: 1-2 hours plus a 24-hour watch.

2. When to use and when not to

Use it when:

  • You onboard a new tenant and set its tier limits.
  • You move a tenant to a different tier, or change the limits of a tier.
  • A cost incident showed that the gateway did not stop runaway usage.
  • PB-AGT-11-03 found one tenant driving a spike and you need a temporary lower limit for that tenant.

Do not use it when:

  • You need per-run, per-user or step limits inside the agent. Use PB-AGT-11-01.
  • You are diagnosing a cost spike. Use PB-AGT-11-03.
  • You still need to decide the ceiling values or what happens at a ceiling. Use PB-AGT-11-04 first.
  • An agent is looping right now. Contain it with the kill switch in PB-AGT-09-01. A gateway limit change is slower and affects every tenant in the tier.
  • You are tuning RAG retrieval cost. Use PB-RAG-12-01.

3. What you must know first

  • llm-token-limit returns 429 Too Many Requests when tokens-per-minute is exceeded, and 403 Forbidden when token-quota is exceeded. Agents back off on 429 and stop the run with a budget message on 403.
  • One counter exists per counter-key value, across every scope where the policy uses that key. Key on @(context.Subscription.Id) with one APIM subscription per tenant. A constant or a client IP puts tenants behind a shared gateway into one counter.
  • The policy tracks usage separately at each gateway, including regional and workspace gateways. It does not add up usage across the whole instance.
  • Concurrent requests can briefly exceed a limit, because the actual token count is known only after the backend responds. With stream: true, prompt and completion tokens are always estimated.
  • estimate-prompt-tokens="true" can stop requests before they reach the backend once the limit is already exceeded. It may reduce performance.
  • The remaining-quota header is an estimate. It can read higher than the true remaining quota. Do not build client logic on it reading exactly 0.
  • token-quota-period is a fixed window that starts at the UTC timestamp truncated to the period, for example the start of the UTC month.
  • On v2 tiers, rate limiting uses a token bucket. Keep tokens-per-minute identical wherever the same counter key is used at more than one scope.
  • llm-emit-token-metric runs in the inbound section only. It allows up to 5 custom dimensions. Each dimension tracks at most 100 unique values and each namespace at most 1,000 active time series. Data beyond that is silently discarded.
  • Custom metrics with dimensions is a preview feature of Azure Monitor. It must be turned on in Application Insights and on the APIM diagnostic before metrics appear.
  • Some OpenAI models return no token usage when streaming unless the request sets include_usage to true.

4. Prerequisites and access

  • APIM in front of the model endpoints, on a tier that supports llm-token-limit (Developer, Basic, Basic v2, Standard, Standard v2, Premium or Premium v2).
  • One APIM product per tenant tier, and one APIM subscription per tenant in its tier product.
  • Tier limits agreed in the budget policy (PB-AGT-11-04). Gateway limits sit slightly above the code limits in PB-AGT-11-01.
  • Policy and named values managed as code, with review, in the deployment pipeline.
  • Application Insights connected to APIM, logging enabled for the LLM APIs, and custom metrics with dimensions enabled (preview).
  • APIM gateway logs sent to Log Analytics in resource-specific mode, so they land in ApiManagementGatewayLogs. In Azure diagnostics mode they land in AzureDiagnostics instead, and the queries in this runbook do not work.
  • Roles: API Management Service Contributor on the APIM instance; Monitoring Reader to read Application Insights data and run log queries.
  • An approved change ticket naming the tenants, the new limits and the rollback owner.

Placeholders: ${SUBSCRIPTION_ID}, ${RESOURCE_GROUP}, ${APIM_NAME}, ${APIM_GATEWAY_URL}, ${APIM_MGMT_API_VERSION}, ${TIER_PRODUCT_ID}, ${TENANT_APIM_SUB_ID}, ${TENANT_SUB_KEY}, ${OTHER_TENANT_SUB_KEY}, ${API_PATH}, ${DEPLOYMENT_NAME}, ${OPENAI_API_VERSION}, ${TPM_LIMIT}, ${QUOTA_LIMIT}. Keep subscription keys in a secret store, not in shell history.

5. Steps

Outline of the steps. The author adds the final commands in T06; commands below are checked where marked.

Stage 1: Prepare

Step 1. Confirm the tenant's APIM subscription and product.

SVC="https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.ApiManagement/service/${APIM_NAME}"
az rest --method get \
  --url "${SVC}/subscriptions/${TENANT_APIM_SUB_ID}?api-version=${APIM_MGMT_API_VERSION}" \
  --query "{state:properties.state, scope:properties.scope}"

There is no az apim subscription command in Azure CLI, so this step calls the Subscription - Get REST operation. Set ${APIM_MGMT_API_VERSION} to 2024-05-01. The response does not include the subscription keys.

Verify: state is active and scope ends with /products/${TIER_PRODUCT_ID}. No other tenant shares this subscription.

Step 2. Export the current policy and named values, and commit them.

mkdir -p policies/before
az rest --method get \
  --url "${SVC}/products/${TIER_PRODUCT_ID}/policies/policy?api-version=${APIM_MGMT_API_VERSION}&format=rawxml" \
  > "policies/before/${TIER_PRODUCT_ID}.json"
for nv in tenant-tpm-standard tenant-quota-standard; do
  az apim nv show --resource-group "${RESOURCE_GROUP}" --service-name "${APIM_NAME}" \
    --named-value-id "${nv}" --query "{name:name, value:value}" > "policies/before/${nv}.json"
done
git add policies/before && git commit -m "Snapshot APIM policy and named values before token limit change"

format accepts xml or rawxml. The response is JSON, with the policy XML in properties.value.

Verify: the policy file contains the current policy XML, each named-value file contains its current value, and the commit is in the change ticket. This is the rollback point.

Step 3. Set the tier limits as named values.

Use az apim nv create for a new tier and az apim nv update for an existing one. Create both the rate and the quota values.

az apim nv update --resource-group "${RESOURCE_GROUP}" --service-name "${APIM_NAME}" \
  --named-value-id "tenant-tpm-standard" --value "${TPM_LIMIT}"
az apim nv update --resource-group "${RESOURCE_GROUP}" --service-name "${APIM_NAME}" \
  --named-value-id "tenant-quota-standard" --value "${QUOTA_LIMIT}"

Verify: az apim nv show returns the new values for both names. Both values are slightly above the matching code limits in PB-AGT-11-01.

Stage 2: Apply and test in staging

Step 4. Apply the tier policy to the staging tier product.

Use the policy in snippets/apim/policies/tenant-token-limit.md. Deploy it through the pipeline at product scope.

Verify: the deployment succeeds and the product policy shows llm-token-limit and llm-emit-token-metric, both in inbound.

Step 5. Drive traffic above the rate limit from one test subscription.

Use the load test snippets/ops/apim-limit-test.sh (to be added in T06). It sends requests from two subscriptions to:

curl -sS -o /dev/null -w "%{http_code}\n" -X POST \
  "${APIM_GATEWAY_URL}/${API_PATH}/deployments/${DEPLOYMENT_NAME}/chat/completions?api-version=${OPENAI_API_VERSION}" \
  -H "Ocp-Apim-Subscription-Key: ${TENANT_SUB_KEY}" -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write 200 words."}],"max_tokens":300}'

Verify: the test subscription gets 200, then 429 with a Retry-After header. The other subscription gets only 200.

Step 6. Exhaust the quota on a test subscription with a very low quota.

Verify: the test subscription gets 403. Record the exact 403 response body in the snippet notes, so the agent client can tell a quota 403 from other 403s.

Step 7. Check the agent handles both codes.

Verify: on 429 the agent backs off and retries after Retry-After. On 403 it stops the run and shows the budget message from PB-AGT-11-04. It does not crash.

Stage 3: Roll out to production

Throttles live tenants

A wrong counter key or a low limit throttles every tenant in the tier, or none. Production agents may fail mid-run. Confirm counter-key="@(context.Subscription.Id)" in the reviewed change before you deploy. To undo, redeploy the policy committed in Step 2 (Section 7).

Step 8. Deploy the policy and named values to production through the pipeline.

Verify: the pipeline run succeeds and the deployed product policy matches the reviewed commit.

Step 9. Send one canary request with a tenant subscription in the tier.

Verify: the response is 200 and includes the x-remaining-tokens and x-remaining-quota-tokens headers.

Stage 4: Watch

Step 10. Check token metrics per subscription.

customMetrics
| where timestamp > ago(1h)
| extend sub = tostring(customDimensions["Subscription ID"])
| summarize tokens = sum(value) by name, sub, bin(timestamp, 15m)
| order by timestamp desc

If the metric names cannot be confirmed, use the LLM gateway log instead. ApiManagementGatewayLlmLog records PromptTokens, CompletionTokens and TotalTokens, and joins to ApiManagementGatewayLogs on CorrelationId to get ApimSubscriptionId. LLM logging must be enabled for the API.

Verify: the query returns rows for the changed tenants and for at least one unchanged tenant.

Step 11. Tell tenant owners their limits.

Send each tenant owner the rate limit, the monthly quota, the reset time (UTC) and what 429 and 403 mean.

Verify: each owner has acknowledged the message in the change ticket.

Step 12. Watch for 24 hours.

Verify: no unexpected 429 or 403 alerts for tenants outside the change, and token metrics continue to arrive.

6. Monitoring and alerts

The thresholds below are team defaults, not platform recommendations. Tune them to each tenant's baseline.

Signal Source Alert when Action
429 responses per APIM subscription ApiManagementGatewayLogs (ResponseCode, ApimSubscriptionId) Above 5% of requests for 15 minutes Check that tenant's agents for loops (PB-AGT-09-01)
403 responses per APIM subscription ApiManagementGatewayLogs (ResponseCode, ApimSubscriptionId) Any quota 403 Notify the tenant owner and the account lead
Tokens per subscription per hour llm-emit-token-metric custom metric Above 3x that tenant's baseline Start PB-AGT-11-03
Token metrics stop arriving llm-emit-token-metric custom metric No data for 30 minutes while requests succeed Check the dimension and time-series limits and the diagnostic settings

7. Rollback

Each risky step has an undo:

Step Rollback
Step 3 (named values) Run az apim nv update with the values saved in Step 2.
Step 4 and Step 8 (policy) Redeploy the policy committed in Step 2 through the pipeline.
Tier move for a tenant Move the tenant's subscription back to its previous tier product.

Roll back when any of these happen:

  1. Tenants outside the change get 429 or 403 after the deployment.
  2. Production agents fail on 403 during normal hours.
  3. The canary in Step 9 fails.

Verify after rollback: the deployed policy matches the Step 2 snapshot, and the 429 and 403 rates return to baseline.

8. Known issues and fixes

Symptom Likely cause Fix
All tenants throttled together Counter key is constant, or the client IP of a shared gateway Key on @(context.Subscription.Id), one subscription per tenant
Every tenant gets the same limit Policy at API scope with one tier's named values Put the policy at product scope, one product per tier
Quota exceeded mid-month for a large tenant Quota sized from averages Raise the tier through change control, or add a daily quota warning
Agents crash on 403 Client treats 403 as a bug Map quota 403s to a budget message and stop the run
Users see raw 429s instead of the agent's budget message Gateway limit lower than the code limit Keep gateway limits slightly above the PB-AGT-11-01 code limits
Metrics missing in Application Insights Custom metrics with dimensions not enabled, or the policy not in inbound Enable custom metrics (preview) and keep llm-emit-token-metric in inbound
Some tenants missing from metrics More than 100 values for one dimension; data silently discarded Use gateway logs for per-tenant chargeback
No token counts on streamed calls Model returns no usage when streaming by default Set include_usage to true in the request
Unpredictable throttling on a v2 tier Different tokens-per-minute values for the same key at different scopes Use one value per key across all scopes
Spike on a subscription that belongs to no tenant Evaluation or test jobs use a production subscription Give evaluation jobs their own subscription

9. AWS and open-source equivalents

On AWS, API Gateway usage plans set request throttling and a request quota per API key. They count requests, not tokens. AWS states that usage plan throttling and quotas are best-effort, not hard limits, and should not be relied on to control costs. Enforce token ceilings in the agent code (PB-AGT-11-01) instead. Amazon Bedrock also has its own service quotas.

Open-source LLM gateways such as LiteLLM can limit and budget each virtual key. A LiteLLM rate limit returns 429 with retry-after and rate_limit_type headers. Exceeding a key's spend budget returns a budget_exceeded error.

10. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: in the sandbox APIM, the dry-runner puts test subscription A in a tier product with a low rate limit and a low monthly quota, and subscription B in a tier product with higher limits. The dry-runner drives traffic from both, then rolls back the policy and named values to the Step 2 snapshot.

Pass criteria:

  • Subscription A gets 429 with a Retry-After header when it exceeds the rate limit.
  • Subscription A gets 403 when it exceeds the quota, and the agent stops with the budget message.
  • Subscription B gets no 429 or 403 during the test.
  • Token metrics show both subscriptions.
  • After rollback, the deployed policy and named values match the Step 2 snapshot.

Record

Item Detail
Playbook PB-AGT-11-02 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

11. Sources