Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-3 hours
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (slow diagnosis means continued overspend)

Investigate an agent cost spike

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-17). Author: Sandeep Velu · dry-run: Yasar · sign-off: Sivaraj. Work items: AGT-11-T09, AGT-11-T10, AGT-11-T11, AGT-11-T12.

1. Summary

When a budget alert fires, token metrics jump, or finance reports a cost increase on model deployments.

Risk: Medium (slow diagnosis means continued overspend). Typical duration: 1-3 hours.

2. When to use and when not to

Use it when: When a budget alert fires, token metrics jump, or finance reports a cost increase on model deployments.

Do not use it when: to be written by the author.

3. What you must know first

  • Azure budget alerts use cost data that is typically 8 to 24 hours old and are evaluated every 24 hours; use token telemetry for live diagnosis
  • Budgets only notify; they do not stop spend
  • OpenTelemetry GenAI conventions record token usage on spans (gen_ai.usage.input_tokens and gen_ai.usage.output_tokens)
  • Trace sampling undercounts totals; use APIM token metrics for totals and traces for causes
  • Common causes: loops, longer prompts after a change, a model swap, and one heavy tenant

4. Prerequisites and access

  • Log Analytics Reader on the Application Insights workspace
  • Token metrics from APIM and GenAI span attributes in traces
  • Recent deploy and model change history

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Confirm the spike in APIM token metrics by subscription and API.
  2. Find which agent, tenant and hour drove it with the trace query.
  3. Compare steps per run and tokens per step with the baseline for that agent.
  4. Check deploys, prompt changes and model version changes in the window.
  5. If a loop is the cause, start PB-AGT-09-01.
  6. If one tenant is the cause, contact the tenant owner and apply a temporary lower limit.
  7. If a change is the cause, roll it back or tighten the prompt or step budget.
  8. Verify token use per hour returns to baseline.
  9. Record cause, cost impact and fix in the cost incident log.

Draft snippet: Tokens by agent and hour from trace attributes

// Adjust attribute names to your instrumentation. Traces may be sampled.
dependencies
| where timestamp > ago(24h)
| extend agent = tostring(customDimensions["gen_ai.agent.name"]),
         tenant = tostring(customDimensions["app.tenant_id"]),
         inTok = toint(customDimensions["gen_ai.usage.input_tokens"]),
         outTok = toint(customDimensions["gen_ai.usage.output_tokens"])
| where isnotempty(agent)
| summarize input_tokens = sum(inTok), output_tokens = sum(outTok), calls = count()
    by agent, tenant, bin(timestamp, 1h)
| order by input_tokens desc
| take 50

Snippets to add:

  • snippets/ops/cost-spike.kql: Tokens by agent and hour from GenAI span attributes (kql)
  • snippets/docs/cost-incident.md: Cost incident log template (markdown)

6. Monitoring and alerts

Signal Alert when Action
Tokens per hour per agent Above 3x the 7-day baseline Start this runbook
Azure budget forecast alert Forecast above 100% of monthly budget Review the month-to-date breakdown and planned load
Average input tokens per step Up 50% after a release Review prompt and context changes in that release

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Trace totals far below the bill Sampling in the telemetry exporter Use APIM metrics for totals and traces only for breakdowns
Cost up but tokens flat Model or deployment type changed price Check deployment changes and price per token in the cost analysis
Spike with no matching tenant Evaluation or test jobs using production keys Give evaluation jobs their own APIM subscription
Input tokens per step keep growing in a run Conversation history passed in full on every step Trim or summarise history and cap context size

9. AWS and open-source equivalents

On AWS, use Bedrock invocation logging with CloudWatch metrics and AWS Budgets, which also work on delayed billing data. Open-source stacks use OpenTelemetry with Prometheus or Langfuse cost views.

10. Validation record

Pending. Planned dry-run scenario: In the sandbox, a second engineer replays a seeded loop for one tenant and a prompt change that doubles context. Pass: the engineer identifies both causes from telemetry within 1 hour and the fixes bring tokens per hour back to baseline.

11. Sources