Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 2-6 hours
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (hasty cuts can harm answer quality; slow action burns budget)

Investigate a RAG cost spike

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-16). Author: Gayathri Ramachandran · dry-run: Yasar · sign-off: Deeparani. Work items: RAG-12-T01, RAG-12-T02, RAG-12-T03, RAG-12-T04.

1. Summary

A budget alert fires, the daily bill jumps, or token usage rises without a matching rise in users.

Risk: Medium (hasty cuts can harm answer quality; slow action burns budget). Typical duration: 2-6 hours.

2. When to use and when not to

Use it when: A budget alert fires, the daily bill jumps, or token usage rises without a matching rise in users.

Do not use it when: to be written by the author.

3. What you must know first

  • Token metrics ProcessedPromptTokens, GeneratedTokens and TokenTransaction split by ModelDeploymentName
  • Cached prompt tokens appear as cached_tokens and are billed at a discount
  • A reindex re-bills extraction and embedding calls
  • Semantic ranker is billed per query with a non-empty search string
  • Do not use the legacy Ratelimit or Latency metrics for Azure OpenAI

4. Prerequisites and access

  • Cost Management Reader on the subscriptions
  • Monitoring Reader on Azure OpenAI, AI Search and Foundry resources
  • App telemetry with tokens, prompt version and tenant per request
  • List of recent releases, reindex jobs and resets

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Confirm the spike in Cost Management and find the resource and meter.
  2. For Azure OpenAI, chart prompt and completion tokens by deployment per hour.
  3. Compare tokens per request with the previous week to separate volume from size.
  4. Check releases, prompt versions and model changes at the start of the spike.
  5. Check for loops: retries, agent iterations or a client calling repeatedly.
  6. For search or extraction meters, check indexer runs, resets and skillset changes.
  7. Apply the quick fix: roll back the change, cap retries or disable a runaway schedule.
  8. Confirm hourly spend returns to baseline.
  9. Plan structural fixes with the levers in PB-RAG-12-03.
  10. Record cause, cost impact and prevention in the incident report.

Draft snippet: Hourly prompt and completion tokens per deployment for the last 48 hours

#!/usr/bin/env bash
# Hourly prompt and completion tokens per deployment for the last 48 hours.
set -euo pipefail
: "${AOAI_RESOURCE_ID:?}"
START="$(date -u -d '48 hours ago' +%Y-%m-%dT%H:%M:%SZ)"

az monitor metrics list \
  --resource "${AOAI_RESOURCE_ID}" \
  --metrics ProcessedPromptTokens GeneratedTokens \
  --aggregation Total \
  --interval PT1H \
  --start-time "${START}" \
  --filter "ModelDeploymentName eq '*'" \
  --output json \
  | jq -r '.value[] | .name.value as $metric | .timeseries[]
      | (.metadatavalues[0].value) as $deployment
      | .data[] | select(.total != null)
      | [$metric, $deployment, .timeStamp, .total] | @tsv'

Snippets to add:

  • snippets/ops/cost/tokens_by_deployment.sh: Hourly tokens per deployment from Azure Monitor (bash)
  • snippets/ops/queries/tokens-per-request.kql: Tokens per request by prompt version and tenant (kql)
  • snippets/ops/templates/cost-incident.md: Cost incident report template (markdown)

6. Monitoring and alerts

Signal Alert when Action
Cost Management budget per resource group Actual above 80% of monthly budget, or forecast above 100% Start this runbook.
ProcessedPromptTokens per hour per deployment Above 1.5 times the same hour last week Check releases and loops.
Requests per client (app telemetry) Above 3 times the normal rate for 15 minutes Throttle the client and investigate.

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Prompt tokens per request doubled Top-k or chunk size raised, or chat history not truncated Restore limits and cap context tokens.
Request count spike from one client Retry loop without back-off on 429 responses Add exponential back-off and a retry cap, and rate-limit the client.
Extraction meter spike An indexer reset or skillset edit reprocessed the corpus Use targeted reset docs or reset skills (preview) next time and require approval for resets.
Prompt cache hits dropped The prompt prefix changed so it no longer matches Keep static instructions first and identical; put variable content last.

9. AWS and open-source equivalents

On AWS, use Cost Explorer and the CloudWatch token metrics for Amazon Bedrock; the volume-versus-size split is the same. For self-hosted models the cost driver is GPU hours, so watch utilisation and queue depth.

10. Validation record

Pending. Planned dry-run scenario: A second engineer causes a sandbox spike (raises top-k and adds a retry loop) and follows the runbook from the budget alert. Pass: both causes are found within 2 hours and spend returns to baseline after the fix.

11. Sources