Skip to content
⚠️ Before You Start
Risk: Low
Duration: 2-3 days
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Low (read-only monitoring; the risk is acting on wrong numbers if prices are stale)

Set up the unit-economics dashboard and budget alerts

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-25). Author: Deeparani · dry-run: Gayathri Ramachandran · sign-off: Sivaraj. Work items: RAG-12-T13, RAG-12-T14, RAG-12-T15, RAG-12-T16.

1. Summary

Before a pilot goes to production, and for every live RAG workload that does not yet track cost per answer.

Risk: Low (read-only monitoring; the risk is acting on wrong numbers if prices are stale). Typical duration: 2-3 days.

2. When to use and when not to

Use it when: Before a pilot goes to production, and for every live RAG workload that does not yet track cost per answer.

Do not use it when: to be written by the author.

3. What you must know first

  • Token totals come from Azure Monitor platform metrics; cost per tenant needs app telemetry
  • Cached tokens are billed differently, so capture cached_tokens per request
  • Cost Management budgets alert on actual and forecast spend
  • Keep unit prices in one place so a price change is a one-line update

4. Prerequisites and access

  • Log Analytics workspace with diagnostic settings for Azure OpenAI and AI Search
  • Application Insights telemetry with tokens, prompt version, tenant and latency per answer
  • Cost Management access and current unit prices
  • Agreed targets for cost per answer and p95 latency

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Log prompt, cached and completion tokens, deployment, prompt version, tenant and stage timings for every answer.
  2. Store unit prices in one configuration table.
  3. Build a query for cost per answer by tenant and day.
  4. Build p50 and p95 latency by stage.
  5. Add platform metrics: tokens by deployment, search latency and throttling.
  6. Publish the views as an Azure Monitor workbook or dashboard.
  7. Create Cost Management budgets per workload.
  8. Create alerts on cost per answer and p95 latency.
  9. Review the dashboard weekly with the product owner.

Draft snippet: Daily cost per answer and p95 latency by tenant from Application Insights

// Set prices per million tokens from the current price list before use.
let input_per_m = 0.0;
let cached_per_m = 0.0;
let output_per_m = 0.0;
customEvents
| where timestamp > ago(30d) and name == "rag_answer"
| extend tenant = tostring(customDimensions["tenant"]),
         prompt = todouble(customMeasurements["prompt_tokens"]),
         cached = todouble(customMeasurements["cached_tokens"]),
         completion = todouble(customMeasurements["completion_tokens"]),
         total_ms = todouble(customMeasurements["total_ms"])
| extend cost = ((prompt - cached) * input_per_m + cached * cached_per_m + completion * output_per_m) / 1e6
| summarize answers = count(), cost_per_answer = avg(cost), p95_ms = percentile(total_ms, 95)
    by tenant, day = bin(timestamp, 1d)
| order by day desc, cost_per_answer desc

Snippets to add:

  • snippets/ops/queries/unit-economics.kql: Cost per answer and p95 latency by tenant (kql)
  • snippets/app/telemetry.py: Emit per-answer telemetry with token counts (python)
  • snippets/ops/workbooks/rag-unit-economics.json: Azure Monitor workbook definition (json)

6. Verify it works

A second engineer deploys the dashboard in the sandbox and drives 500 test answers across two tenants. Pass: cost per answer matches a manual calculation within 5%, and both alerts fire on a simulated breach.

7. Known issues and fixes

Symptom Likely cause Fix
Cost per answer does not match the bill Missing cached-token discount, embeddings or search costs Reconcile monthly with Cost Management and add the missing meters.
Some tenants show no cost Tenant id not logged on every request Make tenant id a required telemetry field and reject events without it.

8. Monitoring and alerts

Signal Alert when Action
Cost per answer (dashboard query) 25% above target for a day Start PB-RAG-12-01.
Cost Management budget actual spend 80% of monthly budget Notify the product owner and review levers (PB-RAG-12-03).

9. AWS and open-source equivalents

On AWS, combine CloudWatch metrics for Amazon Bedrock, application logs and AWS Budgets. Open-source stacks can use OpenTelemetry with Prometheus and Grafana. The cost-per-answer formula stays the same.

10. Validation record

Pending. Planned dry-run scenario: A second engineer deploys the dashboard in the sandbox and drives 500 test answers across two tenants. Pass: cost per answer matches a manual calculation within 5%, and both alerts fire on a simulated breach.

11. Sources