Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 2-8 hours
Last Validated: Not yet validated (draft)
Kind: Runbook
Risk Note: Medium (wrong fixes add cost or reduce quality without fixing the slow stage)

Diagnose and fix a latency regression end to end

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-17). Author: Deeparani · dry-run: Sivaraj · sign-off: Prasanna. Work items: RAG-12-T05, RAG-12-T06, RAG-12-T07, RAG-12-T08.

1. Summary

p95 answer time breaches the target, or users report slow answers after a release or traffic growth.

Risk: Medium (wrong fixes add cost or reduce quality without fixing the slow stage). Typical duration: 2-8 hours.

2. When to use and when not to

Use it when: p95 answer time breaches the target, or users report slow answers after a release or traffic growth.

Do not use it when: to be written by the author.

3. What you must know first

  • Measure each stage; most regressions come from one stage
  • Azure OpenAI exposes Time to Response, Normalized Time to First Byte, Time Between Tokens and Time to Last Byte; some are not available for Standard deployments
  • Latency that rises with tokens is expected; latency that rises without more tokens points to load
  • AI Search throttling shows in ThrottledSearchQueriesPercentage, often while indexing runs
  • Semantic ranking adds processing time on the top 50 results
  • Streaming improves perceived latency, not total time

4. Prerequisites and access

  • Per-stage timings in app telemetry: retrieval, re-ranking, generation and time to first token
  • Monitoring Reader on Azure OpenAI and AI Search
  • Load-test script and a sandbox
  • Latency budget per stage

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Confirm the breach in app telemetry and find the slow stage.
  2. If generation is slow, chart time to response and tokens per request by deployment.
  3. If output tokens grew, cap maximum output tokens or tighten the prompt.
  4. If tokens are flat but time rose, check deployment load, 429 responses and region, and consider more capacity.
  5. If retrieval is slow, check SearchLatency and throttling, and whether indexing runs at the same time.
  6. Add replicas or move indexing off-peak if queries are throttled.
  7. If re-ranking is slow, reduce candidate count or skip re-ranking for simple queries.
  8. Turn on streaming in the user interface if it is not already used.
  9. Load-test the fix in the sandbox, then release it.
  10. Record per-stage timings before and after.

Draft snippet: Time retrieval, time to first token and total generation for one question

"""Time retrieval, generation and time to first token for one question."""
import os
import time

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from azure.search.documents import SearchClient
from openai import AzureOpenAI

credential = DefaultAzureCredential()
search = SearchClient(os.environ["SEARCH_ENDPOINT"], os.environ["INDEX_NAME"], credential)
llm = AzureOpenAI(azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
                  azure_ad_token_provider=get_bearer_token_provider(
                      credential, "https://cognitiveservices.azure.com/.default"),
                  api_version=os.environ["AZURE_OPENAI_API_VERSION"])
question = "What is the notice period for contractors?"

t0 = time.perf_counter()
chunks = [doc["chunk"] for doc in search.search(search_text=question, top=5, select=["chunk"])]
t1 = time.perf_counter()
stream = llm.chat.completions.create(
    model=os.environ["CHAT_DEPLOYMENT"], stream=True,
    messages=[{"role": "system", "content": "Answer only from the sources."},
              {"role": "user", "content": "\n\n".join(chunks) + "\n\nQuestion: " + question}])
first_token = None
for part in stream:
    if first_token is None and part.choices and part.choices[0].delta.content:
        first_token = time.perf_counter()
t2 = time.perf_counter()
print(f"retrieval {1000 * (t1 - t0):.0f} ms, first token {1000 * ((first_token or t2) - t1):.0f} ms, "
      f"generation {1000 * (t2 - t1):.0f} ms")

Snippets to add:

  • snippets/ops/latency/stage_timer.py: Per-stage timer including time to first token (python)
  • snippets/ops/queries/search-latency.kql: Search query duration percentiles by hour (kql)
  • snippets/ops/latency/aoai_latency.sh: Azure OpenAI latency metrics by deployment (bash)

6. Monitoring and alerts

Signal Alert when Action
App p95 end-to-end latency Above target for 15 minutes Start this runbook.
AzureOpenAITimeToResponse (average) per deployment Above 1.5 times baseline for 15 minutes Check tokens per request and load.
SearchLatency Above 1.5 times baseline for 10 minutes Check throttling and indexing activity.

7. Rollback

To be written by the author: how to undo each risky step.

8. Known issues and fixes

Symptom Likely cause Fix
Time to first token high only at peak hours Shared Standard capacity under load Consider provisioned throughput or a second deployment, and add back-off.
Search latency rises during the working day Indexer runs throttle queries Schedule indexing off-peak or add replicas.
The re-ranker doubles latency Too many candidates, or a cross-region call Re-rank fewer candidates in the same region.
Answers slower after a prompt change Longer outputs Set maximum output tokens and ask for concise answers.

9. AWS and open-source equivalents

On AWS, CloudWatch provides Amazon Bedrock invocation latency and token metrics, and OpenSearch Service exposes search latency; the stage-by-stage method is the same. Self-hosted models add GPU queue time to watch.

10. Validation record

Pending. Planned dry-run scenario: A second engineer adds query load to the sandbox search service while an indexer runs, and raises the maximum output tokens. Pass: the runbook finds both causes from metrics and telemetry, and p95 returns within target after the fixes.

11. Sources