Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 days with measurements
Last Validated: Not yet validated (draft)
Kind: Decision guide
Risk Note: Medium (some levers trade answer quality for cost)

Pick cost and latency levers: caching, batching, model tiering, truncation and compression

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-17). Author: Gayathri Ramachandran · dry-run: Yasar · sign-off: Deeparani. Work items: RAG-12-T09, RAG-12-T10, RAG-12-T11, RAG-12-T12.

1. Summary

Before scaling a pilot, or when cost per answer or latency misses the target.

Risk: Medium (some levers trade answer quality for cost). Typical duration: 1-2 days with measurements.

2. When to use and when not to

Use it when: Before scaling a pilot, or when cost per answer or latency misses the target.

Do not use it when: to be written by the author.

3. What you must know first

  • Prompt caching is on by default for supported models and needs at least 1,024 identical leading prompt tokens
  • Semantic caching in API Management returns stored answers for similar prompts; a loose score threshold can return wrong answers
  • Global Batch is 50% cheaper than Global Standard with a 24-hour target, for offline work only
  • Model tiering sends simple questions to a smaller, cheaper model
  • Scalar quantisation to int8 makes the vector index about four times smaller; rescoring protects quality
  • Truncating context and output is the cheapest lever and the riskiest for quality

4. Options

Outline of the decision process; the author describes each option.

  1. Break down cost per answer: input tokens, output tokens, retrieval, re-ranking and hosting.
  2. List the levers that target the biggest line.
  3. Estimate savings and latency effect for each lever.
  4. Test each lever on its own against the golden set and record the quality change.
  5. Reject levers that breach the quality tolerance.
  6. Combine the accepted levers and test again.
  7. Record the decision with numbers and a rollback for each lever.

Draft snippet: Vector search section with int8 scalar quantisation and rescoring

{
  "vectorSearch": {
    "algorithms": [
      {
        "name": "hnsw-1",
        "kind": "hnsw",
        "hnswParameters": { "m": 4, "efConstruction": 400, "efSearch": 500, "metric": "cosine" }
      }
    ],
    "compressions": [
      {
        "name": "sq-int8",
        "kind": "scalarQuantization",
        "scalarQuantizationParameters": { "quantizedDataType": "int8" },
        "rescoringOptions": {
          "enableRescoring": true,
          "defaultOversampling": 4,
          "rescoreStorageMethod": "preserveOriginals"
        }
      }
    ],
    "profiles": [ { "name": "hnsw-sq", "algorithm": "hnsw-1", "compression": "sq-int8" } ]
  }
}

Snippets to add:

  • snippets/search/vector-compression.json: Vector compression with int8 scalar quantisation and rescoring (json)
  • snippets/apim/semantic-cache-policy.md: API Management semantic cache policy (llm-semantic-cache-lookup and store) (markdown)
  • snippets/eval/cost_levers.py: Cost per answer model for comparing levers (python)

5. Decision table

Situation Choose Why
to be written

To be written by the author from the outline above.

7. Known issues and fixes

Symptom Likely cause Fix
The semantic cache returns an answer to a different question Score threshold too loose, or cache shared across users Lower the threshold, partition with vary-by per user or tenant, and exclude sensitive flows.
Prompt cache hit rate near zero Variable content placed before static instructions Put static system text first and keep it identical between requests.
The small model fails on hard questions The router sends complex questions to the cheap tier Route by measured difficulty and fall back to the larger model on low confidence.

8. Monitoring and alerts

Signal Alert when Action
Share of prompt tokens served from cache (cached_tokens / prompt_tokens) Halves from baseline over a day Check the prompt prefix and cache configuration.
Quality score on sampled answers after a lever change Drop beyond the agreed tolerance Roll back that lever.

9. AWS and open-source equivalents

On AWS, Amazon Bedrock offers prompt caching and batch inference, and OpenSearch supports vector quantisation. Open-source stacks can add a semantic cache and quantised indexes; the decision process is the same.

10. Validation record

Pending. Planned dry-run scenario: A second engineer applies two levers in the sandbox: reordering the prompt so static text comes first, and int8 quantisation on a new vector field. Pass: cached-token share, cost per 1,000 answers and vector index size move as predicted, and the golden-set score stays within tolerance.

11. Sources