RAG-12 · RAG cost and latency tuning¶
Pilots ship without cost-per-answer or latency budgets, so teams learn about cost spikes and slow answers from the bill or from users. A rushed optimisation sprint before scale-up then risks breaking answer quality.
- Who reads it: GenAI engineers, RAG leads and DevOps or FinOps engineers responsible for production RAG cost and performance
- Outcome: Known cost per answer and p95 latency, with alerts
- Pack owner: Deeparani (catalog owner role: RAG Lead)
Master catalog entry¶
- Catalog name: RAG Cost & Latency Tuning Guide (P2)
- Why it matters: Unit economics decide whether a pilot is allowed to scale
- What it removes: Removes the panic optimisation sprint before go-live
- Stack: Azure Monitor, caching layer, embedding batch jobs
- Maturity target: Level 2 — caching, batching, model tiering and truncation levers
Playbooks¶
| ID | Story | Title | Kind | Complexity | Urgency | Status | Owner | Planned publish |
|---|---|---|---|---|---|---|---|---|
| PB-RAG-12-01 | RAG-12-S1 | Investigate a RAG cost spike | Runbook | Medium | 5 | draft | Gayathri Ramachandran | 2026-11-16 |
| PB-RAG-12-02 | RAG-12-S2 | Diagnose and fix a latency regression end to end | Runbook | High | 5 | draft | Deeparani | 2026-11-17 |
| PB-RAG-12-03 | RAG-12-S3 | Pick cost and latency levers: caching, batching, model tiering, truncation and compression | Decision guide | Medium | 3 | draft | Gayathri Ramachandran | 2026-12-17 |
| PB-RAG-12-04 | RAG-12-S4 | Set up the unit-economics dashboard and budget alerts | How-to | Medium | 4 | draft | Deeparani | 2026-11-25 |