RAG-02 · Retrieval evaluation and regression gates¶
Without a golden set and fixed metrics, teams argue about whether answers are worse and cannot tell retrieval faults from generation faults. Changes to prompts, models or indexes then ship without any regression check.
- Who reads it: GenAI engineers owning RAG quality and DevOps engineers running CI pipelines
- Outcome: Every RAG change gated by measured scores
- Pack owner: Prasanna (catalog owner role: RAG Lead)
Master catalog entry¶
- Catalog name: Retrieval Evaluation Kit (P1)
- Why it matters: Turns 'the answers feel wrong' into a measurable, fixable number
- What it removes: Diagnoses whether failures are retrieval or generation in minutes
- Stack: RAGAS, promptfoo, Python, Azure AI Evaluation SDK
- Maturity target: Level 3 — recall@k, MRR, nDCG, context precision, automated in CI
Playbooks¶
| ID | Story | Title | Kind | Complexity | Urgency | Status | Owner | Planned publish |
|---|---|---|---|---|---|---|---|---|
| PB-RAG-02-01 | RAG-02-S1 | Build a golden set and measure recall@k, MRR and nDCG | How-to | Medium | 4 | draft | Prasanna | 2026-11-18 |
| PB-RAG-02-02 | RAG-02-S2 | Run RAG quality evaluation with the Azure AI Evaluation SDK | How-to | Medium | 4 | draft | Devadharshini Raja | 2026-11-26 |
| PB-RAG-02-03 | RAG-02-S3 | Triage 'the answers feel wrong': retrieval or generation? | Runbook | Medium | 5 | draft | Prasanna | 2026-11-04 |
| PB-RAG-02-04 | RAG-02-S4 | Evaluation gate checklist for prompt, model and index changes | Checklist | Low | 4 | draft | Devadharshini Raja | 2026-11-17 |