RAG-05 · Document ingestion: build, refresh and recover¶
Client corpora arrive with scans, odd formats, duplicates and deletions that the first pipeline never handled. When an indexer or skillset fails at night, teams do not know whether to reset, rerun or rebuild, and deleted files stay searchable.
- Who reads it: GenAI engineers building ingestion pipelines and DevOps engineers operating indexers and functions
- Outcome: Fresh, complete index with failures recovered same day
- Pack owner: Yogeshwaran (catalog owner role: RAG Lead)
Master catalog entry¶
- Catalog name: Document Ingestion Framework (P1)
- Why it matters: Client corpora are always messier than the statement of work assumed
- What it removes: Parsing, OCR, table extraction, dedup and refresh handled generically
- Stack: Document Intelligence, Fabric, Python, Azure Functions
- Maturity target: Level 3 — 12+ format handlers with quality reporting
Playbooks¶
| ID | Story | Title | Kind | Complexity | Urgency | Status | Owner | Planned publish |
|---|---|---|---|---|---|---|---|---|
| PB-RAG-05-01 | RAG-05-S1 | Build a format-aware ingestion pipeline with Document Intelligence and Azure Functions | How-to | High | 3 | draft | Kranthi | 2026-11-30 |
| PB-RAG-05-02 | RAG-05-S2 | Recover from indexer and skillset failures and throttling | Runbook | Medium | 5 | draft | Kranthi | 2026-11-04 |
| PB-RAG-05-03 | RAG-05-S3 | Handle source deletions, duplicates and scheduled refresh | Runbook | Medium | 4 | draft | Ranjith | 2026-11-05 |
| PB-RAG-05-04 | RAG-05-S4 | Corpus intake and ingestion quality checklist | Checklist | Low | 3 | draft | Kandiah Chockalingam | 2026-11-17 |