Skip to content

Chunking Change Review and Troubleshooting Checklist

Use this checklist when reviewing pull requests that modify document splitters, chunk size, overlap parameters, layout extraction settings, or chunk metadata. Also use it during operational triage when investigating user reports of truncated, cut-off, or out-of-context answers in production.

1. Pull Request Review Gate

  • PR Link / Change Ticket: ${PR_LINK_OR_TICKET}
  • Target Corpus Archetype: structured_manual | narrative_policy | tabular_financial | code_or_api
  • Engineer (Author): ${AUTHOR_NAME}
  • Reviewer (RAG Lead): ${REVIEWER_NAME}
  • Review Date: YYYY-MM-DD

Parameter and Profile Alignment

  • Decision record match: chunk_size and chunk_overlap match the agreed profile in PB-RAG-01-01. The pull request links to the approved decision record.
  • SplitSkill overlap constraint: For Azure AI Search Microsoft.Skills.Text.SplitSkill, overlap <= maximumPageLength / 2. Overlap above 50% is rejected with HTTP 400.
  • Embedding token ceiling: The largest chunk token count, tokenized with the exact target model tokenizer (e.g. tiktoken for OpenAI models), stays below 80% of the embedding model's maximum input token limit (e.g. under 6,500 tokens for an 8,191-token limit).
  • Preview feature declaration: If the Document Layout skill is used, the skillset explicitly declares API version 2024-11-01-preview and labels the feature as preview with an agreed fallback.

Metadata and Structural Preservation

  • Required citation metadata: Every output chunk dictionary retains source_path, page_number, chunk_id, and heading_path across 20 sampled extractions.
  • Natural sentence boundaries: Inspected 20 boundary chunks confirm splits occur strictly on natural separators (\n\n, \n, ". ", "? ", "! ") and never mid-word or mid-sentence.
  • Table structure preserved: Table chunks retain markdown table headers or enclosing tags; no single table row is orphaned without header context.
  • Parent context retention: Section titles and heading breadcrumbs are preserved in heading_path to prevent out-of-context retrieval.

Benchmark and Capacity Validation

  • Benchmark regression gate: Golden query evaluation shows Recall@10 and nDCG@10 remain within ±2% of the baseline scores recorded in PB-RAG-01-01.
  • Search partition capacity: Projected chunk count and vector index growth leave at least 25% storage and vector partition headroom on the search service tier.
  • Embedding cost approved: Total re-embedding token spend and run time are projected and approved by the RAG Lead before initiating full-corpus ingestion.
  • Zero-downtime runbook scheduled: Deployment scheduled using shadow index creation and atomic alias cut-over following PB-RAG-01-03.
  • Rollback script verified: Atomic alias swap script is staged and verified to point ${SEARCH_ALIAS} back to ${PREVIOUS_INDEX} if production regression occurs.

2. Production Incident and Troubleshooting Triage

Use this section during incident triage when users report incomplete, cut-off, or ungrounded answers.

  • Incident ID / Ticket: ${INCIDENT_ID}
  • Reported Symptom: Truncated answer | Out-of-context answer | Citation page mismatch | Indexer HTTP 400 | Duplicate search results
  • Investigator: ${INVESTIGATOR_NAME}
  • Investigation Date: YYYY-MM-DD

Triage Actions

  • Truncation diagnosis: Query the search index for the cited chunk_id. Check if raw chunk text ends mid-clause or mid-sentence. If split mid-sentence, switch to sentence-aware separators or increase overlap.
  • Out-of-context answer diagnosis: Retrieve all prompt context chunks for the failed query. Check if chunks lack parent headings or breadcrumbs in heading_path. Prepend section hierarchy to chunk text in the prompt template.
  • Citation mismatch diagnosis: Compare the chunk's page_number against the original source blob page. Verify extractor page tracking did not drift across form feeds or multi-column layouts.
  • Indexer HTTP 400 diagnosis: Inspect Azure Monitor indexer execution logs. Confirm chunk token length does not exceed embedding payload limits, and verify overlap <= maximumPageLength / 2.
  • Duplicate chunk diagnosis: Inspect search results for duplicate chunks across old and new document IDs. Verify ${SEARCH_ALIAS} points to a single clean index and that legacy chunks were purged.

3. Sign-off Record

Role Name Status Date (YYYY-MM-DD) Notes
Author (GenAI engineer) ${AUTHOR_NAME} Signed YYYY-MM-DD PR passed local evaluation
Reviewer (RAG Lead) ${REVIEWER_NAME} Approved YYYY-MM-DD Overlap and metadata verified
Deployment Owner ${DEPLOYMENT_OWNER} Scheduled YYYY-MM-DD Scheduled window via PB-RAG-01-03