Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 30-60 minutes per review
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Missed checks cause retrieval quality regression or a costly full-corpus re-embed

Chunking change review and troubleshooting checklist

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off. Author: Kandiah Chockalingam · Planned dry-run: Ranjith · Planned sign-off: Yogeshwaran.

1. Summary

This checklist reviews changes to document chunking configurations, text splitters, and extractor settings before merging code to production. It prevents retrieval degradation, truncation errors, missing citation metadata, and costly full-corpus re-embeddings. Engineers and peer reviewers use it during pull request evaluation and when diagnosing out-of-context or cut-off answers in production.

Risk is Medium (missed checks lead to a costly full-corpus re-embed). Typical duration is 30-60 minutes per review. Layout-aware chunking with the Document Layout skill is preview with API version 2024-11-01-preview.

2. When to use and when not to

Use this checklist when:

  • Reviewing pull requests that change text splitters, chunk size, overlap parameters, or layout extraction settings.
  • Verifying chunk metadata fields (source_path, page_number, heading_path) before rolling out ingestion pipeline changes.
  • Preparing for a corpus re-index or re-chunking deployment to verify that candidate configurations satisfy embedding and storage limits.
  • Investigating user incident reports of incomplete, truncated, or ungrounded answers in production.
  • Diagnosing search relevance regressions or citation section errors following an ingestion pipeline update.

Do not use it when:

  • Choosing the initial chunking strategy and benchmark profile for a new corpus type. Use PB-RAG-01-01 instead.
  • Building a format-aware layout extraction pipeline with Document Intelligence. Follow PB-RAG-01-02.
  • Running the zero-downtime re-index and alias migration on a live production corpus. Follow PB-RAG-01-03.
  • Triaging whether answer degradation is caused by retrieval failure or LLM generator failure. Use PB-RAG-02-03 instead.

3. What you must know first

  • Text Split skill overlap limit. In Azure AI Search Microsoft.Skills.Text.SplitSkill, overlap cannot exceed half of maximumPageLength (overlap <= maximumPageLength / 2). Overlaps above 50% are rejected with an HTTP 400 error. Check Split skill cognitive search skill.
  • Chunk keys and citation invalidation. Modifying chunk boundaries changes generated chunk IDs and content hashes. Previous document-to-chunk citations, vector caches, and evaluation baselines become invalid.
  • Embedding model token ceiling. Embedding models enforce strict token input limits (check current limits in the official docs, such as 8,191 tokens for standard embedding models). Splitting by character count without tokenizer verification risks API payload rejection or silent truncation. Check Chunk large documents for RAG and vector search in Azure AI Search.
  • Metadata inheritance. Layout-aware extractors (such as the Document Layout skill, preview 2024-11-01-preview) must pass structural metadata (source_path, page_number, heading_path) into every child chunk so retrieval filters and citation renderers function. Check Chunk and vectorize with the Document Layout skill.
  • Storage expansion from overlap. Increasing overlap above 20% increases total chunk volume and vector index storage non-linearly, accelerating partition quota consumption.
  • Sentence and table integrity. Chunks must break on natural structural boundaries (paragraphs \n\n, line breaks \n, sentence endings ". ", "? ", "! ") and never split words or markdown table rows. Splitting inside markdown tables leaves table rows orphaned without headers, corrupting model comprehension.

4. Checklist

4.1 Prerequisites and configuration review

Checks completed before opening or approving a chunking pull request:

  • Profile alignment: GenAI engineer checks that chunk_size and chunk_overlap match the agreed decision record from PB-RAG-01-01. Verify: The pull request description links to the approved decision record and specifies the corpus archetype.
  • Overlap limit check: Reviewer verifies overlap <= maximumPageLength / 2 in the skillset JSON or splitter configuration. Verify: In skillset JSON, overlap is less than or equal to half of maximumPageLength; no configuration exceeds 50% overlap.
  • Preview feature check: If the Document Layout skill is used, engineer confirms API version 2024-11-01-preview is declared and documented as preview. Verify: The skillset JSON declares "@odata.type": "#Microsoft.Skills.Custom.DocumentLayoutSkill" or layout configuration with API version 2024-11-01-preview.

4.2 Pre-merge quality and validation gate

Checks that the engineer runs locally or in CI against candidate chunks:

  • Token ceiling margin: Engineer tokenizes the 50 largest sample chunks using the target tokenizer (e.g. tiktoken for OpenAI models). Verify: Every chunk token count stays below 80% of the embedding model context limit (e.g. under 6,500 tokens for an 8,191-token limit).
  • Metadata preservation: Engineer checks that every output chunk dictionary retains required citation keys (source_path, page_number, chunk_id, and heading_path) across 20 sample extractions. Verify: No sample chunk has null or empty metadata values for citation keys.
  • Sentence boundary integrity: Engineer inspects 20 boundary chunks to confirm splits occur on natural separators (\n\n, \n, ". ", "? ", "! "). Verify: No chunk starts or ends mid-word or mid-sentence.
  • Table structure retention: For structured documents, engineer verifies that table chunks retain markdown table headers or enclosing tags. Verify: No single table row exists in a chunk without column header context.
  • Benchmark regression gate: Engineer runs the retrieval evaluation suite against the golden query set. Verify: Recall@10 and nDCG@10 remain within ±2% of the baseline scores recorded in PB-RAG-01-01.
  • PR review checklist attached: Reviewer copies snippets/chunking-review-checklist.md into the pull request description and confirms all checks pass. Verify: The PR description contains the completed checklist with reviewer sign-off.
<!-- Review checklist snippet excerpt from snippets/chunking-review-checklist.md -->
## Chunking change review
- [ ] Profile matches the decision record (link)
- [ ] Overlap is less than half the maximum length
- [ ] Largest chunk is under the embedding model input limit
- [ ] Every chunk has source_path, page_number, heading_path
- [ ] Benchmark: recall@10 and nDCG@10 within tolerance of baseline
- [ ] Token cost per answer within budget
- [ ] Re-chunk runbook PB-RAG-01-03 scheduled for production
- [ ] Reviewer name and date recorded

4.3 Production release and rollback readiness

Checks before deploying chunking changes to production:

  • Re-chunk runbook scheduled: If updating a live index, engineer confirms runbook PB-RAG-01-03 has an approved change window, rollback owner, and alias switch plan. Verify: Change ticket specifies the shadow index name (${INDEX_NAME}-v2) and the target alias (${SEARCH_ALIAS}).
  • Quota and capacity verified: DevOps engineer reviews the search service tier quota and verifies at least 25% storage and vector partition headroom exists for projected chunk expansion. Verify: IndexStorageUsage and IndexVectorUsage remain below 75% of tier quotas.
  • Cost impact verified: RAG Lead confirms total corpus re-embedding token spend has been projected and approved before initiating a full ingestion run. Verify: Estimated token spend is recorded and approved in the change ticket.

Do not overwrite live indexes in place

Modifying chunking parameters in an active skillset or re-running an indexer directly against the live production index corrupts active vector data, causes mixed-chunk citation errors, and cannot be undone quickly. Always build a secondary shadow index (${INDEX_NAME}-v2) and repoint the index alias as described in PB-RAG-01-03.

  • Rollback command staged: Engineer verifies that the atomic rollback command is tested in the sandbox. If retrieval quality drops or latency spikes post-switch, execute the alias rollback immediately:
# Emergency alias rollback to previous index
curl -sS -X PUT "${SEARCH_ENDPOINT}/aliases('${SEARCH_ALIAS}')?api-version=2024-07-01" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${SEARCH_TOKEN}" \
  -d '{"name": "'"${SEARCH_ALIAS}"'", "indexes": ["'"${PREVIOUS_INDEX}"'"]}'

Verify: Calling curl -sS "${SEARCH_ENDPOINT}/aliases('${SEARCH_ALIAS}')?api-version=2024-07-01" returns "indexes": ["${PREVIOUS_INDEX}"].

4.4 Troubleshooting triage (Incident & quality degradation)

Checks when users report incomplete, cut-off, or ungrounded answers in production:

  • Truncation diagnosis: When answers cut off abruptly, engineer queries the chunk index by chunk_id and checks raw chunk text. Verify: If chunk ends mid-clause or mid-sentence, verify splitter separators; switch to RecursiveCharacterTextSplitter or increase overlap.
  • Out-of-context answer diagnosis: When answers lack essential document context or hallucinate, engineer inspects the retrieved chunks injected into the prompt. Verify: If chunks lack parent titles, verify that heading_path is populated and prepended to chunk text in the prompt template.
  • Citation mismatch diagnosis: When citations link to incorrect document sections, engineer checks the chunk page_number against the original source blob. Verify: Check if extractor page tracking drifted across form feeds or multi-column PDF layouts.
  • Embedding error diagnosis: When indexer fails with HTTP 400 or payload errors, engineer inspects indexer execution logs in Azure Monitor. Verify: Identify chunks exceeding the model token limit; decrease maxPageLength or configure token-based splitting.
  • Duplicate chunk diagnosis: When search returns near-identical results, engineer checks whether re-chunked documents were ingested without purging legacy chunks. Verify: Confirm ${SEARCH_ALIAS} points to a single clean index and old indexers are disabled.

5. Known issues and fixes

Symptom Likely cause Fix
Answers quote half a sentence or cut off mid-thought Chunk boundary splits mid-sentence Use sentence-aware separators (\n\n, \n, ". ", "? ", "! ") or increase overlap to 10-15%.
Citations point to the wrong page or missing section Page metadata lost during splitting Carry page numbers from the extractor through each split into chunk metadata (page_number, heading_path).
Index storage or vector partition grows sharply Overlap too high (>30%) or duplicate parent document indexing Reduce overlap to 10-15% and index only leaf chunks, projecting parents to blob storage only.
Embedding API returns 400 Bad Request Chunk exceeds model context token limit Tokenize with model-specific tokenizer (tiktoken) rather than counting characters; set maxPageLength lower.
Skillset returns HTTP 400: Overlap cannot exceed half of maxPageLength overlap configured > maximumPageLength / 2 in Text Split skill Adjust overlap <= maximumPageLength / 2 in skillset JSON.
Duplicate chunks returned in search queries Re-chunk written into active index without purge Build shadow index and execute atomic alias cut-over following PB-RAG-01-03.
Table cells returned without column headers Chunker splits table rows into separate chunks without headers Use layout-aware table chunker that prepends table headers to each row chunk.
Out-of-context answers from ambiguous paragraphs Chunks lack hierarchical document context Prepend heading_path or section breadcrumbs (Document > Section > Subsection) to each chunk.

6. Monitoring and alerts

The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Alert when Action
IndexStorageUsage (Platform metric) Growth >120% of planned projection after re-index Review chunk counts, verify overlap percentage, and purge duplicate indexers or temporary indices.
IndexVectorUsage (Platform metric) Total vector storage above 80% of tier partition quota Stop re-indexing. Apply vector compression (scalar/binary) or provision additional partitions.
SearchLatency (Platform metric) 95th percentile latency >1.5x baseline for 15 minutes Check retrieved chunk volume per query; adjust top parameter, reduce chunk count, or scale search replicas.
ThrottledSearchQueriesPercentage (Platform metric) Average above 1% for 5 minutes during or after re-index Scale search service replicas or throttle ingestion workers.
User feedback tagged 'incomplete answer' or 'truncated' (Application telemetry) Weekly count above team threshold (>3 incidents in 7 days) Run golden evaluation set and execute this checklist to verify chunk coherence.
FailedDocumentsCount (Indexer execution metric) Greater than 0 during scheduled ingestion run Check indexer execution log in Azure Monitor; verify chunk token sizes and skill overlap parameters.

7. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon Bedrock Knowledge Bases Fixed-size and hierarchical chunking strategies Overlap configured as percentage (up to 20%); max token limits enforced by Bedrock model family; hierarchical chunking generates parent-child relationships automatically.
AWS Kendra Built-in document splitting and hierarchical structure extraction Splits text automatically based on headings and tables; token limits managed by Kendra service; custom splitting requires pre-ingestion Lambda.
LangChain RecursiveCharacterTextSplitter and TokenTextSplitter Overlap configured in characters or tokens; token length must match embedding tokenizer (e.g. tiktoken); metadata must be preserved in Document(metadata={...}).
LlamaIndex SentenceSplitter and NodeParser Splits by sentences respecting window chunks; includes metadata extractors (TitleExtractor, QuestionsAnsweredExtractor) in node parser pipeline.
Unstructured.io chunk_by_title layout-aware chunking Groups elements under section headings; automatically preserves table HTML and hierarchical breadcrumbs in chunk metadata.

8. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A second engineer reviews two candidate pull requests for chunking changes: one with a seeded defect (overlap set to 60% of maximumPageLength and missing page_number metadata) and one compliant change. The dry-runner applies this checklist and tests triage steps against a test corpus with seeded truncation.

Pass criteria: The checklist successfully catches both planted defects (overlap limit error and missing metadata) in the PR gate, passes the clean PR, and accurately triages the truncated query.

Record

Item Detail
Playbook PB-RAG-01-04 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

9. Sources

Checked on 30 October 2026.