Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 days including a benchmark on a sample corpus
Last Validated: Not yet validated (draft)
Kind: Decision guide
Risk Note: A poor default degrades retrieval relevance across all queries and forces an expensive re-index later

Choose a chunking strategy by document type

Draft: not validated

Not yet dry-run. Do not use on a live system until the validation record is complete and signed off.

1. Summary

This decision guide establishes the strategy, window size, overlap, and metadata extraction model for chunking documents into search-ready passages. It classifies enterprise documents into four distinct archetypes:

  • Structured manuals: Technical documentation, SOPs, and policies structured with hierarchical headings.
  • Tables and forms: Financial filings, spec sheets, and forms dominated by tabular structures.
  • Long continuous prose: Narrative reports, research papers, and legal transcripts lacking distinct Markdown headings.
  • Short records: FAQs, customer support tickets, and catalog items under 1,000 characters.

The recommended default for enterprise corpora is structure-aware Markdown heading chunking with layout extraction (Azure Document Intelligence prebuilt-layout with LangChain MarkdownHeaderTextSplitter and a secondary recursive character splitter at 2,000 characters / ~500 tokens with 25% overlap).

Facts checked against Microsoft Learn and AWS documentation on 27 October 2026. The Document Layout skill for Azure AI Search is preview (REST API 2024-11-01-preview or 2024-11-30-preview ).

2. When to use and when not to

Use this guide when:

  • Starting any new RAG project or designing an ingestion pipeline.
  • Introducing a new document archetype or file format into an existing corpus.
  • Retrieval evaluation indicates answers are split across chunk boundaries, tables are severed, or search results contain high duplicate chunk density.
  • Benchmarking candidate chunk sizes and overlaps against retrieval metrics (recall@10, nDCG@10) and token budgets.

Do not use it when:

  • You are writing the layout-aware extraction and chunking code. Use PB-RAG-01-02.
  • You are re-chunking and re-embed a live production corpus without downtime. Use PB-RAG-01-03.
  • You are conducting a production change review or troubleshooting indexing failures. Use PB-RAG-01-04.
  • You are selecting the retrieval mode (keyword, vector, hybrid, semantic ranker). Use PB-RAG-08-01.

3. What you must know first

  1. The chunking trade-off. Small chunks (128–256 tokens) maximise vector specificity and pinpoint retrieval, but risk severing multi-step procedures or table rows. Large chunks (1,024–2,048 tokens) preserve contextual coherence and tabular integrity, but dilute semantic vector similarity and consume more of the LLM context window. Microsoft Learn recommends starting near 512 tokens (~2,000 characters) with ~25% overlap.
  2. Layout preservation before chunking. Plain text extraction strips Markdown headings, table boundaries (<table>, <tr>), and page numbers. A naive splitter cuts tables mid-row, resulting in nonsensical numerical answers. Always parse structured documents into Markdown via a layout engine (Document Intelligence prebuilt-layout) before chunking.
  3. Token size versus character count. Embedding models enforce strict token limits (for example, text-embedding-3-large supports up to 8,191 tokens; check current limits in the docs). Splitters measure characters by default (typically 1 token ≈ 4 characters in English). Pass a model tokenizer length function when strict token ceilings are required.
  4. Overlap rules and constraints. Overlap preserves semantic continuity across chunk borders. Overlap must always be strictly less than chunk size; in the Azure AI Search TextSplit skill, overlap must be less than half the maximum page length. Overlap above 30% (team default recommendation) inflates vector storage quota and creates duplicate search hits.
  5. Metadata lineage for citations and security. Chunks cannot exist in isolation. Every chunk must retain source_path, page, heading_path (breadcrumb), and a deterministic chunk_id. This metadata is mandatory for source citations (PB-RAG-06) and security trimming / ACL filters (PB-RAG-04).
  6. Re-chunking triggers a full re-index. Changing chunk size, overlap, or strategy changes all document chunk IDs and vector embeddings. It is not an in-place update; it requires re-indexing the corpus into a new index with an index alias switch (PB-RAG-01-03 and PB-RAG-09-01).

4. Options

Decision tree

Use this decision tree to identify the appropriate strategy for each document archetype:

graph TD
    Start["New document type or corpus"] --> Q1{"Is record standalone and < 1,000 chars?"}
    Q1 -- Yes --> OptD["Option D: One chunk per record<br/>(short_records)"]
    Q1 -- No --> Q2{"Does content contain multi-row tables or forms?"}
    Q2 -- Yes --> OptC["Option C: Table-protected layout chunking<br/>(tables_and_forms)"]
    Q2 -- No --> Q3{"Does content have hierarchical headings?"}
    Q3 -- Yes --> OptB["Option B: Markdown heading chunking<br/>(structured_manual - DEFAULT)"]
    Q3 -- No --> OptA["Option A: Recursive character chunking<br/>(long_prose)"]

The 9-stage evaluation process

Follow this workflow to select, benchmark, and document the chunking strategy:

  1. Classify the corpus. Group incoming documents into archetypes (structured_manual, tables_and_forms, long_prose, short_records). Verify: Every document in the intake batch is mapped to one of the four archetypes.
  2. Select candidate profiles. Load starting profiles from snippets/chunking-profiles.yaml. Verify: snippets/chunking-profiles.yaml loads and contains profile definitions for each archetype present in the corpus.
  3. Extract layout sample. Extract 10–50 representative sample files per archetype with prebuilt-layout into cached Markdown. Verify: The local cache contains .md files preserving # headings, <table> blocks, and <!-- PageBreak --> markers.
  4. Generate candidate chunks. Run candidate profiles against the sample Markdown using snippets/chunk_layout.py. Verify: Output chunks pass report(chunks, profile) with 0 over_max and 0 broken_tables.
  5. Populate scratch index. Index each candidate chunk set into a dedicated sandbox Azure AI Search index using the target embedding model. Verify: Scratch index document count equals the candidate chunk count and index status reports healthy.
  6. Execute golden query set. Run 30–100 representative queries with labelled relevance to evaluate recall@10, nDCG@10, and chunk boundary split rate. Verify: Evaluation run completes with computed retrieval accuracy metrics for each candidate profile.
  7. Evaluate capacity and cost. Measure total chunk count, vector index size, and embedding token consumption against service partition limits. Verify: Projected corpus vector size is within partition quota limits (check current limits in the docs).
  8. Record decision. Document the chosen profiles, parameters, and benchmark evidence in snippets/chunking-decision-record.md. Verify: The decision record has all candidate scores, final profile assignments, and operational rationale recorded.
  9. Obtain sign-off. Gain RAG Lead sign-off before implementing the profile in the production ingestion pipeline. Verify: The sign-off table in the decision record has dated approvals from the author and RAG Lead.

Destructive: modifying production chunking requires a full re-index

Never apply chunking parameter changes directly to a live production index. Re-chunking alters all document IDs and vectors, leaving orphaned chunks and degrading search results. Always test candidates in a scratch index and execute production changes using the zero-downtime index alias switch procedure in PB-RAG-01-03.

Option A: Recursive character chunking (long_prose)

Splits text hierarchically on paragraph breaks (\n\n), newlines (\n), sentence boundaries ('.' followed by a space), and spaces until chunks fall within character limits.

  • Best for: Unstructured continuous prose, academic research papers, narrative reports, legal briefs, and interview transcripts lacking distinct Markdown headings.
  • Cost and risk: Low compute cost; runs locally without layout extraction when source text is clean. Risk: will cut across logical sections if applied to structured manuals.
  • Carried out by: LangChain RecursiveCharacterTextSplitter or Azure AI Search TextSplit skill (splitMode: pages).

Option B: Structure-aware Markdown heading chunking (structured_manual)

A two-pass strategy: first splits content along Markdown headings (#, ##, ###) to respect author-defined topic boundaries, then applies a secondary recursive splitter to sections exceeding the target chunk size. Prepends the hierarchical heading breadcrumb to each chunk.

  • Best for: Technical documentation, standard operating procedures (SOPs), knowledge base articles, and corporate policy manuals.
  • Cost and risk: Requires layout analysis (prebuilt-layout) to generate Markdown. Drastically reduces chunk boundary cut rates and enriches chunks with structural metadata.
  • Carried out by: PB-RAG-01-02 using MarkdownHeaderTextSplitter and chunk_layout.py.

Option C: Table-protected layout chunking (tables_and_forms)

Extracts documents to layout Markdown with HTML tables (<table>...</table>). Masks tables prior to text splitting, keeping tables whole up to 4,000 characters. Tables exceeding chunk size are split along row boundaries (<tr>), repeating the table header row on each fragment.

  • Best for: Financial statements, balance sheets, product specification matrices, and forms dominated by tabular figures.
  • Cost and risk: Requires Document Intelligence prebuilt-layout. Eliminates numerical hallucination from severed table rows. Risk: wide or deeply nested tables require dedicated table processing (PB-RAG-10-01).
  • Carried out by: PB-RAG-01-02 with keep_tables_whole: true.

Option D: One chunk per record (short_records)

Ingests each record, FAQ pair, or database entry as an atomic chunk without internal splitting.

  • Best for: Support ticket solutions, FAQ question-answer pairs, CRM contact logs, and product catalog cards under 1,000 characters.
  • Cost and risk: Zero chunking overhead; zero risk of severed context. Risk: records exceeding the embedding model context limit will fail ingestion.
  • Carried out by: Direct ingestion script without text splitting. Option D does not use chunk_layout.py.

Option E: Semantic chunking by embedding distance

Breaks text into sentences, computes vector embeddings for each sentence, and places split points where the cosine distance between adjacent sentences exceeds a statistical percentile threshold (e.g. 95th percentile).

  • Best for: Multi-topic essays, meeting transcripts, and stream-of-consciousness documents where topic transitions occur without explicit headings.
  • Cost and risk: High compute cost and latency (requires embedding every sentence individually). Benchmark evidence rarely justifies the compute overhead on structured corporate documents.
  • Carried out by: LangChain SemanticChunker (experimental).

Comparison of options

Feature Option A (long_prose) Option B (structured_manual) Option C (tables_and_forms) Option D (short_records) Option E (Semantic)
Extractor required None or prebuilt-layout Document Intelligence prebuilt-layout Document Intelligence prebuilt-layout None (Direct read) None or Text extract
Split logic Paragraphs / sentences Markdown headings (#, ##, ###) Markdown headings + Table masking None (Atomic record) Sentence cosine distance
Table protection No (cuts mid-table) Partial (unless masked) Yes (Atomic or row-split) Not applicable No
Heading breadcrumb No Yes (prefix_heading: true) Yes No No
Pre-processing cost Negligible Low (layout model only) Low (layout model only) None High (N embeddings)
Recommended size 2,000 chars / 500 overlap 2,000 chars / 500 overlap 4,000 chars / 0 overlap Full record (< 1,000 chars) Variable by distance

Snippet 1: Chunking profiles by archetype

Load starting configurations from snippets/chunking-profiles.yaml. A short representative profile for the default structured_manual archetype:

# Representative profile from snippets/chunking-profiles.yaml
structured_manual:
  profile_id: structured_manual
  extractor: prebuilt-layout
  strategy: markdown_headers
  headers: ["#", "##", "###"]
  header_levels: 3
  chunk_size: 2000
  chunk_overlap: 500
  prefix_heading: true

The complete set of starting profiles for all four archetypes is defined in snippets/chunking-profiles.yaml.

Snippet 2: Decision record template

Record benchmark evaluation metrics and final profile choices in snippets/chunking-decision-record.md. An illustrative excerpt:

# Chunking Decision Record: ${CORPUS_NAME}
- **Project:** ${PROJECT_NAME} | **Author:** ${AUTHOR_NAME}
- **Embedding model:** ${EMBEDDING_MODEL} (default 3072 dims)
- **Target latency budget:** ${LATENCY_TARGET_P95}

## Benchmark Results on Golden Query Set
| Candidate Profile | Archetype Tested | Recall@10 | nDCG@10 | Chunk Cut Rate (%) |
|---|---|---|---|---|
| Candidate A (Fixed 2000/500) | structured_manual | ${CAND_A_R10} | ${CAND_A_NDCG} | ${CAND_A_CUT_PCT}% |
| Candidate B (Markdown H1-H3) | structured_manual | ${CAND_B_R10} | ${CAND_B_NDCG} | ${CAND_B_CUT_PCT}% |

The full template with inventory tables, rationale notes, and approval signatures is maintained in snippets/chunking-decision-record.md.

5. Decision table

Situation Choose Why
Standard operating procedure (SOP) or technical guide with H1–H3 headings Option B (structured_manual) Preserves section boundaries, prevents splitting numbered procedure steps, and prepends heading paths for retrieval grounding.
Financial reports, balance sheets, and spec sheets with multi-column tables Option C (tables_and_forms) Keeps <table> blocks whole up to 4,000 characters. Prevents broken rows which cause numerical hallucination.
Narrative whitepapers, legal depositions, and reports lacking standard headings Option A (long_prose) Recursive splitting falls back gracefully to paragraph and sentence boundaries without failing on missing headings.
FAQ articles, CRM interactions, or catalog cards under 1,000 characters Option D (short_records) Preserves the complete context in a single passage with zero boundary splitting or duplicate chunk overhead.
Mixed PDF corpus containing both narrative sections and multi-row tables Option C (tables_and_forms) with secondary split Protects tables atomically while recursively splitting narrative text under Markdown headings.
Dense technical text where queries target exact part numbers or codes Option B with 1,000 chars / 200 overlap Smaller chunks concentrate vector similarity around the target entity. Use hybrid search (PB-RAG-08-01) for exact lexical matches.
High-level research queries requiring synthesis across multiple paragraphs Option B with 4,000 chars / 400 overlap Larger chunk size provides broader contextual background for the LLM to synthesise themes without requiring high k retrieval.
Documents using decorative bold text instead of actual Markdown headings Option A (long_prose) with header_levels: 0 Markdown header splitters cannot detect bold text as headers, which can result in huge un-split chunks.
Benchmark reveals answers frequently severed between two consecutive steps Option B with 500 character overlap Increasing overlap from 200 to 500 characters preserves continuity across adjacent procedural steps.
Multi-topic meeting transcripts or essays with no heading structure Option E (Semantic chunking) Identifies natural semantic shift points where cosine similarity between consecutive sentences drops.

Recommended default:

Deploy Option B (structured_manual) with layout extraction:

  • Extractor: Azure Document Intelligence prebuilt-layout
  • Primary splitter: LangChain MarkdownHeaderTextSplitter (header_levels: 3)
  • Secondary splitter: RecursiveCharacterTextSplitter
  • Target chunk size: 2,000 characters (~500 tokens)
  • Chunk overlap: 500 characters (~25% overlap)
  • Metadata prefix: prefix_heading: true (attaches heading breadcrumb to text)

When to deviate:

  • Deviate to Option C (tables_and_forms) when more than 20% of the corpus consists of financial tables, specification sheets, or forms.
  • Deviate to Option A (long_prose) when documents contain flat prose without Markdown headings, or when heading analysis reveals decorative pseudo-headings.
  • Deviate to Option D (short_records) when ingested entities are individual discrete records under 1,000 characters.
  • Deviate chunk size downwards (256 tokens / 1,000 characters) when queries target discrete factual values, dates, or item codes.
  • Deviate chunk size upwards (1,024 tokens / 4,000 characters) when queries demand expansive narrative context or document summarisation.
  • Never deviate based on intuition. Require a benchmark on the golden query set demonstrating a measurable increase in recall@10 (> 3%, team default gate) before approving a non-default configuration.

Rule of thumb: Start with Markdown layout chunking at 512 tokens with 25% overlap; never deviate without benchmark evidence on a golden set showing higher recall@10.

7. Known issues and fixes

Symptom Likely cause Fix
Answers miss the second half of a numbered procedure Fixed-size character chunks severed steps across an arbitrary length boundary Switch to Option B (structured_manual) and increase overlap to 500 characters (~25%).
Tables retrieved as jumbled numbers disconnected from column headers Plain text extraction stripped HTML table markup, flattening rows into disjointed strings Extract via Document Intelligence prebuilt-layout and apply Option C (tables_and_forms) to keep tables inside <table> blocks.
Retrieval returns multiple near-duplicate chunks for a single query Chunk overlap set too high (> 35%), generating duplicate passage variants in the index Reduce chunk_overlap to 10–20% and deduplicate search results by parent document before passing context to the LLM.
Empty or microscopic chunks (under 50 characters) polluting the search index Splitter triggered on consecutive empty headings or decorative visual dividers Strip empty headings during layout cleanup and enforce a minimum chunk size filter (e.g. drop chunks under 100 characters).
Embedding ingestion fails with HTTP 400 (max_tokens_exceeded or context_length_exceeded) Splitter sized chunks by character count, but dense text (code, technical tokens) exceeded model token limits Pass a model-specific tokenizer length function (e.g. tiktoken for OpenAI models) to chunk() instead of raw character len.
LLM cannot cite source page numbers accurately in answers Splitter discarded Document Intelligence <!-- PageBreak --> markers during ingestion Parse page break comments during splitting and attach the exact page number to every chunk metadata (see PB-RAG-01-02).
Vector search fails to retrieve exact part numbers or acronyms Pure vector search over large chunks dilutes the semantic signal of short unique tokens Reduce chunk size to 1,000 characters and implement hybrid search with BM25 keyword matching (PB-RAG-08-01).
Old chunks still appear in search results after a chunking change Re-indexing was run in place without deleting orphaned chunks from the previous schema Rebuild the index into a new index container and cut over traffic using an index alias (PB-RAG-01-03 and PB-RAG-09-01).

8. Monitoring and alerts

The thresholds below are illustrative team defaults, not platform recommendations. Tune them against a measured baseline.

Signal Alert when (team defaults) Action
RecallAt10OnGoldenSet (CI evaluation pipeline) Drops by > 3% compared to baseline on evaluation run Block the pull request; inspect failure queries for chunk boundary splits or missed tables.
ChunkBoundarySplitRate (CI evaluation pipeline) Exceeds 5% of queries on golden set Increase chunk overlap or upgrade from Option A to Option B.
TableParseErrorCount (Ingestion pipeline logs) > 0 table parsing errors per 1,000 documents Check Document Intelligence layout output; verify table masking regex.
OversizedChunkDropCount (Ingestion pipeline logs) > 0 chunks exceed model token limit Verify secondary recursive splitter bounds and tokenizer measure function.
VectorQuotaUsage (Azure AI Search platform metric) Exceeds 80% of partition quota during indexing Check current limits in the docs. Scale partitions before proceeding with full re-index.
ThrottledSearchQueriesPercentage (Azure AI Search platform metric) Above 1% for 5 minutes during benchmark indexing Throttle indexing worker concurrency or schedule re-indexing off-peak.

9. Cost and capacity notes

  • Layout analysis cost: Document Intelligence prebuilt-layout is billed per 1,000 pages analysed. Check current prices before approval. Caching the extracted Markdown locally makes all subsequent chunking and benchmarking experiments free.
  • Embedding token volume: Smaller chunk sizes with generous overlap increase the total number of chunks and vectors. For example, moving from 1,000 characters with 10% overlap (net step 900 characters) to 500 characters with 25% overlap (net step 375 characters) yields 900 / 375 = 2.4x as many chunks and vectors for the same source text, directly increasing embedding API costs and index vector quota consumption.
  • Vector index quota: Vector index quota in Azure AI Search is a hard limit per partition. Check current service limits in the docs. A re-indexing migration requires holding two full indexes simultaneously during validation, requiring 2x vector and storage capacity.
  • Semantic chunking overhead: Option E requires embedding every individual sentence during ingestion before computing cosine distances. This increases embedding API consumption by an order of magnitude compared to heuristic Markdown splitting.

10. AWS and open-source equivalents

Platform Equivalent Key differences
Azure AI Search Document Layout skill (preview) and Text Split cognitive skill Document Layout skill integrates prebuilt-layout directly into Azure AI Search indexer skillsets (preview, REST API 2024-11-01-preview or 2024-11-30-preview). Text Split skill supports fixed-size page splitting.
Amazon Bedrock Knowledge Bases Managed chunking strategies Offers managed Fixed-size chunking (tokens and overlap percentage), Hierarchical chunking (parent-child), Semantic chunking, and No chunking. Amazon Textract provides layout analysis.
Open-source (LangChain) MarkdownHeaderTextSplitter + RecursiveCharacterTextSplitter Exact equivalent to Option B. Offers SemanticChunker for Option E. Runs on local compute with no cloud vendor lock-in.
Open-source (LlamaIndex) SentenceSplitter and HierarchicalNodeParser HierarchicalNodeParser produces parent-child chunk hierarchies where large parent chunks provide LLM context while small child chunks are indexed for vector search.
Open-source extraction unstructured or marker Open-source layout extraction packages that convert PDFs to Markdown without external API fees, but require dedicated GPU instances for high-throughput OCR.

11. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: A second engineer takes a representative sample of 50 documents covering two archetypes (35 structured technical manuals and 15 tabular financial reports).

They execute the 9-stage evaluation process in a sandbox Azure AI Search index using the candidate profiles in snippets/chunking-profiles.yaml and the golden evaluation query set. The engineer records benchmark metrics in snippets/chunking-decision-record.md.

Pass criteria:

  1. The decision table maps each document archetype to the candidate profile that achieves the highest recall@10 on the golden query set within the token budget.
  2. The selected profiles achieve Recall@10 >= 0.85 and ChunkBoundarySplitRate <= 5% on structured manuals and tabular documents.
  3. All candidate profiles parse successfully in the reference chunker without raising ConfigurationError or ChunkError.

12. Sources

Checked on 27 October 2026.

  • PB-RAG-01-02: Build layout-aware chunking with Document Intelligence and LangChain (carries out Options B and C).
  • PB-RAG-01-03: Re-chunk and re-embed a live corpus without downtime (zero-downtime production cutover).
  • PB-RAG-01-04: Chunking change review and troubleshooting checklist (pre-deployment audit).
  • PB-RAG-08-01: Choose the retrieval mode: keyword, vector, hybrid, semantic ranker or external re-ranker.