Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 3-5 days for a first working pipeline
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Medium (poor extraction gives confident wrong answers on numbers)

Extract tables and figures into retrievable chunks

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-12-08). Author: Kranthi · dry-run: Ranjith · sign-off: Yogeshwaran. Work items: RAG-10-T05, RAG-10-T06, RAG-10-T07, RAG-10-T08.

1. Summary

When building or fixing an index where answers depend on tables, charts or diagrams.

Risk: Medium (poor extraction gives confident wrong answers on numbers). Typical duration: 3-5 days for a first working pipeline.

2. When to use and when not to

Use it when: When building or fixing an index where answers depend on tables, charts or diagrams.

Do not use it when: to be written by the author.

3. What you must know first

  • Content Understanding returns text_sections with Markdown and location metadata, and normalized_images when images are requested
  • Index projections write one search document per chunk
  • Image dimensions must be between 50 x 50 and 10,000 x 10,000 pixels
  • Password-protected PDFs must be unlocked before indexing
  • Keep a table with its caption and headers in one chunk where possible
  • Store page and location so citations point to the right page

4. Prerequisites and access

  • Extraction path chosen with PB-RAG-10-01
  • Foundry resource attached to the skillset for billing
  • Azure OpenAI deployments for figure descriptions and embeddings
  • Blob indexer with allowSkillsetToReadFileData set where the skill needs file_data
  • Index fields for chunk text, page or location, and image path

5. Steps

Outline of the steps; the author adds the commands and a check after each one.

  1. Create the blob data source and set allowSkillsetToReadFileData on the indexer where needed.
  2. Define the skillset with the Content Understanding skill, requesting images and location metadata.
  3. Add a GenAI Prompt skill to describe figures if the decision guide chose verbalisation.
  4. Add the embedding skill and index projections for chunks and figure descriptions.
  5. Create index fields for chunk, page, image path, figure description and vectors.
  6. Run on the 50-document sample and inspect the chunks that hold tables and figures.
  7. Ask the table and figure questions from the golden set and score the answers.
  8. Tune chunk size, table handling and the figure prompt, then repeat.
  9. Record the final configuration and scores in the pack.

Draft snippet: Extract tables from a PDF with the layout model and print them as Markdown

"""Extract tables from a PDF with the layout model and print them as Markdown."""
import os

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.identity import DefaultAzureCredential

client = DocumentIntelligenceClient(endpoint=os.environ["DOCINTEL_ENDPOINT"],
                                    credential=DefaultAzureCredential())

with open("samples/annual-report.pdf", "rb") as handle:
    poller = client.begin_analyze_document("prebuilt-layout", body=handle)
result = poller.result()

for number, table in enumerate(result.tables or [], start=1):
    grid = [["" for _ in range(table.column_count)] for _ in range(table.row_count)]
    for cell in table.cells:
        grid[cell.row_index][cell.column_index] = cell.content.replace("|", "/")
    page = table.bounding_regions[0].page_number if table.bounding_regions else "?"
    print(f"\nTable {number} (page {page})")
    print("| " + " | ".join(grid[0]) + " |")
    print("|" + "---|" * table.column_count)
    for row in grid[1:]:
        print("| " + " | ".join(row) + " |")

Snippets to add:

  • snippets/extract/tables_to_markdown.py: Extract tables with the Document Intelligence layout model (python)
  • snippets/skillsets/multimodal-skillset.json: Skillset with extraction, figure description, embeddings and index projections (json)
  • snippets/defs/index-multimodal.json: Index fields for chunks, pages and images (json)

6. Verify it works

A second engineer builds the pipeline in the sandbox from the pack and asks 20 table questions and 10 figure questions. Pass: the share of correct answers meets the target set for the corpus (for example 80%), and every answer cites the right page.

7. Known issues and fixes

Symptom Likely cause Fix
Merged table cells come out shifted Plain-text output loses table structure Use Markdown output so tables stay as HTML with rowspan and colspan.
Figure descriptions are generic The description prompt lacks domain context Pass the document title and section heading, and ask for values, labels and units.
Some images are skipped Images outside the size limits Check indexer warnings and pre-process images that are too small or too large.

8. Monitoring and alerts

Signal Alert when Action
Indexer warnings on the enrichment indexer Warnings on more than 5% of documents in a run Review warnings for skipped images and truncated text.

9. AWS and open-source equivalents

Amazon Textract returns tables as cell blocks that you rebuild into Markdown, and Amazon Bedrock Data Automation is the managed alternative for document insights; check which outputs it gives for your document types. The chunking and figure description steps are the same.

10. Validation record

Pending. Planned dry-run scenario: A second engineer builds the pipeline in the sandbox from the pack and asks 20 table questions and 10 figure questions. Pass: the share of correct answers meets the target set for the corpus (for example 80%), and every answer cites the right page.

11. Sources