Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 3-5 days for the first version
Last Validated: Not yet validated (draft)
Kind: How-to
Risk Note: Medium (pipeline faults lead to missing or duplicated content)

Build a format-aware ingestion pipeline with Document Intelligence and Azure Functions

Draft outline: facts checked on 2026-11-02, not validated

Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-30). Facts marked <!-- verify --> in the source of this page are not confirmed by any source. They are listed at the end. Items marked preview have no SLA. Author: Kranthi · dry-run: Yogeshwaran · sign-off: Prasanna. Work items: RAG-05-T01, RAG-05-T02, RAG-05-T03, RAG-05-T04.

1. Summary

In one minute

Clients send us messy files: PDFs, scans, Word files, spreadsheets, and some we cannot read. Before we can search them, each file must be read, checked and saved as clean text. This guide builds a small pipeline that does that automatically, and never loses a file.

  1. A file lands in a storage container.
  2. Azure sends an event, which starts an Azure Function.
  3. The function works out whether it has seen this exact content before.
  4. It reads the file: Document Intelligence for PDFs, scans and Office files, a simple reader for text.
  5. It saves clean Markdown and a short quality record.
  6. If anything goes wrong, it saves a dead-letter record that says why.
  7. Once a day, a report counts what happened.

The promise: every file ends in exactly one of three ways. It is processed, skipped as a duplicate, or dead-lettered with a reason. Nothing disappears.

Why we build it this way

  • One rule for every file. Each file gets a clear outcome, so a client question like "did you read my 30 files?" has an answer.
  • Pay once. Reading files with Document Intelligence is billed per page. A hash check stops us paying twice for the same content. Check the current price before approval.
  • Safe to run twice. Azure can deliver the same event more than once. The pipeline gives the same result either way.
  • Failures are visible. A dead-letter record names the stage and the reason, so a failed file is a to-do item, not a mystery.
  • Cheap to extend. A new file type is one line in a table and, if needed, one small reader.

Words you will see

Word Plain meaning
Ingestion Getting files from the client into a form we can search
Trigger Something that starts the function automatically, here a new file
Event Grid The Azure service that sends "a new file arrived" messages
Container A folder-like space in Azure Blob Storage
Hash A short fingerprint of a file's content. Same content, same hash.
Duplicate A second file with exactly the same content, even if the name differs
Idempotent Safe to run twice: the second run changes nothing
Dead letter A saved record of a file we could not process, with the reason
Quality record A small note about a processed file: pages, tables, characters, whether it looks empty
OCR Reading text from a picture of text, such as a scan

2. When to use and when not to

Use this guide when:

  • The built-in indexer and skills cannot handle the corpus, for example custom formats.
  • You need your own duplicate rules or your own failure records.
  • Files are pushed to you, not pulled from a source you can index.

Do not use it when:

  • The built-in indexer plus the Document Layout or Content Understanding skill fits. Use that first. It needs no code.
  • You only need to split text into chunks. Use PB-RAG-01-02.
  • An indexer or skillset is already failing. Use PB-RAG-05-02.
  • You are handling deletions or scheduled refresh. Use PB-RAG-05-03.
  • You want to check a new corpus before building. Use PB-RAG-05-04.

Which route should I choose?

Ask the questions in order. The first "yes" wins.

# Question Route
1 Do the files sit in Blob Storage, ADLS Gen2 or OneLake in formats the indexer and skills read? Built-in indexer and skill. No custom pipeline.
2 Do you need custom formats, your own duplicate rules, a push model or your own failure records? This pipeline.
3 Does the client already keep files in Microsoft Fabric? This pipeline with a Fabric front door, or the OneLake indexer (next section).
4 Is it a one-off bulk load of thousands of files? Batch analysis for the backlog, then this pipeline for new files.

Where Microsoft Fabric fits

The catalog lists Fabric in the stack. Fabric is not the default route. It is an option for clients who already keep files there. Facts from the Microsoft pages:

  • Data Factory in Fabric builds pipelines that can run on a schedule or on an event, and its copy activity moves data from many sources. The page says it connects to "more than 170 data sources".
  • OneLake is Fabric's data lake. It supports the Azure Data Lake Storage Gen2 and Blob Storage APIs, and shortcuts can point to ADLS Gen2, Blob Storage, Amazon S3 and Google Cloud Storage.
  • Azure AI Search has a OneLake files indexer. It needs a Fabric workspace and lakehouse, a search service on the Basic tier or higher in the same tenant, and does not support Parquet files. The page does not say whether it is generally available or preview.
Option How it works Use when
A. Fabric as a front door A Data Factory pipeline copies the client's files into our raw container. Everything after that is this guide. The client lands files in Fabric and you want one pipeline for all clients
B. OneLake indexer Azure AI Search reads the lakehouse directly. No Function. The formats are readable by the indexer and skills, and you do not need our duplicate or dead-letter rules
C. Read OneLake from the Function The Function reads OneLake through its Storage-compatible endpoints. You must keep files in OneLake and need custom handling

3. What you must know first

Concept What it means for you Source
A new file sends an event that starts the function. The Event Grid based blob trigger needs extension 5.x or later and a general-purpose v2 storage account. It has lower latency than polling. The Flex Consumption plan supports only the event-based trigger. Use source="EventGrid". Do not rely on the old polling trigger. Event Grid blob trigger page, Blob trigger page
Events can arrive twice and out of order. Event Grid delivers at least once, says duplicates are possible, and does not guarantee order. Retries run with growing delays for up to 24 hours. By default Event Grid tries up to 30 times and keeps an event for 1,440 minutes. Dead-lettering is off until you turn it on. The pipeline must be safe to run twice. Turn on Event Grid dead-lettering. Event Grid delivery and retry page
The docs do not describe retries for the Event Grid trigger. The polling trigger retries five times and then writes to a poison queue. The pages do not say what happens when your code fails under the Event Grid source. Never rely on the platform to retry. This guide keeps its own dead-letter record. Blob trigger page
The function loads the whole blob into memory in languages other than C#. Flex Consumption instances come in 512 MB, 2,048 MB and 4,096 MB sizes. The default function timeout is 30 minutes on Flex Consumption, with no fixed maximum. On the Consumption plan it is 5 minutes, with a 10-minute maximum. Check the file size before you read it. Choose the plan and size on purpose. Blob trigger page, Flex Consumption page, hosting options page
Document Intelligence has limits. The Standard tier allows 15 analyse requests per second by default, and a 429 means "slow down". A Standard file can be up to 500 MB and 2,000 pages. The free tier handles 4 MB and the first 2 pages. Check current limits in the docs. Retry on 429 with backoff and cap how many files run at once. Service limits page
Different readers handle different formats. Layout reads PDF, images, Word, PowerPoint, Excel and HTML. The Content Understanding skill also reads TXT, MD, RTF and EML. The blob indexer reads more again, including JSON, XML, ZIP and legacy Office files. Pick the reader per format in step 1. Layout page, Content Understanding page, blob indexer page
Confidence is a number from 0 to 1, but the docs give no threshold. They say to flag low-confidence results for human review. The team picks the "looks empty" threshold from a baseline. Confidence scores page

4. Prerequisites and access

Item Details Check
Storage account General-purpose v2, with a ${RAW_CONTAINER} for incoming files and an ${INGEST_CONTAINER} for results, hashes, logs and dead letters Both containers exist
Function app Python, on a plan that supports the Event Grid blob trigger. Flex Consumption is the plan in Microsoft's tutorial. The app starts and shows no errors
Document Intelligence resource A resource with a custom subdomain. Regional endpoints do not support Microsoft Entra sign-in. ${DOCINTEL_ENDPOINT} is a custom-subdomain address
Event subscription Created on the storage account for the "Blob Created" event, with dead-lettering on A test upload reaches the function
Format inventory A list of the client's file types with counts and sizes Every type is in the table in step 1
Sandbox All work happens in the sandbox. See docs/sandbox.md. No client data in the sandbox

Who needs which access

Who Needs Role
The function's managed identity Read new files in ${RAW_CONTAINER} A Storage Blob role.
The same identity Write results to ${INGEST_CONTAINER} A Storage Blob write role.
The same identity Call Document Intelligence Microsoft's page names Cognitive Services User for a service principal. It names no role for a managed identity.
Event Grid Write dead-lettered events to a container An RBAC role on the storage account if you use managed identity.

No keys, secrets or real names go in this guide. Use placeholders such as ${DOCINTEL_ENDPOINT}.

5. Steps

All reference code is Python. The small blocks below are copied from the files in snippets/. Longer code stays in snippets/, so this page stays short.

File What it holds
snippets/ingest_core.py The pipeline logic, with no Azure Functions dependency
snippets/function_app.py The Azure Functions entry point
snippets/test_ingest.py A dry run with no Azure resources, including the 30-file test

Stage 1. Prepare

Step 1. List the file types and pick a reader for each.

Plain words: decide, for every kind of file the client sends, who reads it. Unknown kinds are rejected on purpose, with a reason.

Format Reader Note
PDF, JPG, JPEG, PNG, BMP, TIF, TIFF, HEIF layout (Document Intelligence) OCR is built in, so scans work
DOCX, PPTX layout Each slide counts as one page
XLSX layout Table analysis is not supported for XLSX.
HTML layout
TXT, MD, CSV text Read as plain text. For rows and columns see PB-RAG-01-01.
RTF, EML, JSON, XML Not in the starter set Content Understanding or the blob indexer read them. Add a reader, or reject.
ZIP, GZ Not in the starter set Decide: unpack first, or reject.
DWG, EXE and anything else None Dead-letter with unsupported_format

The catalog target is 12+ format handlers. The starter code covers 15 file types with two readers. To add a type, add one line to HANDLERS. To add a new kind of reader, add one function to READERS.

HANDLERS = {   # file type -> who reads it. "layout" = Document Intelligence prebuilt-layout
    **{e: "layout" for e in (".pdf", ".jpg", ".jpeg", ".png", ".bmp", ".tif", ".tiff", ".heif",
                             ".docx", ".pptx", ".xlsx", ".html")},   # confirm .xls or .xlsx in the Layout docs
    **{e: "text" for e in (".txt", ".md", ".csv")},                  # read as plain text, no paid service
}
LOW_TEXT_CHARS_PER_PAGE = int(os.environ.get("LOW_TEXT_CHARS_PER_PAGE", "0"))  # TODO(author): set from the sandbox baseline

Verify: every file type in the inventory has a row. Types with no reader are listed as "reject".

Step 2. Create the storage, the function app and the event subscription.

Plain words: make the places files land and the wiring that wakes the function.

The event subscription endpoint cannot be changed

After you create an event subscription you cannot update its endpoint URL. If you recreate the function app, delete and recreate the subscription. Events that arrive while the subscription is missing are lost. Create it in the sandbox first, and plan a short quiet period for any live change.

  1. Create the storage account (general-purpose v2) with the two containers.
  2. Create the function app. Turn on its managed identity. Deploy the three files in snippets/.
  3. Set the app settings: RawStorage (identity-based connection to the storage account), INGEST_CONTAINER_URL, DOCINTEL_ENDPOINT, MAX_BYTES, CLAIM_STALE_SECONDS and LOW_TEXT_CHARS_PER_PAGE. Values are placeholders until the sandbox baseline sets them.
  4. Create the event subscription on the storage account. Event type: Blob Created. Endpoint: the function's blob extension webhook. Filter the subject to begin with /blobServices/default/containers/${RAW_CONTAINER}/blobs/.
  5. Turn on dead-lettering for the event subscription and point it at a container. It is off by default.
  6. In host.json, check the Storage extension bundle version, and set blob concurrency (maxDegreeOfParallelism) so the number of files running at once stays under the Document Intelligence rate limit.

Microsoft's step-by-step tutorial is the source for items 2 to 4: see the Event Grid blob trigger page in Sources. The page's own extensionBundle numbers disagree with each other.

Verify: upload one small .txt file to ${RAW_CONTAINER}. Within a minute the function log shows ingest <name> -> processed, and processed/ holds a .md and a .json file.

Stage 2. Build

Step 3. Add the trigger.

Plain words: the function wakes up for each new file, reads its name and size, and hands it to the pipeline.

CFG = {"max_bytes": int(os.environ["MAX_BYTES"]),               # TODO(author): choose from the instance memory size
       "stale_after": int(os.environ["CLAIM_STALE_SECONDS"])}   # TODO(author): choose from the function timeout
_deps = {}


def deps():
    if not _deps:                                              # build the clients once per instance, not once per file
        _deps["store"], _deps["client"] = make_clients()
    return _deps["store"], _deps["client"]

@app.blob_trigger(arg_name="blob", path="raw/{name}", connection="RawStorage", source="EventGrid")
def ingest(blob: func.InputStream):
    store, client = deps()
    name = blob.name.split("/", 1)[-1]
    data = blob.read() if blob.length <= CFG["max_bytes"] else b""   # never load an oversize file into memory
    outcome = run(name, data, blob.length, store, client, CFG, time.time())
    logging.info("ingest %s -> %s", name, outcome)

The path must match your container name.

Verify: python -m pytest -q snippets/test_ingest.py passes, including test_function_app_registers_one_blob_trigger.

Step 4. Check the file before reading it.

Plain words: reject files that can never work. This is cheap and saves money.

def ext_of(name):
    return os.path.splitext(name)[1].lower()


def sha256_of(data):
    return hashlib.sha256(data).hexdigest()

def intake_check(name, size, max_bytes):
    """Return a dead-letter reason, or None if the file may go on."""
    if size == 0:
        return "empty_file"
    if size > max_bytes:
        return "too_large"
    if ext_of(name) not in HANDLERS:
        return "unsupported_format"

Verify: an empty file, an oversize file and a .dwg file each give empty_file, too_large and unsupported_format, and none of them calls Document Intelligence.

Step 5. Claim the content, so a copy is not read twice.

Plain words: the first copy of any content "wins" by creating a small marker named after its hash. A second copy finds the marker and stops. Creating a blob only if it does not exist is a single, atomic action, so two copies arriving at the same moment cannot both win.

def create_blob(container, name, text):
    """Create a blob only if it does not exist yet. This is the 'first copy wins' lock."""
    try:
        container.upload_blob(name, text, overwrite=False)
    except ResourceExistsError as e:
        raise FileExistsError(name) from e

def claim_hash(store, digest, name, now, stale_after):
    """First copy wins. Returns new, retake, skipped_duplicate or redelivery."""
    key = f"hashes/{digest}.json"
    mine = json.dumps({"name": name, "state": "working", "time": now})
    try:
        store.create(key, mine)                        # fails if another copy already claimed it
        return "new"
    except FileExistsError:
        info = json.loads(store.read(key))
    if info["state"] == "working" and now - info["time"] > stale_after:
        store.write(key, mine)                         # the first worker died: take over
        return "retake"
    return "redelivery" if info["name"] == name else "skipped_duplicate"

def release_claim(store, digest):
    store.delete(f"hashes/{digest}.json")              # lets a later retry through
Result Meaning What happens
new First time we see this content Go on and read it
skipped_duplicate The same content under a different name Stop. Log it.
redelivery The same event again for the same name Stop. Do not log it as a new file.
retake A claim is "working" for longer than CLAIM_STALE_SECONDS The first worker died. Take over.

Verify: a copy of a processed file under a new name gives skipped_duplicate. The same event twice gives redelivery. Both leave the number of processed files unchanged.

Step 6. Read the file.

Plain words: send PDFs, scans and Office files to Document Intelligence. Read text files directly.

This step spends money

Document Intelligence bills by the page. Run it on the sample first, never on the whole corpus. Check the current price before approval. A file that fails after the claim is released can be read again on retry, so a bug that fails every file can bill every file again. Watch the pages analysed.

def analyse(client, data, waits=(2, 5, 13, 34)):
    for wait in waits:                                 # backoff pattern from the service-limits guidance
        try:
            poller = client.begin_analyze_document(
                "prebuilt-layout", body=io.BytesIO(data), output_content_format=DocumentContentFormat.MARKDOWN)
            return poller.result()
        except HttpResponseError as e:
            if e.status_code != 429:                   # only retry "too many requests"
                raise
            time.sleep(wait)
    raise RuntimeError("still throttled")

def read_layout(client, data):
    result = analyse(client, data)
    pages, chars = len(result.pages or []), len(result.content or "")
    quality = {"pages": pages, "tables": len(result.tables or []), "chars": chars,
               "low_text": chars < LOW_TEXT_CHARS_PER_PAGE * max(pages, 1)}   # scans often come back with little text
    return result.content or "", quality


def read_text(client, data):
    text = data.decode("utf-8")                        # a non-UTF-8 file raises here and is dead-lettered
    return text, {"pages": 1, "tables": 0, "chars": len(text), "low_text": False}


READERS = {"layout": read_layout, "text": read_text}

For one large backlog, the Document Intelligence batch analysis API can take up to 10,000 documents in one request from a blob container. Results stay for 24 hours, and per-file results only appear when the whole batch finishes. The page does not say whether it is generally available or preview.

Verify: a scanned image comes back with low_text: true and is still processed. A throttled file succeeds on retry. A file that fails with a non-429 error is not retried.

Step 7. Save the results.

Plain words: write the clean Markdown, a short quality record, and mark the claim "done".

def save_result(store, name, digest, text, quality, now):
    store.write(f"processed/{digest}.md", text)
    store.write(f"processed/{digest}.json", json.dumps({"source": name, "sha256": digest, **quality}))
    store.write(f"hashes/{digest}.json", json.dumps({"name": name, "state": "done", "time": now}))
    store.delete(f"deadletter/{digest}-{name}.json")   # clear an old failure after a good retry
Field in the quality record Meaning
source The original file name
sha256 The content fingerprint
pages, tables, chars What Document Intelligence found
low_text true when the file has little text for its page count. A review hint, not a failure.

Verify: each processed file has one .md and one .json under processed/, named by its hash.

Step 8. Dead-letter every failure, with a reason.

Plain words: when a file cannot be processed, save a note that says which file, which stage and why. Free the claim so a fixed file can be retried.

def dead_letter(store, name, digest, stage, reason, now):
    record = {"source": name, "sha256": digest, "stage": stage, "reason": reason, "time": now}
    store.write(f"deadletter/{digest}-{name}.json", json.dumps(record))
    return "dead_lettered"

def process(name, data, size, store, client, cfg, now):
    digest = sha256_of(data) if data else ""
    reason = intake_check(name, size, cfg["max_bytes"])
    if reason:
        return dead_letter(store, name, digest, "intake", reason, now)
    state = claim_hash(store, digest, name, now, cfg["stale_after"])
    if state in ("skipped_duplicate", "redelivery"):
        return state
    try:
        text, quality = READERS[HANDLERS[ext_of(name)]](client, data)
    except Exception as e:                             # keep the reason, never lose the file
        release_claim(store, digest)
        return dead_letter(store, name, digest, "read", f"read_failed:{type(e).__name__}", now)
    save_result(store, name, digest, text, quality, now)
    return "processed"
Reason Stage Typical fix
empty_file intake Ask the client to resend
too_large intake Split the file, or raise MAX_BYTES after checking memory
unsupported_format intake Add a reader, or tell the client
read_failed:<ErrorName> read Read the error name: a locked PDF, a bad encoding, or a service error

Verify: a corrupt file gives one dead-letter record that holds the reason, and the hash claim is gone, so a later retry is allowed.

Step 9. Log every outcome and build the daily report.

Plain words: write one line per file, then count the lines. The report is how you prove "every file ended one way".

def run(name, data, size, store, client, cfg, now):
    outcome = process(name, data, size, store, client, cfg, now)
    if outcome != "redelivery":                        # a redelivered event is not a new file
        record = {"source": name, "format": ext_of(name), "outcome": outcome}
        store.write(f"log/{sha256_of(data) if data else ''}-{name}.json", json.dumps(record))
    return outcome

def daily_report(store):
    log = [json.loads(t) for _, t in store.items("log/")]
    dead = [json.loads(t) for _, t in store.items("deadletter/")]
    quality = [json.loads(t) for n, t in store.items("processed/") if n.endswith(".json")]
    return {
        "files": len(log),
        "by_outcome": dict(Counter(r["outcome"] for r in log)),
        "by_format": dict(Counter(f'{r["format"] or "(none)"} {r["outcome"]}' for r in log)),
        "dead_letter_reasons": dict(Counter(d["reason"] for d in dead)),
        "low_text_files": sum(q["low_text"] for q in quality),
    }
Report field Meaning
files Files with a logged outcome
by_outcome Counts of processed, skipped_duplicate and dead_lettered
by_format The same counts for each file type
dead_letter_reasons How many dead letters have each reason
low_text_files Processed files flagged as having little text

Verify: files equals the number of files uploaded, and the three outcome counts add up to files.

Step 10. Connect storage and the Document Intelligence client.

The real storage wrapper, BlobStore, is in snippets/ingest_core.py. It has the same methods as the in-memory store the tests use, so the same code runs in both places.

def make_clients():
    cred = DefaultAzureCredential()                    # managed identity in Azure, your own sign-in locally
    container = ContainerClient.from_container_url(os.environ["INGEST_CONTAINER_URL"], credential=cred)
    return BlobStore(container), DocumentIntelligenceClient(os.environ["DOCINTEL_ENDPOINT"], cred)

Verify: in the sandbox, make_clients() returns without an error and a test upload reaches processed/.

Stage 3. Hand off and operate

Step 11. Hand the results to chunking and indexing.

Pick the processed Markdown up from processed/. For chunking see PB-RAG-01-02. To keep each user's permissions on the chunks, see PB-RAG-04-02. This guide stops at clean Markdown.

Verify: a processed file can be opened and its text is readable.

Step 12. Retry a dead-lettered file.

  1. Read the dead-letter record and fix the cause.
  2. Upload the file again, or copy it within the container so a new event fires.
  3. Check that the outcome is now processed, and that the dead-letter record is gone.

Verify: the report shows the file once, as processed.

6. Verify it works

The 30-file test

A second engineer drops 30 mixed files into a sandbox ${RAW_CONTAINER}. Each file must end in one of the three ways, and the report must match.

Files Count Expected outcome What it proves
Good PDF, Word, PowerPoint, Excel and HTML files 18 processed The main route works. One of them is throttled on its first try, so retry works.
Exact copies of two of those, under new names 2 skipped_duplicate Duplicates are caught
Scanned images 2 processed, with low_text: true OCR works and weak scans are flagged
Text, Markdown and CSV files 3 processed The no-cost reader works
DWG, EXE and ZIP files 3 dead_lettered, reason unsupported_format Unknown types are rejected, not lost
An empty file 1 dead_lettered, reason empty_file Empty input is caught
A corrupt PDF 1 dead_lettered, reason read_failed:<ErrorName> A read failure keeps its reason

Expected report: files 30, processed 23, skipped_duplicate 2, dead_lettered 5.

Run it with no Azure resources first: python -m pytest -q snippets/test_ingest.py. It runs this exact set of 30 files against stand-ins. It also checks the cases in the next section.

Then run it in the sandbox:

  1. Upload the 30 files to ${RAW_CONTAINER}.
  2. Wait until nothing new appears in log/ for the time you set in the baseline.
  3. Run the report. Compare each count with the table.
  4. Open each dead-letter record. Check the reason.
  5. List ${RAW_CONTAINER} and log/. Every uploaded file has exactly one log record.

Pass criteria: every file ends as processed, skipped as a duplicate, or dead-lettered with a reason, and the report matches the table. No two processed files hold the same content.

Cases the test also covers

Case Expected
The same event arrives twice redelivery. Nothing is counted twice.
Two copies arrive at the same moment One is processed, the other skipped_duplicate
A dead-lettered file is fixed and resent It becomes processed. The old dead-letter record is removed.
A file over MAX_BYTES too_large, and the file is never read into memory
A worker dies half way A fresh claim is left alone. A stale one is taken over.
Throttling (429) Retried with backoff. Other errors are not retried.

Cases to think about for your client

Case What to do
A file updated in place, with the same name and new content It has a new hash, so it is processed again as a new file. The old output stays. Replacement and deletion are in PB-RAG-05-03.
Near-duplicates, such as the same text saved by two programs Not caught: only exact content matches.
A password-locked PDF Document Intelligence needs the lock removed first. It dead-letters as read_failed. Ask the client for an unlocked copy.
A file whose extension lies, such as a PDF named .txt It fails to decode and dead-letters as read_failed.
A very long PDF The Standard tier takes up to 2,000 pages per file. Split longer files. Check current limits in the docs.
A file uploaded in several parts Event Grid raises "Blob Created" when the block blob is fully committed, so you do not see a half-uploaded file. For ADLS Gen2 uploads the event follows FlushWithClose.
Odd file names: spaces, accents, very long names Names are used in record keys. Test your client's real names in the sandbox.

7. Known issues and fixes

Symptom Likely cause Fix
The same document is indexed twice Duplicate event, or a copy under a new name The hash claim stops both. Check that step 5 runs before step 6.
A file is skipped as a duplicate but was never processed The first worker died after claiming The claim goes stale after CLAIM_STALE_SECONDS and the next copy takes over
429 from Document Intelligence at peak A burst of uploads passes the analyse rate limit The code retries with backoff. Also set blob concurrency in host.json. Ask for a higher limit by support ticket if it lasts.
The function times out on a big file Waiting for a long analyse job Check the plan's timeout. For big backlogs use batch analysis, or Durable Functions, which Flex Consumption supports with Azure Storage or the Durable Task Scheduler.
The function runs out of memory The blob is loaded into memory Set MAX_BYTES below what the instance can hold. Larger instance sizes exist.
A scan has almost no text Poor scan, or very small print The low_text flag shows it. Turn on the ocrHighResolution add-on for small print (extra cost: check the current price). Ask for a better scan.
A dead-lettered file never gets fixed Nobody reads the dead letters Alert on new dead letters (section 8) and assign an owner
A fixed file is skipped as a duplicate The claim was not released Step 8 releases it on failure. Check that your own error handling does not catch the error first.
Events never reach the function The subject filter is wrong, or the subscription was deleted Check the filter. Event Grid dead-letters undelivered events only if you turned it on.
The subscription broke after a redeploy The endpoint URL cannot be updated Delete and recreate the subscription
403 from Document Intelligence The function's identity has no role, or the endpoint is regional Use the custom-subdomain endpoint and give the identity a role.
The bill is higher than expected The same files were analysed again Check that claims are kept for successful files. Watch pages analysed. Never re-run on the whole corpus.

8. Monitoring and alerts

Thresholds are team defaults, not platform facts. Tune them against your sandbox baseline.

Signal Source Alert when Action
New dead-letter records deadletter/ in ${INGEST_CONTAINER} Above the agreed daily level. Review by reason. Fix the cause. Retry (step 12).
dead_lettered share of files Daily report Above the agreed share. Check the file types in by_format
low_text_files Daily report Above the agreed level. Ask the client for better scans
skipped_duplicate share Daily report A sudden rise The client may be resending a whole batch
Errors in FunctionAppLogs Azure Monitor Logs Any error that is not a dead letter Read the log. Check the identity and the settings.
OnDemandFunctionExecutionCount (Flex Consumption) Platform metric Far above the number of uploads Events are being repeated or looping. Check the subscription.
Events in the Event Grid dead-letter container Storage Any An event never reached the function. Check the endpoint and the filter.
Event Grid delivery failures and dead-lettered counts Platform metrics. Any rise Check the endpoint
Pages analysed by Document Intelligence Azure Monitor. Above the number of files uploaded Stop and check for repeated analysis

9. AWS and open-source equivalents

Platform Equivalent Key differences
Amazon S3, AWS Lambda, Amazon Textract An S3 event notification invokes a Lambda function. Textract reads the file. S3 event notifications are "designed to be delivered at least once", so keep the hash claim. Lambda's default timeout is 3 seconds and the maximum is 900 seconds (15 minutes). Textract handles multipage documents with asynchronous Start... and Get... operations, and multipage input must be PDF or TIFF. Never write the output back to the bucket that triggers the function: it can loop.
Amazon SQS A dead-letter queue with maxReceiveCount A message moves to the dead-letter queue after it is received that many times without being deleted. The dead-letter queue must be in the same account and region.
Open source Docling or Unstructured can replace the layout model. Any queue and any object store can replace the events and containers. The hash claim, the dead-letter record and the daily report stay the same
Microsoft Fabric Data Factory pipelines and OneLake See "Where Microsoft Fabric fits" in section 2

10. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: the dry-runner builds the pipeline in the sandbox from this guide only, then runs the 30-file test in section 6. They plant these faults: a duplicate under a new name, a corrupt file, an unsupported type, an empty file, a throttled file, the same event sent twice, and a worker stopped half way.

Pass criteria: every planted fault is caught and ends in the expected outcome, the report matches the table, and no file is lost.

11. Sources

Checked on 2 November 2026.

Open points for the pack owner (remove after sign-off)

  1. Fabric: keep options A to C as documented options (this draft), or record Fabric as out of scope?
  2. Hosting plan: Flex Consumption (the plan in Microsoft's tutorial), or another?
  3. Duplicates: exact content only (this draft), or near-duplicates too?
  4. Where the records live: blobs in the ingest container (this draft), or a table or database?
  5. Scope: does this pipeline stop at clean Markdown (this draft), or also chunk and embed?
  6. Archives: unpack ZIP and GZ files, or reject them?
  7. Verify before publishing: the Event Grid trigger's retry behaviour, the extension bundle version, roles for the managed identity, metric names for Event Grid and Document Intelligence, the OneLake indexer status, the batch analysis status, and Docling and Unstructured format support.