Skip to content
⚠️ Before You Start
Risk: Low
Duration: 2-4 hours per batch
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Low (catches problems early, before indexing cost)

Corpus intake and ingestion quality checklist

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Outline checked against the sources on 3 November 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-17). Author: Kandiah Chockalingam · dry-run: Ranjith · sign-off: Yogeshwaran. Work items: RAG-05-T13, RAG-05-T14, RAG-05-T15, RAG-05-T16.

How to read this page

New to this? Read section 1 and the "Think of it as" box, then go to section 4 and work from part A down. Each item says who does it and how to check it. Section 3 explains the words and the costs. You can read it later.

1. Summary

A client has sent us a pile of files. Before we pay to read and index them, we check the pile: can we read every file, is anything missing or there twice, do we know who may see what, and what will it cost? This checklist is the goods-in check.

Run it once for every new corpus or large batch, before indexing. Run part G again after the first ingestion run. The GenAI engineer runs it with the business analyst. The client agrees the fixes.

Think of it as the goods-in desk at a warehouse. You count the boxes, reject the damaged ones, spot the ones that arrived twice, work out the shipping bill, and sign the delivery note. Only then do the boxes go on the shelves.

Risk: Low (catches problems early, before indexing cost). Typical duration: 2-4 hours per batch.

How it is organised

Part Question it answers Done by
A. Agree the rules Are we allowed to use these files, and what do we test against? Business analyst
B. Count and sort What exactly did we receive? Is it safe to open? GenAI engineer
C. Can we read it? Which files will fail, and why? GenAI engineer
D. Is the content right? Duplicates, versions, scans, sensitive content Business analyst, GenAI engineer
E. Metadata and permissions Do we know who may see what, and can we filter? Business analyst
F. Report and decide Go or hold, what it costs, what the client fixes RAG Lead
G. After the first run Did ingestion keep its promise, and did the bill match? GenAI engineer

The flags the script gives a file

Flag Plain meaning Usual action
too_large, too_many_pages Over the size or page limit Split or exclude
password_protected A PDF that needs a password Client sends an unlocked copy
password_protected_or_old_format An old Office file, or one with a password Open one file to tell which
corrupt Cannot be opened, or is not the type its name says Client re-exports
empty_file Zero bytes Client re-sends
unsupported_format, archive_unpack_first We cannot read it as it is Convert, unpack or exclude
image_size_out_of_range A picture that is too small or too big Re-scan or resize
junk_file A temporary or system file Remove from the list
duplicate_of:<file> Exactly the same content as another file Skip. Keep the first
name_clash_with:<file> The same name if you ignore case Rename

2. When to use and when not to

Use this checklist when:

  • A client hands over a new corpus or a large batch, before it is indexed.
  • A client sends a replacement or a big top-up batch (run parts B to F again on the new files only).
  • Ingestion of an earlier batch gave many dead letters and you want to know why.
  • You must give the project lead a cost estimate before approval.

Do not use it when:

  • You need to build the pipeline that reads the files. Use PB-RAG-05-01 instead.
  • An indexer or skillset is failing on a live index. Use PB-RAG-05-02 instead.
  • You handle deleted or changed source files after go-live. Use PB-RAG-05-03 instead.
  • You need to build the permission filters. Use PB-RAG-04-02 instead. This checklist only checks that permission data arrived.

3. What you must know first

  • Size and page limits depend on the tier. Document Intelligence Standard (S0) allows 500 MB and 2,000 pages for each document. The free tier (F0) allows 4 MB and only the first two pages are analysed, so a long document passes with most of it missing. Check current limits in the docs.
  • Azure AI Search has its own file limit. An indexer limits the source file size and the characters it extracts, and both depend on the search tier. Oversized blobs are errors unless you choose to index their metadata only. Check current limits in the docs.
  • The Layout skill reads a fixed list of formats. PDF, JPEG, JPG, PNG, BMP, TIFF, DOCX, XLSX, PPTX and HTML. Images must be between 50 x 50 and 10,000 x 10,000 pixels. Password-locked PDFs must be unlocked first. A document that needs more than five minutes in the layout model times out, and the charge still applies.
  • The blob indexer reads a different, longer list. It includes DOC, XLS, PPT, MSG, EML and ZIP. By default it stops at a blob with an unsupported content type, so decide the path before you test formats.
  • Exact duplicates are found by hash. Near duplicates are not. The same content under two names gives the same SHA-256. A second version of a document does not.
  • A scan can hide inside a text PDF. A scanned page that someone already ran through OCR has a text layer, so the script counts it as text. If that old OCR was poor, the text is poor. Item D3 covers it.
  • Pages are what you pay for. Document Intelligence counts each PDF or TIFF page, each Excel sheet and each PowerPoint slide as one page. Word and HTML files count one page for each block of 3,000 characters.

Where the money goes

Never guess a price. The playbook gives the sums. You get the price from the official pricing page on the day, and you check the current price before approval.

Cost What drives it How to estimate it Where to get the price
Reading files with Document Intelligence Billable pages The script's Billable pages line, then pages ÷ 1,000 × ${PRICE_PER_1000_PAGES} Azure Document Intelligence pricing. Use the price of the model you run
Reading the same file twice Every re-run after a fix is billed again Fix flagged files before the first run. Duplicates are skipped by hash in PB-RAG-05-01 The same page
Files that time out A file over five minutes still bills Split very large files at intake (items B5 and C1) The same page
The free tier Not billed, but it reads only two pages Do not use F0 for a real batch The same page
The search service Tier, storage and vector size of the index Estimate chunks from pages, then check the index size against the tier limits Azure AI Search pricing
Embeddings The text of every chunk Estimate tokens from pages, then use the price of your embedding model Check the current price of your model
More than 20 documents for each indexer each day A billable Foundry resource must be attached to the skillset Plan for it when you use the Layout skill The Document Layout skill page
AWS Textract Pages The same page count Amazon Textract pricing
People Hours of checking and client replies The duration at the top of this page is a team estimate. Your project lead

The script prints the page sums for you. Give it your price and it prints the cost sums too (see the build steps at the end of section 4).

Plain-words glossary

Word Plain meaning
Batch One delivery of files from the client, with one ${BATCH_ID}
Flag A problem the inventory script found in one file
SHA-256 A fingerprint of a file's content. Same content, same fingerprint
Image-only page A page that is a picture of text, so it has no text layer and needs OCR
Billable page A page the service will charge for, counted by the rules above
Manifest A client's list of files with their metadata (owner, groups, dates)
Dead letter A file the pipeline could not process, kept with its reason (see PB-RAG-05-01)

4. Checklist

Tick each item as you go. Every item says who does it and how to check it. Build the script once, using the last part of this section, or copy it from the folder snippets/intake/.

Before you start

You need Python 3.10 or later, the script from snippets/intake/inventory.py, a read-only copy of the batch in ${INTAKE_FOLDER}, and the report template snippets/intake/corpus-intake-report.md.

pip install "pypdf[crypto]" pillow
python snippets/intake/inventory.py ${INTAKE_FOLDER} --max-mb ${MAX_MB} --max-pages ${MAX_PAGES} --manifest ${MANIFEST_CSV} --require owner,acl_groups --price-per-1000 ${PRICE_PER_1000_PAGES}

Verify: the script prints Files:, By format:, Billable pages and Files with a flag: lines and writes inventory.csv. Its exit code is 1 when anything is flagged, so a pipeline can stop on it. It prints warnings for damaged PDFs. That is expected. Leave out --manifest or --price-per-1000 if you have neither yet.

Never change the client's files

Work on a copy. Do not repair, rename, unlock or convert files in the original delivery. Keep the original so the client can see what they sent.

Part A. Agree the rules

Done by: business analyst, with the client contact.

  • A1. Name the batch and its contact. Give the batch a ${BATCH_ID} and name one client contact who can answer questions and approve fixes. Check: the report header shows the batch ID and the contact.
  • A2. Confirm we may use the files. Ask the client to confirm in writing that they may share the files, and to say if they hold personal or confidential data. Check: a written confirmation is stored with the batch.
  • A3. Write down the limits you test against. Record the Document Intelligence tier, the search tier and the size and page limits from the docs, with today's date. Check: the report header shows the tiers and limits. If the tier is free (F0), the report says that only two pages of each file are read, and the team picks another tier for a real batch.
  • A4. Agree what is in scope. List the formats, languages, date range and document types the client wants searchable, and what happens to anything outside them. Check: the scope list is in the report and the client has seen it.
  • A5. Ask for the client's packing list. Ask for the number of files and the total size they believe they sent. Ask for file hashes if they can make them. Check: the numbers are in the report (used in B1).
  • A6. Agree who signs and how long we keep it. Name who signs the report, and how long we keep the batch and the report. Check: both are written in the report header.

Part B. Count and sort

Done by: GenAI engineer.

  • B1. Check you received everything. Compare the client's file count and size with yours. A cut-off upload shows as fewer files or a smaller total. Check: the numbers match, or the gap is written down and sent to the client. If the client sent hashes, compare them too.
  • B2. Scan the copy for malware. Use your organisation's scanner on ${INTAKE_FOLDER} before anything opens the files. Client files come from outside our network. Check: the scan report is clean, or the infected files are removed and the client is told.
  • B3. Run the inventory script on the copy. Use the command in "Before you start". For a very large batch, run it on each sub-folder and keep one CSV for each. It reads every file to make its hash, so it takes time. Check: inventory.csv has one row for every file, including files with no extension.
  • B4. Count files by format, size and pages. Copy the totals into the report. Check: every format in the batch appears in the report, and the Billable pages line is copied for the cost estimate (F4).
  • B5. Look at the largest files. Sort inventory.csv by size and by pages and open the top few. The largest files are the ones most likely to hit a limit or time out. Check: each one opens and its page count looks right.

Part C. Can we read it?

Done by: GenAI engineer.

  • C1. Flag files over the size or page limit. Look for too_large and too_many_pages. Check: each flagged file has an action in the report: split it, or exclude it. Two limits count, the Document Intelligence one and the search indexer one.
  • C2. Flag password-protected files. Look for password_protected (PDF) and password_protected_or_old_format (Office). Open one Office file to see which it is. Check: each file is returned to the client for an unlocked copy. Do not ask for passwords by email. A PDF with only an owner password opens without one and is not flagged.
  • C3. Flag corrupt and empty files. Look for corrupt and empty_file. A .pdf that is really a different type also shows as corrupt. Check: each file is returned for a new export. If the client sent hashes, compare them to catch a damaged upload.
  • C4. Flag formats we cannot read. Look for unsupported_format and archive_unpack_first. Old Office files (.doc, .ppt, .xls) and email files (.msg, .eml) are not on the Layout skill list. Check: each file is converted, unpacked and re-checked (a zip can hold zips), or excluded with the client's agreement.
  • C5. Flag pictures that are too small or too large. Look for image_size_out_of_range. A multi-page TIFF counts each page as one page. Check: each image is re-scanned or resized, or excluded.
  • C6. Remove junk. Look for junk_file, such as ~$ lock files, Thumbs.db and .DS_Store. Check: the junk files are removed from the list of files to index. The client's copy is not touched.
  • C7. Check names that clash. Look for name_clash_with. Two names that differ only by case are one file on Windows and two on Linux. Check: each pair is renamed or one of them is excluded.
  • C8. Check odd names and long paths. Sort inventory.csv by name length. Look for leading or trailing spaces, unusual characters and very long paths, which can fail on some systems. The script does not flag these. Check: each odd name is renamed in the working copy, and the new name is written next to the old one in the report.

Part D. Is the content right?

Done by: business analyst and GenAI engineer.

  • D1. Skip exact duplicates. Look for duplicate_of. The script keeps the first name in sort order. Check: each duplicate is listed in the report with the file it copies. The pipeline in PB-RAG-05-01 skips them again by hash.
  • D2. Ask about versions. Look through the file names for groups such as v1, v2, final, copy and dates. These are not exact duplicates, so the script does not find them. Check: the client names the current version of each group. Older versions are excluded or kept on purpose.
  • D3. Estimate how many pages are scans. Read the Estimated image-only pages line. The script tests the first 20 pages of each PDF, so a long PDF with a scanned appendix can be missed. A scan with an old, poor text layer counts as text, so open a few text PDFs and look for garbled text. Check: the share is in the report. If it is above what the cost estimate assumed, raise it with the project lead.
  • D4. Open a sample by eye. Open a few files of each format, chosen across the batch. Check: each file is readable, in the expected language, not blank and not marked draft.
  • D5. Test the hard cases. Ask the client which files have tables, forms, several columns, handwriting, rotated pages or right-to-left text. Run one of each through the layout model in the sandbox and read the result. Check: the output keeps the text and tables in the right order, or the file is added to the "needs review" list.
  • D6. Look for sensitive content. Look for personal data, passwords in spreadsheets and unredacted records. Check: the client decides in writing to mask, exclude or accept each case.

Part E. Metadata and permissions

Done by: business analyst. The GenAI engineer runs the script.

  • E1. Agree the metadata we need. List the fields used for filters (for example document type and date) and the fields used for permissions (for example owner and groups). Check: the list of required fields is in the report.
  • E2. Check the manifest. Save the manifest as a CSV with a file column. The names must match the paths in the batch exactly, with / between folders. Run the script with --manifest and --require. It reports files missing from the manifest, rows with no file and empty required values. Check: the script prints no manifest: lines, or each line has an agreed fix.
  • E3. Check that permissions arrived. Every file needs its permission data. A file with none is held back, so it is visible to nobody until fixed. It is never visible to everybody. Build the filters in PB-RAG-04-02. Check: the number of files without permissions is in the report, and the client has the list.
  • E4. Check the index can store the metadata. The indexer fills only the metadata fields that already exist in the index and silently discards the rest. Check: every required field exists in the index definition before the first run.

Part F. Report and decide

Done by: RAG Lead, with the client contact.

  • F1. Fill in the report. Use snippets/intake/corpus-intake-report.md and attach inventory.csv. Check: every row has a result and an action.
  • F2. Agree a fix for every flagged file. The choices are return, convert, split, exclude or accept the risk. Each fix has an owner and a date. Check: the client has replied in writing. No flagged file is left without an action.
  • F3. Decide go or hold. Hold the batch if more than 5% of it fails the intake checks. This is a team default, not a platform fact. Check: the decision and its date are in the report.
  • F4. Get the cost approved. Take the Billable pages line (ready now, and after the fixes). Multiply by the current price using the sums in "Where the money goes". Add the search tier and the embeddings. Check the current price before approval. Check: the project lead has approved the estimate, and the report shows both numbers, so nobody is surprised after the fixes.
  • F5. Freeze the batch. Keep the report, inventory.csv and the manifest with the batch. Treat any file that arrives later as a new batch. Check: the three files are stored together under ${BATCH_ID}.

Part G. After the first ingestion run

Done by: GenAI engineer. Run this once the pipeline has processed the batch.

  • G1. Check every file is accounted for. Files received = processed + skipped duplicates + dead-lettered + excluded at intake. Check: the numbers add up. Use the daily report from PB-RAG-05-01.
  • G2. Review the dead letters. Compare each reason with the intake flags. Check: every dead letter was predicted at intake. If one was not, add its cause to part C.
  • G3. Compare processed text with the source. Open a sample of processed files next to their originals. Check: the text is complete, tables are intact, the order is right and no output is blank.
  • G4. Review the low-text files. Files marked low_text are probably scans. Check: the number matches the scan estimate from D3, and the client has a list of files that need a better scan.
  • G5. Ask known questions. Pick five questions whose answers you know are in the batch. Check: each question returns the right file. Send misses to the retrieval owner.
  • G6. Compare the bill with the estimate, then close the batch. Read the pages billed from your Azure usage and cost reports and compare them with the estimate from F4. Update the report with the real numbers and store it with the batch. Check: the report shows the estimate and the real figure side by side. A gap of more than the team accepts goes to the RAG Lead.

Build the intake script

Build the script once, then reuse it for every batch. The blocks below are the whole of inventory.py, in order. Paste them one after the other into snippets/intake/inventory.py, or copy the finished file from the same folder. Each block is one small piece, so you can read it on its own.

Group 1: set up and shared tools

  1. Create the file snippets/intake/inventory.py. Start it with a short note on what it does.
"""Corpus intake inventory for PB-RAG-05-04.

Walks a batch folder, counts files, flags the ones that will fail later, finds exact duplicates,
estimates the pages you will be billed for and checks the metadata manifest.
It only reads the files. It never changes them.

    pip install "pypdf[crypto]" pillow
    python inventory.py ${BATCH_FOLDER} --max-mb 500 --max-pages 2000 --manifest manifest.csv --require owner,acl_groups
"""
  1. Add the imports. They are all Python standard library, except PIL (pillow) and pypdf.
import argparse
import csv
import hashlib
import math
import re
import sys
import zipfile
from collections import Counter
from pathlib import Path

from PIL import Image
from pypdf import PdfReader
  1. Add the settings. These are the lists and limits the script uses. Change them in one place only.
ACCEPT = {".pdf", ".jpg", ".jpeg", ".png", ".bmp", ".tif", ".tiff", ".heif",
          ".docx", ".pptx", ".xlsx", ".html", ".txt", ".md", ".csv"}   # same list as the pipeline in PB-RAG-05-01
IMAGES = {".jpg", ".jpeg", ".png", ".bmp", ".tif", ".tiff"}             # .heif is not opened here: no plugin
OFFICE = {".docx", ".pptx", ".xlsx"}
PLAIN = {".txt", ".md", ".csv"}                                         # read as plain text, no paid service
ARCHIVES = {".zip", ".gz", ".7z", ".rar", ".msg", ".eml"}                # cannot be read as one file
JUNK_PREFIX = ("~$", "._")
JUNK_NAMES = {".ds_store", "thumbs.db", "desktop.ini"}
IMAGE_MIN, IMAGE_MAX = 50, 10000      # pixels per side, from the Layout skill page: check current limits in the docs
SAMPLE_PAGES = 20                     # pages tested for text in each PDF, to keep the run fast
CHARS_PER_PAGE = 3000                 # Word and HTML are billed in blocks of this size: check current rules in the docs
XML_READ_LIMIT = 20_000_000           # never read more than this from one part of an Office file
OLE_SIGNATURE = b"\xd0\xcf\x11\xe0"   # old Office files, and Office files protected with a password
  1. Add the fingerprint function. Plain words: it reads a file in 1 MB pieces, so even a huge file does not fill the memory, and returns its SHA-256.
def sha256_of(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(1 << 20), b""):
            digest.update(block)
    return digest.hexdigest()

Verify: run python -c "import inventory" in snippets/intake/. It prints nothing and shows no error.

Group 2: read each kind of file

  1. Add the PDF page counter. Plain words: it counts pages, flags a PDF with too many, and tests the first 20 pages for text. A page with no text is probably a scan.
def count_pdf_pages(reader, info, max_pages):
    pages = len(reader.pages)
    info["pages"] = info["billable"] = pages
    if pages > max_pages:
        info["flags"].append("too_many_pages")
    sample = list(reader.pages[:SAMPLE_PAGES])
    blank = sum(1 for p in sample if not (p.extract_text() or "").strip())
    info["scanned_pages"] = round(pages * blank / len(sample)) if sample else 0
  1. Add the PDF check. Plain words: a PDF with a real password cannot be read. A PDF with an empty password opens fine, so it is not flagged.
def check_pdf(path, info, max_pages):
    reader = PdfReader(str(path))
    if reader.is_encrypted and not reader.decrypt(""):     # an empty password is fine, a real one is not
        info["flags"].append("password_protected")
        return
    count_pdf_pages(reader, info, max_pages)
  1. Add the image check. Plain words: it opens the picture to prove it is not damaged, then checks its size in pixels.
def check_image(path, info):
    with Image.open(path) as img:
        img.verify()                                       # raises if the file is damaged
    with Image.open(path) as img:                          # verify() leaves the image unusable
        width, height = img.size
        info["pages"] = getattr(img, "n_frames", 1)
    info["billable"] = info["scanned_pages"] = info["pages"]   # a picture of text is always OCR work
    if min(width, height) < IMAGE_MIN or max(width, height) > IMAGE_MAX:
        info["flags"].append("image_size_out_of_range")
  1. Add the Office helpers. Plain words: a Word, Excel or PowerPoint file is a zip of small XML files. The helpers read one part with a size cap, then count billable pages the way the pricing rules do.
def read_part(z, name):
    """Read one part of an Office file as text, with a size cap. Returns '' if the part is missing."""
    if name not in z.namelist():
        return ""
    with z.open(name) as f:
        return re.sub(r"<[^>]+>", "", f.read(XML_READ_LIMIT).decode("utf-8", "ignore"))

def office_pages(z, ext):
    """Billing estimate: a sheet or a slide is one page, Word is one page for each block of characters."""
    names = z.namelist()
    if ext == ".xlsx":
        return max(1, sum(1 for n in names if re.match(r"xl/worksheets/sheet\d+\.xml$", n)))
    if ext == ".pptx":
        return max(1, sum(1 for n in names if re.match(r"ppt/slides/slide\d+\.xml$", n)))
    return max(1, math.ceil(len(read_part(z, "word/document.xml")) / CHARS_PER_PAGE))
  1. Add the Office check. Plain words: a file that starts with the old Office signature is either old or password-protected. Otherwise the zip must be sound.
def check_office(path, info):
    with open(path, "rb") as f:
        head = f.read(4)
    if head == OLE_SIGNATURE:
        info["flags"].append("password_protected_or_old_format")
        return
    with zipfile.ZipFile(path) as z:                       # a .docx is a zip file
        if z.testzip() is not None:
            info["flags"].append("corrupt")
        else:
            info["billable"] = office_pages(z, info["ext"])
  1. Add the HTML check. Plain words: HTML is billed in blocks of text, so it counts the characters left after the tags are removed.
def check_html(path, info):
    text = path.read_text(encoding="utf-8", errors="ignore")[:XML_READ_LIMIT]
    info["billable"] = max(1, math.ceil(len(re.sub(r"<[^>]+>", "", text)) / CHARS_PER_PAGE))

Verify: python -c "import inventory" still shows no error.

Group 3: judge one file, then the whole batch

  1. Add the early check. Plain words: some files need no deeper look. Junk and empty files stop here. So do archives and formats we do not read.
def early_flag(path, info):
    """Return a flag when the file needs no deeper look, otherwise None."""
    if path.name.lower() in JUNK_NAMES or path.name.startswith(JUNK_PREFIX):
        return "junk_file"
    if info["bytes"] == 0:
        return "empty_file"
    info["sha256"] = sha256_of(path)                       # junk and empty files get no hash
    if info["ext"] in ARCHIVES:
        return "archive_unpack_first"
    if info["ext"] not in ACCEPT:
        return "unsupported_format"
  1. Add the router. Plain words: it sends each file to the check for its type.
def deep_check(path, info, max_pages):
    ext = info["ext"]
    if ext == ".pdf":
        check_pdf(path, info, max_pages)
    elif ext in IMAGES:
        check_image(path, info)
    elif ext in OFFICE:
        check_office(path, info)
    elif ext == ".html":
        check_html(path, info)
  1. Add the main check for one file. Plain words: it builds the file's record, runs the early check, then the deep check. Any read error means the file is damaged.
def inspect(path, max_bytes, max_pages):
    info = {"file": path.name, "ext": path.suffix.lower(), "bytes": path.stat().st_size, "pages": 0,
            "billable": 0, "scanned_pages": 0, "sha256": "", "flags": []}
    flag = early_flag(path, info)
    if flag:
        info["flags"].append(flag)
        return info
    if info["bytes"] > max_bytes:
        info["flags"].append("too_large")
    try:
        deep_check(path, info, max_pages)
    except Exception:                                      # any read error means the file is damaged
        info["flags"].append("corrupt")
        info["billable"] = 0
    return info
  1. Add the duplicate finder. Plain words: files with the same fingerprint are the same content. The first name in sort order is kept, and the rest get a flag.
def mark_duplicates(rows):
    first = {}
    for row in sorted(rows, key=lambda r: r["file"]):      # the first name in sort order is kept
        digest = row["sha256"]
        if not digest:
            continue
        if digest in first:
            row["flags"].append("duplicate_of:" + first[digest])
        else:
            first[digest] = row["file"]
  1. Add the name clash finder. Plain words: two names that differ only by upper and lower case are a trap when files move between Windows and Linux.
def mark_name_clashes(rows):
    """Names that differ only by case are one file on Windows and two on Linux."""
    seen = {}
    for row in sorted(rows, key=lambda r: r["file"]):
        key = row["file"].lower()
        if key in seen:
            row["flags"].append("name_clash_with:" + seen[key])
        else:
            seen[key] = row["file"]
  1. Add the manifest check. Plain words: it compares the client's list with the real files. It finds files with no row, rows with no file and empty required values.
def check_manifest(manifest, required, files):
    """Return the problems in the manifest: missing rows, empty values and rows with no file."""
    problems = []
    with open(manifest, newline="", encoding="utf-8-sig") as f:
        listed = {r["file"]: r for r in csv.DictReader(f)}
    for name in sorted(files - listed.keys()):
        problems.append(f"{name}: not in the manifest")
    for name, row in sorted(listed.items()):
        if name not in files:
            problems.append(f"{name}: in the manifest but not in the batch")
        for col in required:
            if not (row.get(col) or "").strip():
                problems.append(f"{name}: empty {col}")
    return problems
  1. Add the batch runner. Plain words: it walks the folder, checks every file and runs the batch-wide checks.
def run(folder, max_mb, max_pages, manifest=None, required=()):
    root = Path(folder)
    paths = sorted(p for p in root.rglob("*") if p.is_file())
    rows = [inspect(p, max_mb * 1024 * 1024, max_pages) for p in paths]
    for row, path in zip(rows, paths):
        row["file"] = path.relative_to(root).as_posix()
    mark_duplicates(rows)
    mark_name_clashes(rows)
    problems = check_manifest(manifest, required, {r["file"] for r in rows}) if manifest else []
    return rows, problems

Verify: python -c "import inventory" still shows no error.

Group 4: sums, report and the command line

  1. Add the page and cost sums. Plain words: "ready now" is the pages of files with no flag. "Held back" is the pages of flagged files that you would still pay for once fixed. Duplicates, junk and empty files are never paid for. The price is yours: the script never knows it.
def billable_pages(rows):
    """Return (pages ready now, pages held back by a flag). Duplicates, junk and empty files are never paid for."""
    ready = sum(r["billable"] for r in rows if not r["flags"] and r["ext"] not in PLAIN)
    held = sum(r["billable"] for r in rows if r["flags"] and not any(
        f.startswith("duplicate_of") or f in ("junk_file", "empty_file") for f in r["flags"]))
    return ready, held

def cost_lines(rows, price_per_1000):
    ready, held = billable_pages(rows)
    lines = [f"Billable pages, estimate: {ready} ready now, {held} more if every flagged file is fixed"]
    if price_per_1000 is not None:                         # you give the price: check the current price before approval
        lines.append(f"Estimated cost: {ready * price_per_1000 / 1000:.2f} now, "
                     f"{(ready + held) * price_per_1000 / 1000:.2f} after the fixes (your price, your currency)")
    return lines
  1. Add the report printer. Plain words: it prints the totals first, then each flagged file.
def print_totals(rows, price_per_1000):
    by_ext = Counter(r["ext"] or "(none)" for r in rows)
    pages = sum(r["pages"] for r in rows)
    scanned = sum(r["scanned_pages"] for r in rows)
    print(f"Files: {len(rows)}   Bytes: {sum(r['bytes'] for r in rows)}   Pages: {pages}")
    print("By format:", ", ".join(f"{e}: {n}" for e, n in sorted(by_ext.items())))
    print(f"Estimated image-only pages: {scanned} of {pages}" + (f" ({100 * scanned // pages}%)" if pages else ""))
    print(*cost_lines(rows, price_per_1000), sep="\n")

def report(rows, problems, price_per_1000=None):
    flagged = [r for r in rows if r["flags"]]
    print_totals(rows, price_per_1000)
    print(f"Files with a flag: {len(flagged)} of {len(rows)}")
    for r in flagged:
        print(f"  {r['file']}: {', '.join(r['flags'])}")
    for p in problems:
        print("  manifest:", p)
    return len(flagged) + len(problems)
  1. Add the CSV writer and the command-line entry point.
def write_csv(rows, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["file", "ext", "bytes", "pages", "billable_pages", "scanned_pages", "sha256", "flags"])
        for r in rows:
            w.writerow([r["file"], r["ext"], r["bytes"], r["pages"], r["billable"], r["scanned_pages"],
                        r["sha256"], ";".join(r["flags"])])

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("folder")
    ap.add_argument("--max-mb", type=int, default=500)     # set from your tier: check current limits in the docs
    ap.add_argument("--max-pages", type=int, default=2000)
    ap.add_argument("--manifest")
    ap.add_argument("--require", default="")
    ap.add_argument("--price-per-1000", type=float)        # your current price for 1,000 pages: check before approval
    ap.add_argument("--csv", default="inventory.csv")
    args = ap.parse_args()
    rows, problems = run(args.folder, args.max_mb, args.max_pages, args.manifest,
                         [c for c in args.require.split(",") if c])
    write_csv(rows, args.csv)
    sys.exit(1 if report(rows, problems, args.price_per_1000) else 0)
  1. Add the last two lines of the file.
if __name__ == "__main__":
    main()

Verify: run the script on a small test folder. You should see the totals, the Billable pages line and one line for each flagged file. It exits with code 1 when anything is flagged.

Run the tests

The tests in snippets/intake/test_inventory.py plant the faults for you.

pip install pytest
python -m pytest snippets/intake/test_inventory.py

Verify: every test passes. They cover each flag, duplicates, scans, the manifest, name clashes, empty folders, billable pages and the cost lines.

What the output looks like

This is the real output on the seeded test batch (21 files, limits set to 1 MB and 2 pages). With --price-per-1000 it adds an Estimated cost: line after the Billable pages line.

Files: 21   Bytes: 2005206   Pages: 11
By format: (none): 1, .db: 1, .docx: 4, .dwg: 1, .msg: 1, .pdf: 9, .png: 3, .zip: 1
Estimated image-only pages: 7 of 11 (63%)
Billable pages, estimate: 5 ready now, 5 more if every flagged file is fixed
Files with a flag: 17 of 21
  bad.docx: corrupt
  big.pdf: too_large
  broken.pdf: corrupt
  bundle.zip: archive_unpack_first
  drawing.dwg: unsupported_format
  empty.pdf: empty_file
  fake.png: corrupt
  good.pdf: duplicate_of:copy-of-good.pdf
  locked.docx: password_protected_or_old_format
  locked.pdf: password_protected
  long.pdf: too_many_pages
  mail.msg: archive_unpack_first
  noext: unsupported_format
  sub/good.pdf: duplicate_of:copy-of-good.pdf
  Thumbs.db: junk_file
  tiny.png: image_size_out_of_range
  ~$ok.docx: junk_file

5. Known issues and fixes

Symptom Likely cause Fix
Many files rejected at indexing Limits not checked at intake Split or compress large files before indexing. Run part C again
Long documents are only partly searchable The free tier (F0) reads the first two pages only Use the Standard tier, then re-run the batch
Indexer stops at the first odd file The blob indexer stops at an unsupported content type Exclude the extensions in the indexer settings, or remove the files at intake
Permissions missing on half the files Export from the source dropped the ACLs Ask the client for a permissions export, or index through a source connector
Filters return nothing for a field The field was not in the index, so the value was dropped Add the field to the index and re-run the batch
The same answer from two files Two versions of one document were both indexed Ask the client for the current version and exclude the others (item D2)
Unexpected formats such as .msg or .zip Not in the original scope Add a handler or agree an exclusion with the client
A scanned file gives empty or garbled text Poor scan: low resolution, skew or a stamp over the text Ask for a better scan or send the file to manual review
A scan passes as "text" but the text is garbage An old, poor OCR layer is inside the PDF Re-scan, or send the page images to the layout model instead (item D3)
File counts differ between us and the client Cut-off upload or hidden files skipped Compare with the packing list and hashes, and ask the client to resend
The bill is higher than the estimate Files were re-read after fixes, or files timed out Fix flags before the first run, split big files, and re-check the pages with item G6
Manifest check says "not in the manifest" for every file File names in the manifest use \ or a different case Write the names exactly as the script prints them, with /
The script is very slow It reads every file in full to make the hash Run it on each sub-folder, or on a fast local disk

6. Monitoring and alerts

The thresholds below are team defaults, not platform facts.

Signal Alert when Action
Share of files failing intake checks Above 5% of a batch (team default) Hold the batch and raise with the client
Share of image-only pages Above the level assumed in the estimate Review OCR cost and quality with the project lead
Pages billed against the estimate Above the accepted gap from item G6 Find the cause (re-reads, timeouts) and tell the project lead
Dead letters not predicted at intake Any in the first run Add the cause to part C and re-check the batch
Files without permission data Any Hold those files until the client replies
Files received against the client's packing list Any difference Ask the client to resend

7. AWS and open-source equivalents

The same intake checks apply before loading S3 data sources for Amazon Bedrock Knowledge Bases or Textract pipelines. Only the limits, the format lists and the prices change. The script runs the same way for any target, but set its limits and format list to the target first.

Platform Equivalent Key differences
Amazon Textract Reads JPEG, PNG, PDF and TIFF. Synchronous calls take 1 page. Asynchronous calls take up to 3,000 pages and 500 MB for PDF and TIFF PDFs cannot be password protected. XFA-based PDFs are not supported. Office files are not read. Check current limits in the docs
Amazon Bedrock Knowledge Bases Supports txt, md, html, doc, docx, csv, xls, xlsx and pdf. Each file must be within the 50 MB quota Check the file size quota and the format list before loading S3. Images have their own size limit. Check current limits in the docs
Open source A parser of your choice, such as a PDF text library Each parser has its own limits and failure modes. Run the script first, then test one file of each format. Check the project docs
Cost on AWS Textract is billed by the page Use the same Billable pages line, and check the current price on the Amazon Textract pricing page

8. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: the dry-runner builds a seeded batch with known defects, runs this checklist on it and compares the report with the answer key. In snippets/intake/, create the batch and run the script:

mkdir seed
python -c "import pathlib, test_inventory; test_inventory.seed(pathlib.Path('seed'))"
python inventory.py seed --max-mb 1 --max-pages 2 --price-per-1000 ${PRICE_PER_1000_PAGES}

The batch has 21 files. It plants a password-protected PDF, a damaged PDF, a damaged Office file, a password-protected Office file, an empty file, two duplicates, a scanned PDF, a PDF over the page limit, a file over the size limit, a tiny image, a damaged image, unsupported and archive files, and junk files. The expected output is the sample in section 4.

Pass criteria: every planted defect appears with the right flag (17 files flagged), the page line reads 5 ready now and 5 more after the fixes, the report has an action for each flag, and the cost line matches pages ÷ 1,000 × the price used.

Record

Item Detail
Playbook PB-RAG-05-04 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

9. Sources

Checked on 3 November 2026.

  • PB-RAG-05-01: the pipeline that reads the files after this checklist passes.
  • PB-RAG-05-02: when indexing fails on a live index.
  • PB-RAG-05-03: deletions, duplicates and refresh after go-live.
  • PB-RAG-04-02: builds the permission filters that items E1 to E3 prepare for.