Skip to content
⚠️ Before You Start
Risk: Medium
Duration: 1-2 days per corpus, plus extraction run time
Last Validated: Not yet validated (draft)
Kind: Checklist
Risk Note: Medium (skipping it leads to commitments the pipeline cannot meet)

Per-corpus extraction validation checklist

Draft outline: not validated

Drafted with AI from official sources on 16 Oct 2026. Outline checked against the sources on 5 November 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-25). Author: Kandiah Chockalingam · dry-run: Ranjith · sign-off: Yogeshwaran. Work items: RAG-10-T09, RAG-10-T10, RAG-10-T11, RAG-10-T12.

How to read this page

New to this? Read section 1 and the "Think of it as" line. Then go to section 4 and work from part A down. Each item says how to check it. Section 3 explains the words, the limits, the costs and the pass marks. You can read it later.

1. Summary

A client wants search and answers over documents full of tables, charts and scans. Before we promise scope, quality or price, we test our pipeline on a small, honest sample of their documents. We read the sample, score the results by hand, ask real questions, work out the cost and write down one decision: go, go with conditions, no-go or not decided.

Run it once for every new corpus, and again when the document types, the scanner, the skill, the API version or the price change. The GenAI engineer runs it with the RAG lead. The pre-sales architect uses the result to set scope and price. Maturity is Level 1, so results vary by corpus and nothing is promised until this is done.

Think of it as a test drive before you sign for the car. You do not trust the brochure. You drive the car on the client's own road, in bad weather, and you read the fuel bill.

This playbook covers extraction only. It does not tune chunking, search or the answer model.

Risk: Medium (skipping it leads to commitments the pipeline cannot meet). Typical duration: 1-2 days per corpus, plus extraction run time.

How it is organised

Part Question it answers Done by
A. Agree the rules May we test this, and what are we testing? RAG lead, pre-sales architect
B. Choose the sample Is the sample honest, not convenient? GenAI engineer, client
C. Check the limits Will the files even be accepted? GenAI engineer
D. Run the extraction What failed, what was slow, what was empty? GenAI engineer
E. Score the extraction Are the tables, figures and scans right? Two scorers
F. Score the answers Do real questions get right answers? GenAI engineer, client
G. Cost, time and decision What does it cost, and do we go? RAG lead

2. When to use and when not to

Use this checklist when:

  • You scope or price an engagement whose documents have tables, figures or scans.
  • A client sends a new corpus, or adds a new document type or a new scanner to an existing one.
  • You change the extraction path, the skill, the API version or the settings, and must prove nothing got worse.
  • A price or limit changes and the old go decision may no longer hold.

Do not use it when:

  • You only need to check that files can be opened and counted. Use PB-RAG-05-04 instead.
  • You are choosing between extraction paths. Use PB-RAG-10-01 instead.
  • You are building the extraction. Use PB-RAG-10-02 instead.
  • Extraction is failing, slow or over budget on a live system. Use PB-RAG-10-04 instead.
  • The corpus is plain text only. There is nothing to validate here.
  • You want to tune chunking, search or the answer model. This checklist only finds out whether extraction is the cause.

3. What you must know first

  • The service that reads the file decides what you get. The Azure Content Understanding skill is generally available in the 2026-04-01 REST API. It writes tables and figures as Markdown and can join a table that runs across pages into one table. Check current behaviour in the docs.
  • Some features are preview. Image descriptions for charts and diagrams, and semantic chunking, are preview in 2026-05-01-preview. Image descriptions also need an Azure OpenAI chat model in the Foundry resource. Do not promise a preview feature as a fixed deliverable.
  • The Document Layout skill is the other path. It calls Document Intelligence v4.0 (2024-11-30). It gives 20 free documents for each indexer each day. The Content Understanding skill gives none and always needs a billable Foundry resource. The Layout skill's default output is Markdown. In text mode, tables and figures come out as plain text.
  • The smallest limit in the chain wins. A file must pass the search tier limit, the skill limit and the service limit. A 100 MB file passes Content Understanding and still fails on a Basic search tier. A document that needs more than five minutes of processing times out, and the charge still applies.
  • Scans, handwriting and low-resolution images lower the quality. Document Intelligence v4.0 reads handwriting in only 12 languages. Score scans by source (item E8), never as one group.
  • Measure cost per 1,000 pages, not per document. Documents differ in length, and each service counts pages its own way (see "Where the money goes").
  • Hand-scored results are small-number results. Twenty pages is a rough guide: one mistake moves the score by 5 points. Write "estimate" next to every figure, and never quote a score without its sample size.

The limits that matter

Check current limits in the docs on the day. These were copied on 5 November 2026.

Limit Value Source
Source file size in an indexer, by search tier Free 16 MB, Basic 16 MB, S1 128 MB, S2 256 MB, S3 256 MB Service limits for tiers and SKUs in Azure AI Search
Characters extracted from one file, by search tier Free 256,000, Basic 512,000, S1 4 million, S2 8 million, S3 16 million The same page
Indexer run time Free: 3 minutes for blobs, or 3 to 10 minutes with a skillset. Paid tiers: plan for 2 hours The same page
Layout skill file size and pages S0: 500 MB, and 2,000 pages for PDF and TIFF. F0: 4 MB and only the first 2 pages Document Layout skill
Content Understanding file size and pages 200 MB and 2,000 pages for PDF, TIFF and images (async). For Word, Excel and PowerPoint files the limit is in characters. The skill page points to these limits Azure Content Understanding service quotas and limits
File formats the skill reads Content Understanding skill: PDF, JPEG, JPG, PNG, BMP, HEIF, TIFF, DOCX, XLSX, PPTX, HTML, TXT, MD, RTF, EML. Document Layout skill: PDF, JPEG, JPG, PNG, BMP, TIFF, DOCX, XLSX, PPTX, HTML. Convert anything else first (item C4). The service pages list more formats, but the skill reads only its own list Both skill pages
Content Understanding throughput 1,000 pages or images a minute (S0) The same page
Image size, both skills 50 to 10,000 pixels each side Both skill pages
Skill time-out 5 minutes, and still billed Both skill pages
Document Intelligence throughput S0 default: 15 analyse requests a second. A 429 reply means you are over the limit Document Intelligence service quotas and limits
Handwriting 12 languages, the same list for Document Intelligence v4.0 and Content Understanding Document Intelligence OCR language support; Content Understanding region and language support

Where the money goes

Never guess a price. This playbook gives the sums. You get the price from the official page on the day, and you check the current price before approval.

Cost What drives it How to work it out Where to get the price
Reading the pages Pages the service bills. Each service counts pages its own way. Document Intelligence (Layout skill): one PDF page or TIFF image is one page, one Excel sheet or PowerPoint slide is one page, and Word and HTML count in blocks of 3,000 characters. Content Understanding: Word uses its own pages, one Excel sheet (hidden sheets too) or slide is one page, and text, HTML, Markdown and email count in blocks of 3,000 characters Real bill ÷ pages billed × 1,000 (item G1) Azure Content Understanding pricing, Azure Document Intelligence pricing
Figure descriptions (preview) The chat model that writes them, charged on tokens Run a small sample and read the model line of the bill Check the current price of your model
Re-runs Every run after a fix bills the pages again Plan for at least two full runs (team default) The same pages
Time-outs and failed files Billed even though they fail Split big files before the run. Count failed files in the bill The same pages
Free tier Not billed, but it reads only two pages (Layout skill, F0) Never validate on a free resource The Document Layout skill page
The search service Tier, storage and vector size of the validation index Delete the validation index when you finish Azure AI Search pricing
Embeddings Every chunk of text Tokens from the extracted text, then your model's price Check the current price of your model
Storing extracted images Blob storage for the images the skill writes out Count the images in the sample, scale to the corpus Azure Storage pricing: check the current price
AWS Amazon Textract Pages The same page count Amazon Textract pricing
People Hours to choose, score and write up The duration at the top is a team estimate. The dry-run records the real hands-on and waiting time. If it is outside this estimate, update it before sign-off Your project lead

How to score

Score by hand, on paper or in a sheet, one row for each page. Write the correct answer (the ground truth) before you look at the extracted text.

Measure What to count Right means
Table fidelity Every cell of every table on the page. A merged cell counts once The text matches and the cell is in the right row and column. Header cells count
Number accuracy Every number you decide to check, in tables or in text Exactly right: digits, decimal point, minus sign or brackets, currency sign, unit, date
Figure coverage Every chart, diagram, photo, stamp or signature a person needs to answer a question. Skip decoration It appears in the output as a correct description, as text, or as an image that search can return. A wrong description is not found
Answer accuracy Every test question Right = 1, partly right = 0.5, wrong or no answer = 0. For a "not in the documents" question, only "no answer" is right

Use one document-type name for each kind of document or scan source (for example digital_pdf, scanned_pdf, docx). Start the name of a deliberate worst case with worst- (for example worst-rotated).

The scoring script reads two sheets. Save them as CSV.

# pseudo: pages.csv, one row for each scored page
doc_type,page,table_cells,table_cells_right,figures,figures_found,numbers,numbers_right
digital_pdf,1,10,10,2,2,10,10
scanned_pdf,1,8,6,0,0,5,4

# pseudo: questions.csv, one row for each question
id,kind,result
q1,table,right
q2,not_in_docs,no_answer

kind is table, figure, text or not_in_docs. result is right, partly, wrong or no_answer.

The decision table

These pass marks are team defaults, not platform facts. The pack owner confirmed them on 5 November 2026 (90% tables, 80% figures, 85% answers, and 70% as the No-go line). The pack owner also confirmed the evidence rules (team defaults): at least 50 documents, 20 scored pages for each type, 3 pages for each worst case, 10 hold-out pages for each main type, and 30 questions with at least 10 about tables and 10 about figures. The scores are not trusted when the hold-out is more than 10 points worse than the sample, when the two scorers differ by more than 5 points, or when more than 5% of documents have errors or warnings. Change them when real validations show they are too strict or too easy. The script applies them from one place: the values at the top of snippets/eval/score_validation.py.

Situation Choose Why
Table fidelity at least 90%, figure coverage at least 80%, answer accuracy at least 85%, documents with errors or warnings at most 5%, and every document type passes Go Every measure meets the bar
A measure is between 70% and its bar, or one document type fails, or the cost is over the budget Go with conditions It works for part of the corpus. Write the limits into the scope. If the cost cannot be brought under budget by a cheaper path or a smaller scope, choose No-go
Any measure is under 70% No-go Promising this scope would fail. Offer a smaller scope or another path
The sample is too small (fewer than 50 documents, or a type below its scored-page minimum), a measure was not scored, there is no hold-out, the hold-out is more than 10 points worse, more than 5% of documents have errors or warnings, or the two scorers differ by more than 5 points Not decided The evidence is not good enough. Fix the cause and score again

Plain-words glossary

Word Plain meaning
Corpus All the documents one client wants searched
Sample The small part you test. A badly chosen sample makes the whole result worthless
Hold-out Pages you set aside and do not look at while you tune. You score them once at the end
Ground truth The correct answer, written by a person before they see the output
Table fidelity The share of table cells that come out right, in the right row and column
Figure coverage The share of charts and pictures that become something search can find
Answer accuracy The share of test questions answered correctly
Worst case A page you chose on purpose because it is hard: rotated, faded, handwritten, very wide
Go with conditions It works, but only for some documents. The conditions go into the scope
429 A reply meaning "too many requests". Slow down

4. Checklist

Tick each item as you go. Every item says how to check it.

Part A. Agree the rules

Done by the RAG lead and the pre-sales architect.

  • A1. Write the goal in one sentence, for example "answer questions about the tables in monthly reports". Verify: a person outside the team reads it and agrees.
  • A2. Get written permission to process the sample: where, by whom and for how long. Verify: an email or contract clause is on file.
  • A3. Ask whether the documents hold personal, patient, payment or secret data. If yes, agree masking or a locked-down workspace before you copy anything. Verify: the answer is on file. Never put sample data in a shared or personal tool.
  • A4. Name three people: the two scorers (item E13) and the person who approves the decision (the RAG lead). Verify: three names in the sheet.
  • A5. Decide what you test: the extraction path, the search tier, the region, the API version, and whether a preview feature is allowed. Verify: the values are in the results record.
  • A6. Check the Foundry resource is billable and in a region the skill supports. Document Layout skill billed with a key: the Foundry resource and the search service must be in the same region (billed with Microsoft Entra ID, there is no same-region rule). Content Understanding skill: any supported region, but a different region from the search service adds delay. Verify: both regions are noted and follow the rule for your skill.
  • A7. Set a budget cap for the validation run and name the person who may approve more. Verify: a number and a name.
  • A8. Agree when the sample and the validation index are deleted. Verify: a date in the sheet (item G10 does it).

Part B. Choose the sample

Done by the GenAI engineer with the client.

  • B1. List every document type, scan source, language and layout, with rough shares of the corpus. Verify: the shares add up to about 100%.
  • B2. Take at least 50 documents in all (team default), so the sample has variety: different authors, scanners and layouts. Then take at least 20 scored pages (team default) from each type. Do not drop a small type if the client cares about it. Verify: at least 50 documents, and every type reaches its page minimum.
  • B3. Add the worst cases on purpose, at least 3 pages each (team default): oldest scans, faded or skewed pages, rotated pages, handwriting, stamps, very wide tables, tables across pages, merged cells, multi-column pages, charts with no labels, very long files. Name each type worst-.... Verify: each worst case has its own row.
  • B4. Take whole tables. If a table runs across two pages, take both pages. Verify: no table in the sample is cut off.
  • B5. Let the client choose part of the sample, and draw the rest at random yourself. Verify: the sheet says who chose what.
  • B6. Copy the sample and keep the original untouched. Record a hash of each file. Verify: hashes match after the copy. Run sha256sum or the intake script.
  • B7. Set aside a hold-out set (team default: at least 10 pages from each main type). Do not look at it while you tune. Verify: it is in its own folder and nobody has scored it yet.
  • B8. Keep the sample small enough to score by hand in a day. Verify: the page count fits the time you have. If it does not, cut types, not worst cases.

Part C. Check the limits

Done by the GenAI engineer. For plain intake problems (duplicates, junk files, hashes) run the intake checklist PB-RAG-05-04 first.

  • C1. Run the intake script from PB-RAG-05-04 on the sample. Verify: the report lists every flagged file.
  • C2. Compare every file with the smallest limit in the chain: search tier file size, skill file size and pages, image size, extracted characters. Verify: a table of file, limit, pass or fail.
  • C3. Remove password locks, or mark the file rejected. Never guess a password. Verify: locked files are listed with a decision.
  • C4. Check each file type against the skill's list. Convert or exclude files the skill does not read, and record which. Verify: every unsupported file has a decision. Word and PDF files of the same document can give different results, so note it.
  • C5. Check image sizes (50 to 10,000 pixels each side) and HEIF files. Verify: out-of-range images are listed.
  • C6. List files that may run past five minutes of processing (very long or very large). Plan to split them. Verify: the list exists. A time-out is still billed.
  • C7. Check the free-tier rules. The Layout skill gives 20 free documents for each indexer each day. The Content Understanding skill gives none. Verify: a billable Foundry resource is attached to the skillset for every real run.
  • C8. Estimate the characters in the longest files and compare them with the extracted-character limit of your tier. Verify: no long file is likely to be cut.
  • C9. Plan the throughput. Compare pages a minute with the service limits and plan for 429 replies. Verify: a run plan with the expected hours.

Part D. Run the extraction

Done by the GenAI engineer.

  • D1. Run on a clean validation index, never the production index. Name it with a -validation suffix. Verify: the index name is in the sheet.
  • D2. Record start time, end time, documents and pages billed. Verify: the numbers are in the results record. Take pages billed from the bill or service metrics, not from your own count.
  • D3. List every error, warning and time-out with its file name. Verify: the count matches the indexer status.
  • D4. Record 429 replies and retries. Verify: the number and any delay are noted.
  • D5. Find documents that "succeeded" but came out empty or very short. Verify: compare characters extracted with the page count.
  • D6. Find documents cut at the extracted-character limit. Verify: the end of each long output matches the end of the original.
  • D7. Run the same sample a second time and compare. Verify: differences are small and you can explain them. Extraction can vary between runs.
  • D8. Keep the raw output of every document. Verify: you can re-score without running again, which saves a second bill.
  • D9. Record which preview features you used. Verify: the results record names each one and the API version.

Part E. Score the extraction

Done by two scorers. Use the scoring rules in section 3.

  • E1. Write the ground truth for each scored page before reading the output. Verify: the ground-truth file is dated before the scores.
  • E2. Score the pages: 20 for each type and 3 for each worst case (team defaults). Verify: every type has its rows. Say plainly that 20 pages is a rough guide.
  • E3. Score table fidelity cell by cell. Verify: a percentage for each type.
  • E4. Check table structure: header rows kept, merged cells, tables across pages joined, nested tables, rotated or very wide tables, tables that are pictures, and footnotes under a table. Verify: each case is marked pass or fail in the notes.
  • E5. Count number errors on their own: decimals, thousands separators, negatives in brackets, currency signs, units, dates. Verify: a separate number accuracy figure. One wrong digit is a wrong cell.
  • E6. Score figure coverage. Verify: a percentage for each type.
  • E7. Spot-check figure descriptions (preview): at least 10 for each type. A confident wrong description is worse than none. Verify: wrong descriptions are counted as not found.
  • E8. Score scans by source: low resolution, skew, shadows, faded text, stamps, handwriting. Verify: a score for each scan source, and the handwriting languages checked against the supported list.
  • E9. Check reading order: multi-column pages, side notes, repeated headers and footers, text boxes. Verify: text reads in the right order, and repeated headers do not drown out the content.
  • E10. Check other structure: headings, lists, footnotes, formulas, checkboxes, signatures, page numbers. Verify: each is marked pass or fail.
  • E11. Check every language, including mixed-language pages and right-to-left text. Verify: each language has a score, and unsupported ones are listed. Check each language against the list for your skill: Content Understanding region and language support or Document Intelligence OCR language support.
  • E12. Check Office files: every Excel sheet (including hidden ones) and whether you get values or formulas, PowerPoint slides and notes, Word headings and embedded images. Verify: each case is marked pass or fail.
  • E13. A second person re-scores 10% of the pages. Verify: the two scores differ by no more than 5 points (team default). If they differ more, agree the rule, write one example, and re-score.

Part F. Score the answers

Done by the GenAI engineer with the client.

  • F1. Write the questions with the client: at least 30, with at least 10 about tables and 10 about figures (team defaults). Verify: the count and the mix.
  • F2. Add hard questions: a number that needs two cells, an answer that needs two pages, a unit conversion, and questions the documents cannot answer. Verify: each kind exists. Mark the last kind not_in_docs.
  • F3. Write the correct answer and the source page for each question first. Verify: the answer key is dated before the run and the client has seen it.
  • F4. Run the questions on the validation index. Verify: the answer, the source and the retrieved text are saved for each question.
  • F5. Score each answer: right, partly right, wrong or no answer. A confident wrong answer is a failure. Verify: the totals add up to the number of questions.
  • F6. For every wrong answer, find the cause: extraction, chunking, search or the model. Only extraction faults count against this checklist. Verify: every miss has a cause, and non-extraction causes are handed to the right playbook.
  • F7. If you change the path or the settings, score again from part D. Verify: the results record says which run it describes.

Part G. Cost, time and decision

Done by the RAG lead.

  • G1. Work out cost for 1,000 pages: the real cost of the run ÷ pages billed × 1,000. Verify: both numbers come from the bill, and the sheet says where.
  • G2. Count every cost line: reading pages, figure descriptions, embeddings, search tier, image storage and re-runs. Verify: one line for each. Billing data can lag, so check how soon costs show in Cost Management.
  • G3. Project the full corpus: pages × cost for 1,000 pages ÷ 1,000, times the number of runs you plan (team default: at least two). Verify: the assumptions are written next to the sum.
  • G4. Project the time for the full corpus and compare it with the indexer run limit, the five-minute limit for one document and the throughput limits. Verify: total hours and number of runs.
  • G5. Compare the projection with the agreed budget. Verify: a yes or no, with both numbers.
  • G6. Run the scoring script, or apply the decision table by hand. Verify: the decision follows from the numbers. If the result is "not decided", fix the cause and score again.
  • G7. Write the conditions in plain words: what is out of scope, what needs a manual step. Verify: each condition can be tested, for example "scans under 200 dpi are out of scope".
  • G8. Share the report with the client and get written acknowledgement. Verify: the acknowledgement is on file.
  • G9. Write down when to validate again: a new document type or scanner, a new skill or API version, a price change, or a preview feature going GA. Verify: the triggers are in the record.
  • G10. Delete or lock the sample and the validation index on the agreed date. Verify: the deletion is recorded and the search service cost stops.

Run the scoring script

The script is optional, but it saves time and removes arithmetic mistakes. It reads your two CSV sheets, applies the decision table from section 3 and writes the results record. It never changes your sheets. It needs Python 3 and uses only the standard library, so there is nothing to install.

  1. Copy snippets/eval/score_validation.py to your working folder. Verify: python score_validation.py --help lists the two sheets and the options.
  2. Check the thresholds at the top of the script (BAR, FLOOR and the MAX_ and MIN_ values). They are the team defaults from the decision table. Change them only when the RAG lead agrees a different bar. Verify: the values match the decision table in section 3.
  3. Try it on three rows (below), then run it on your real sheets.
  4. Open the results record it writes. Fill in reviewer, then add part 2 of snippets/eval/extraction-results.yaml by hand: search tier, region, run times, conditions and re-validation triggers. Verify: no field in the record is empty, except a measure you skipped with a written reason.

Try it on three rows

Save these two tiny files as pages.csv and questions.csv. They are too small to decide anything, and the script says so.

# pseudo: pages.csv
doc_type,page,table_cells,table_cells_right,figures,figures_found,numbers,numbers_right
digital_pdf,1,10,10,2,2,10,10
digital_pdf,2,10,9,1,1,12,12
scanned_pdf,1,8,6,0,0,5,4

# pseudo: questions.csv
id,kind,result
q1,table,right
q2,figure,partly
q3,not_in_docs,no_answer

python score_validation.py pages.csv questions.csv --skip holdout --docs 3 --failed-docs 0 --pages-billed 3 --cost ${RUN_COST}

Verify: the output is the list below. The script refuses to decide on a sample this small. That is the check that it works.

Decision: not_decided
  - digital_pdf: 2 scored pages, need 20
  - scanned_pdf: 1 scored pages, need 20
  - 3 questions, need 30
  - fewer than 10 table questions
  - fewer than 10 figure questions
  - 3 documents in the sample, need 50
Written: extraction-results.yaml

For a real run, give the numbers from the bill and the sheets:

python score_validation.py ${PAGES_CSV} ${QUESTIONS_CSV} --holdout ${HOLDOUT_CSV} --docs ${DOCS} --failed-docs ${FAILED_DOCS} --pages-billed ${PAGES_BILLED} --cost ${RUN_COST} --budget-per-1000 ${BUDGET_PER_1000_PAGES} --minutes ${MINUTES} --scorer-gap ${SCORER_GAP}

Use --skip tables, --skip figures or --skip holdout only when that measure is genuinely not in scope, and write the reason in the record.

5. Known issues and fixes

Symptom Likely cause Fix
The sample passes but production fails The sample was chosen for convenience Choose it with the client. Add worst cases and a hold-out set
Time-outs on large files The document needs more than five minutes of processing Split large files before indexing. The timed-out call is still billed
A file fails on a Basic search tier but is fine under the skill limit The search tier limit is smaller than the skill limit Compare with the smallest limit in the chain. Change tier or split the file
A long file ends early in the output The extracted-character limit of the tier Check the limit for your tier. Split the file
Two scorers give different scores The scoring rule is unclear Write the rule and one example per case. Score again
Tables look right but the numbers are wrong Digit errors in scanned pages Count number errors on their own. Add a number check or a manual step as a condition
A table across two pages comes out as two tables The skill did not join them, or the layout is unusual Test cross-page tables on their own. Record the result as a condition
Charts leave no text in the index Image descriptions are not switched on (preview) Decide whether preview is allowed. If not, treat charts as out of scope or add a manual step
A figure description is confident but wrong The model describes what it cannot read Spot-check. Mark charts that carry data as a manual step
Cost per 1,000 pages is far above the estimate Re-runs, time-outs, no free tier or different page counting Read the bill line by line. Check how pages are counted
429 replies during the run You are over the per-second limit Slow the run. Ask for a higher limit through support if you need it
A free-tier run passes but the paid run fails, or the reverse The free tier reads only the first two pages Never validate on a free resource
The hold-out is much worse than the sample You tuned to the sample Fix the cause, never the sample. Take a new hold-out
Answers are wrong but the extraction is right The fault is in chunking, search or the model Log it and hand it to PB-RAG-10-02 or the retrieval playbooks
The script says "Cannot score" A sheet has a missing column, a blank or a word where a number belongs Read the line number in the message and fix that line
The script says "not decided" but the sample feels fine A measure, the hold-out or enough pages or questions is missing Read the reasons. Add the missing evidence. Do not lower the bar to get a go
A client sends more documents after the go New types or scanners were not in the sample Run this checklist again on the new types

6. Monitoring and alerts

Alert thresholds are team defaults, not platform facts, confirmed by the pack owner on 5 November 2026. The five-minute limit is a platform fact (both skill pages). The four-minute alert is our safety margin. Send every alert to the RAG Lead, the owner role of this checklist.

Signal Alert when Action
Documents with extraction errors or warnings in the validation run Above 5% Investigate before any go decision
Cost per 1,000 pages in the validation run Above the agreed budget Revisit the extraction path or the scope
Gap between the two scorers Above 5 points Fix the scoring rule and score again
Hold-out score against the sample More than 10 points worse Stop and look for tuning to the sample
Projected cost against the first real bill More than 20% apart (team default) Find the missing cost line before the next run
Run time for one document Over 4 minutes (team default; the limit is 5 minutes) Split the file before the next run

7. AWS and open-source equivalents

The same checklist applies. Only the limits, the page counting and the prices change. Check the current limits and prices on the day.

Platform Equivalent Key differences
Amazon Textract Reads text, tables and forms from documents. See Amazon Textract limits for documents Check file size, page and format limits. Check how pages are counted for billing
Amazon Bedrock Data Automation A generative AI service that turns documents, images, video and audio into structured output. See Amazon Bedrock Data Automation Check current limits in the AWS docs, and check the current price before approval. Score figures and tables the same way
Open-source parsers Parsers that run on your own compute No per-page fee, but you pay for compute and people. Memory and file size limits are yours to set
Pricing Amazon Textract pricing Check the current price before approval

8. Validation record

Pending. Filled in by the dry-runner (see docs/validation-protocol.md).

Sandbox scenario: a second engineer runs this checklist on a fresh sample from a different corpus. Plant these faults in the sample and the sheets: a locked PDF, a file over the tier limit, a document type with only 5 scored pages, a hold-out that scores 20 points worse than the sample, a scorer gap of 8 points, and a budget set below the real cost.

Pass criteria: every planted fault is caught by an item or by the script. Every field of the results record is filled. The decision follows from the numbers. The tests in snippets/eval/ pass. The time taken is within the stated duration; if it is not, update the duration before sign-off.

Record

Item Detail
Playbook PB-RAG-10-03 at commit [hash]
Dry-run by [Name] (must not be author or reviewer)
Date YYYY-MM-DD
Sandbox [Resource group or tag, tier and region]
Scenario [Scenario followed]
Time taken [Hands-on time] hands-on, [waiting time] waiting
Result [PASS / PASS WITH NOTES / FAIL]
Notes 1. [Step number: defect or explanation; blocking/non-blocking; fixed/logged]
Signed off by [Reviewer name and date]

9. Sources

Checked on 5 November 2026.

  • PB-RAG-10-01: choose the extraction path before you validate it.
  • PB-RAG-10-02: build the extraction this checklist tests.
  • PB-RAG-10-04: what to do when extraction fails or costs too much after go-live.
  • PB-RAG-05-04: check the files before you test extraction.