Copy this sheet once for every corpus. Each row is one item of PB-RAG-10-03, section 4. The playbook says how to check each item. Write the evidence (a file, a link or a number) in the last column.
| Field |
Value |
| Corpus |
${CORPUS_ID} |
| Filled in by |
|
| Date started |
|
| Results record |
extraction-results.yaml |
Part A. Agree the rules
| Item |
Check |
Done |
Who |
Evidence |
| A1 |
Goal written in one sentence and agreed |
|
|
|
| A2 |
Written permission to process the sample |
|
|
|
| A3 |
Sensitive data checked; masking or a locked workspace agreed |
|
|
|
| A4 |
People named, as item A4 asks |
|
|
|
| A5 |
Path, search tier, region, API version and preview use decided |
|
|
|
| A6 |
Billable Foundry resource in a supported region |
|
|
|
| A7 |
Budget cap and approver set |
|
|
|
| A8 |
Deletion date agreed |
|
|
|
Part B. Choose the sample
| Item |
Check |
Done |
Who |
Evidence |
| B1 |
Document types, scan sources, languages and layouts listed, with shares |
|
|
|
| B2 |
At least 50 documents, and the minimum scored pages for each type |
|
|
|
| B3 |
Worst cases added, each named worst-... |
|
|
|
| B4 |
Whole tables taken, including tables across pages |
|
|
|
| B5 |
Client-chosen and random parts recorded |
|
|
|
| B6 |
Copy made and hashes match |
|
|
|
| B7 |
Hold-out set aside and not scored |
|
|
|
| B8 |
Sample fits the time for hand scoring |
|
|
|
Part C. Check the limits
| Item |
Check |
Done |
Who |
Evidence |
| C1 |
Intake script from PB-RAG-05-04 run |
|
|
|
| C2 |
Every file checked against the smallest limit in the chain |
|
|
|
| C3 |
Locked files unlocked or rejected |
|
|
|
| C4 |
Unsupported file types converted or excluded |
|
|
|
| C5 |
Image sizes and HEIF files checked |
|
|
|
| C6 |
Files that may run past five minutes listed for splitting |
|
|
|
| C7 |
Billable Foundry resource attached for every real run |
|
|
|
| C8 |
Longest files under the extracted-character limit |
|
|
|
| C9 |
Run plan with the expected hours |
|
|
|
| Item |
Check |
Done |
Who |
Evidence |
| D1 |
Clean validation index used, name ends in -validation |
|
|
|
| D2 |
Start, end, documents and pages billed recorded |
|
|
|
| D3 |
Errors, warnings and time-outs listed with file names |
|
|
|
| D4 |
429 replies and retries recorded |
|
|
|
| D5 |
Empty or very short outputs found |
|
|
|
| D6 |
Outputs cut at the character limit found |
|
|
|
| D7 |
Second run compared |
|
|
|
| D8 |
Raw output of every document kept |
|
|
|
| D9 |
Preview features and API version recorded |
|
|
|
| Item |
Check |
Done |
Who |
Evidence |
| E1 |
Ground truth written before reading the output |
|
|
|
| E2 |
Pages scored for each type and worst case |
|
|
|
| E3 |
Table fidelity scored cell by cell |
|
|
|
| E4 |
Table structure cases marked pass or fail |
|
|
|
| E5 |
Number accuracy counted on its own |
|
|
|
| E6 |
Figure coverage scored |
|
|
|
| E7 |
Figure descriptions (preview) spot-checked |
|
|
|
| E8 |
Scans scored by source; handwriting languages checked |
|
|
|
| E9 |
Reading order checked |
|
|
|
| E10 |
Other structure checked |
|
|
|
| E11 |
Every language checked |
|
|
|
| E12 |
Office files checked |
|
|
|
| E13 |
10% of pages re-scored by a second person |
|
|
|
Part F. Score the answers
| Item |
Check |
Done |
Who |
Evidence |
| F1 |
Questions written with the client: count and mix |
|
|
|
| F2 |
Hard questions added, including not_in_docs |
|
|
|
| F3 |
Answer key dated before the run |
|
|
|
| F4 |
Questions run; answers, sources and retrieved text saved |
|
|
|
| F5 |
Every answer scored |
|
|
|
| F6 |
Cause found for every wrong answer |
|
|
|
| F7 |
Scored again after any change of path or settings |
|
|
|
Part G. Cost, time and decision
| Item |
Check |
Done |
Who |
Evidence |
| G1 |
Cost for 1,000 pages worked out from the bill |
|
|
|
| G2 |
Every cost line counted |
|
|
|
| G3 |
Full-corpus cost projected |
|
|
|
| G4 |
Full-corpus time projected |
|
|
|
| G5 |
Projection compared with the budget |
|
|
|
| G6 |
Decision from the script or the decision table |
|
|
|
| G7 |
Conditions written in plain words |
|
|
|
| G8 |
Client acknowledgement on file |
|
|
|
| G9 |
Re-validation triggers recorded |
|
|
|
| G10 |
Sample and validation index deleted or locked |
|
|
|