Triage record: ${INCIDENT_ID}
Copy this file for each incident. Fill in one row per failing question. Keep it with the incident.
Do not write keys, tokens, real user names or client names. Use IDs and ${PLACEHOLDERS}.
1. Incident
| Item |
Detail |
| Incident or ticket |
${INCIDENT_ID} |
| Reported by and how |
|
| Triage done by |
|
| Triage started |
YYYY-MM-DD HH:MM |
| Window of first failure |
From YYYY-MM-DD HH:MM to YYYY-MM-DD HH:MM |
| Index or alias queried |
${INDEX_OR_ALIAS} |
| API version used by the probe |
|
| Filter applied for the user |
${USER_FILTER} |
| Filter mode (if the application sets one) |
|
2. Changes in the window
List every release, re-index run, alias swap, model change, prompt change, data load or
permission change between the last good evaluation and the first failure.
| Change |
Date and time |
Made by |
Link or ID |
|
|
|
|
3. Cases
One row per failing question. Rank is the position of the expected document in the first N
results, or - if it is not there. Copy the probe output into the evidence column.
| Case |
Question |
Time |
Expected document id |
Keyword rank |
Vector rank |
Hybrid rank |
Hybrid, no filter rank |
Class |
Evidence |
| 1 |
|
|
|
|
|
|
|
|
|
Class is one of: retrieval miss, ranking issue, filter issue, generation fault.
Every case has exactly one class and one line of evidence.
4. Summary
| Class |
Cases |
Likely root cause |
Hand over to |
| Retrieval miss |
|
|
|
| Ranking issue |
|
|
|
| Filter issue |
|
|
|
| Generation fault |
|
|
|
5. Severity
| Item |
Detail |
| Golden set run |
Date and run ID |
| Retrieval metrics against the baseline |
recall@10 and nDCG@10, with the tolerance used |
| Generation metrics against the baseline |
Groundedness and relevance, with the tolerance used |
| Severity |
Isolated (a few cases) or systemic (the golden set drops beyond tolerance) |
6. Outcome
| Item |
Detail |
| Root cause |
|
| Playbook the incident owner follows |
|
| Incident owner |
|
| New golden questions to add |
Passed to PB-RAG-02-01 on YYYY-MM-DD |
| Triage closed |
YYYY-MM-DD HH:MM |