Online evaluation drift and judge-model upgrade checklist¶
Draft outline: not validated
Drafted with AI from official sources on 16 Oct 2026. Do not use on a live system until it is written, dry-run by someone other than the author and signed off (planned 2026-11-20). Author: Devadharshini Raja · dry-run: Yasar · sign-off: Prasanna. Work items: AGT-07-T13, AGT-07-T14, AGT-07-T15, AGT-07-T16.
1. Summary¶
Monthly for every production agent, and whenever the judge model is retired or upgraded. Use it when online scores move without a release.
Risk: Medium (silent drift or a recalibrated judge hides real quality loss). Typical duration: 2-3 hours per agent.
2. When to use and when not to¶
Use it when: Monthly for every production agent, and whenever the judge model is retired or upgraded. Use it when online scores move without a release.
Do not use it when: to be written by the author.
3. What you must know first¶
- Online scores move for three reasons: user mix, upstream data or tools, and judge changes
- A judge upgrade must be recalibrated before its scores are compared with old ones
- Trace sampling settings change what you measure; the Azure Monitor distro samples at 5 traces per second by default
- Trace data can contain personal data; apply the same access controls as production logs
4. Checklist¶
- Confirm traces and online evaluation results arrived for every day of the period.
- Compare the four Level 2 metrics with the last release gate results.
- Break down any drop by task type, tool and tenant.
- Check for tool schema, data source or model version changes in the period.
- If the judge model changed, score the calibration set with old and new judges and record the offset.
- Re-baseline only after the agent lead accepts the offset in writing.
- Raise a defect for any real drop and link example traces.
- Record the review outcome in the agent's run log.
Draft snippet: Online evaluation and judge calibration config
agent: invoice-triage-agent
online_evaluation:
sample_rate: 0.05 # share of production runs scored by the judge
metrics: [task_success, tool_accuracy, step_efficiency, cost_per_task_usd]
schedule: daily
judge:
deployment: ${JUDGE_DEPLOYMENT}
version_pinned: true
calibration:
dataset: eval/calibration_labelled.jsonl
min_cases: 20
rerun_on: [judge_upgrade, monthly_review]
alerting:
task_success_drop: 0.03
owner: qe-lead
Snippets to add:
snippets/eval/online-eval.yaml: Online evaluation sampling and calibration config (yaml)snippets/docs/drift-review.md: Monthly drift review record (markdown)
5. Known issues and fixes¶
| Symptom | Likely cause | Fix |
|---|---|---|
| Online scores drop with no release | Upstream tool or data change | Check tool error rates and data source change logs for the same period |
| Scores jump after a judge upgrade | New judge is more lenient or stricter | Score the calibration set with both judges and apply the offset before comparing |
| Gaps in daily metrics | Sampling or exporter misconfiguration | Check sampling_ratio or traces_per_second and exporter errors in the app logs |
| Traces show user personal data | Content recording left on | Disable or redact content capture and purge per retention policy |
6. Monitoring and alerts¶
| Signal | Alert when | Action |
|---|---|---|
| Online task success (7-day average) | Drops more than the release tolerance below baseline | Run this checklist within 2 working days |
| Trace ingestion for the agent | No traces for 30 minutes during business hours | Check exporter configuration and the Application Insights connection |
7. AWS and open-source equivalents¶
AgentCore online evaluations score sampled production traces on AWS. Open-source stacks combine OpenTelemetry traces with scheduled DeepEval or LangSmith runs on sampled data.
8. Validation record¶
Pending. Planned dry-run scenario: A second engineer runs the checklist on the sandbox agent after swapping the judge deployment. Pass: the offset is measured on the calibration set, the drift record is complete and no baseline is changed without sign-off.