AGT-07 · Agent evaluation & trajectory scoring¶
Teams judge agents on the final answer only, so a run that called the wrong tool, retried five times or silently swallowed a tool error still passes. Without trajectory evidence and a regression gate, no client signs off autonomous actions and every prompt or model change is a gamble.
- Who reads it: QE engineers, GenAI engineers and agent leads who ship or change agents in client projects
- Outcome: Every agent change gated on task success, tool accuracy, step efficiency and cost per task
- Pack owner: Prasanna (catalog owner role: QE Lead)
Master catalog entry¶
- Catalog name: Agent Evaluation Kit (P1)
- Why it matters: Without trajectory evidence, no client will approve autonomous action
- What it removes: Measures the path, not just the answer — catches silent tool failures
- Stack: LangSmith, Azure AI Evaluation SDK, custom trajectory scorers
- Maturity target: Level 2 — task success, step efficiency, tool accuracy, cost per task
Playbooks¶
| ID | Story | Title | Kind | Complexity | Urgency | Status | Owner | Planned publish |
|---|---|---|---|---|---|---|---|---|
| PB-AGT-07-01 | AGT-07-S1 | Score agent trajectories with the Azure AI Evaluation SDK and custom scorers | How-to | Medium | 4 | draft | Devadharshini Raja | 2026-11-25 |
| PB-AGT-07-02 | AGT-07-S2 | Run the agent regression gate before a prompt, model or tool change | Runbook | Medium | 5 | draft | Sivaraj | 2026-11-05 |
| PB-AGT-07-03 | AGT-07-S3 | Choose agent evaluation metrics, thresholds and judge models | Decision guide | Medium | 3 | draft | Prasanna | 2026-11-27 |
| PB-AGT-07-04 | AGT-07-S4 | Online evaluation drift and judge-model upgrade checklist | Checklist | Low | 4 | draft | Devadharshini Raja | 2026-11-20 |