An agent that triages regressions or fixes lint is a production tool. Its owners measure it, watch its cost, and know when to stop it.
This topic sets up that loop: task success on a held-out set, cost and latency per task, regression suites for agent changes, and drift, incident, and shutoff rules.
Signal | Meaning | Where it is set |
|---|---|---|
Task success rate | Share of held-out tasks the agent resolves, with uncertainty stated. | Lesson 1 |
Cost per resolved task | Total billed spend, including failed and invalid attempts, divided by resolved tasks, in EXAMPLE units. | Lesson 2 |
Regression suite | Pinned cases re-run on every prompt, tool, or permission change. | Lesson 3 |
Drift and shutoff gates | Numeric limits that trigger review or switch the agent off. | Lesson 4 |
Lesson | What it covers |
|---|---|
What counts as resolved, held-out sets, and reading a rate with uncertainty. | |
Cost per attempt, cost per resolved task, and latency arithmetic with EXAMPLE units. | |
Pinned suites that catch breakage when prompts, tools, or permissions change. | |
Drift checks, incident reviews, and numeric rules for switching an agent off. |
0.80. The set stays unused during prompt and tool tuning, so tuning cannot inflate the score.