Teams choose tools from demos and anecdotes, which hide the miss rate. A small labelled test set plus a fair baseline supports task-specific comparison of pass rate, time, and cost per accepted result. The fixed outcomes, times, and costs in these lessons are synthetic fixtures for demonstrating arithmetic, not measurements of any tool or engineer.
Claim | Measurement that tests it |
|---|---|
The tool does the task well | Pass rate on a frozen labelled set with a written done rule. |
The tool saves time | Paired same-task comparison, balanced run order, same done rule, all review and fix time counted. |
The tool is worth its spend | Measured tool spend per accepted result, alongside human effort and quality. |
Lesson | What it covers |
|---|---|
Scope, inputs, outputs, and the written done rule. | |
Sampling real work, labelling answers, and freezing the set. | |
A fixed harness, pass rate with its range, and fair time comparison. | |
Cost per accepted result, side-by-side options, and the re-check. |
Otherwise the rule bends to fit the outputs already seen. A frozen rule keeps every option on the same test.