Sample tasks from the real queue, recent weeks first. Stratify by kind so rare but costly cases keep their share of the set.
Twenty tasks beat two hundred anecdotes when the twenty are sampled and frozen. The example set below uses 20 tasks across three kinds.
Kind | Tasks | Share of 20 |
|---|---|---|
Summarize a tool log | 8 | 40% |
Draft a test plan section | 6 | 30% |
Explain a failure from a trace | 6 | 30% |
tasks = 8 + 6 + 6 = 20
shares: 8 / 20 = 40%, 6 / 20 = 30%, 6 / 20 = 30%
check: 40% + 30% + 30% = 100%Each task gets a reference answer and its pass checks before any tool runs. The label states the facts the output must contain and the faults it must avoid.
Two engineers label the first five tasks independently, then reconcile against source evidence. This can expose rubric problems early but does not guarantee that the remaining labels are correct. Review disputed labels independently and record any correction across all compared options.
Label part | What it records |
|---|---|
Reference points | The faults, lines, or facts a passing output must include. |
Known traps | Plausible but wrong answers the task tends to invite. |
Grader script | Mechanical checks first, human judgment only for the rest. |
Freeze task inputs, labels, and tool settings together with a date. Later runs reuse the same frozen set, so version changes show up as rate changes.
Keep a second small holdout sample untouched during tuning. A set that is tuned against stops measuring generalization.
The sample is conditioned on success, so the rate reads high. Sample from the queue before seeing outputs.
Tuning to the main set overfits it. The holdout still measures how the setup handles unseen tasks of the same kinds.