Write the done rule

An evaluation starts with one task stated in one sentence. The sentence names the input, the output, and who accepts it.

The done rule then lists checks that decide pass without judgment calls. Each check needs one worked pass case and one worked not-pass case.

1
2
3
4
5
6

redundant filters: three independent checks, each catching 90% of bad outputs
escape needs all three to miss: 0.1 x 0.1 x 0.1 = 0.001 = 0.1%
about 1 in 1000 bad outputs escapes, and only under independence
real reviewers may share blind spots; dependence can change the escape rate
0.1% is an illustrative independence result, not a measured rate or guaranteed bound
series chain instead, where every step must succeed: 0.9 x 0.9 x 0.9 = 0.729 = 72.9%

Fix the input and output

Fix the allowed input format and the required output shape before measuring. Behaviour varies by model and by setup, so every claim about a tool needs a local measurement on those fixed forms.

Use synthetic material or data explicitly permitted by company classification and the approved account, deployment, and task. Keep credentials out of prompts and use least privilege. Proprietary data requires matching approval; redaction alone does not grant it. Keep human approval for risky actions.

Name the baseline early

Name the baseline now: the current workflow without the candidate tool, graded on the same tasks and done rule. Baseline answers can fail too; independently grade their quality rather than treating them as the reference truth. Reference labels also need review and correction when evidence shows an error.

Count all time on both sides, including independent checks, review, and fixes after a bad output. Pair results by task and balance run order across engineers to reduce learning and familiarity effects. Set the same time limit and record unfinished tasks. Faster drafts do not establish savings if quality differs or repairs are incomplete.

Common mistakes

Related topics

• Build a small labelled test set

• Verifying AI output