What counts as resolved

Resolved means an independent check passes, not that the agent says it finished. A triage task is resolved when the failing test is identified and a rerun confirms the diagnosis.

Each task needs its pass condition written before the agent runs. Post-hoc grading moves the goalposts and inflates the rate.

A held-out set

A held-out set is a fixed task list never used for prompt or tool tuning. Tuning against it teaches the agent the test, not the job.

Record the prompt version, tool versions, and permission set with every run. Refresh the set when tasks leak into tuning material.

• Fixed tasks with written pass conditions, frozen for a review period.

• No tuning on it: prompts, examples, and tool retries are tuned elsewhere.

• Versioned alongside the agent, so old and new scores stay comparable.

Reading a rate

A rate without uncertainty is a single anecdote. The calc block below works an EXAMPLE set of 50 tasks with 40 resolved.

1
2
3
4
5
6
7

held-out set: n = 50 tasks, resolved r = 40
task success rate = r / n = 40 / 50 = 0.80
standard uncertainty = sqrt(p (1 - p) / n) = sqrt(0.80 x 0.20 / 50)
  = sqrt(0.0032) = about 0.057
about-95 interval = 0.80 +/- 1.96 x 0.057 = 0.80 +/- 0.11 = 0.69 to 0.91
chain check: per-step 0.95 over 10 steps
  = 0.95^10 = 59873693923837890625 / 10^20 = 0.59873693923837890625 = about 0.60

The program below recomputes the lesson numbers from EXAMPLE data. It uses plain lists, with no model involved.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

import sys

outcomes = [True] * 40 + [False] * 10
chain_num = 95 ** 10
n = len(outcomes)
resolved = sum(1 for o in outcomes if o)
rate = resolved / n

if n != 50:
    print("FAIL: expected 50 held-out tasks")
    sys.exit(1)
if resolved != 40:
    print("FAIL: expected 40 resolved tasks")
    sys.exit(1)
if abs(rate - 0.80) > 1e-12:
    print("FAIL: rate arithmetic off")
    sys.exit(1)
if chain_num != 59873693923837890625:
    print("FAIL: chain arithmetic off")
    sys.exit(1)
print("PASS: 40 of 50 held-out tasks resolved for rate 0.80 with 10-step chain 0.59873693923837890625")

Common mistakes

Related topics

• Cost and latency

• Evaluating AI tools