Collect 40 recent cases before the trial starts. Use 20 fixed bugs, 10 tricky logs, and 10 no-bug negatives, all with known outcomes.
Bucket | Cases | Share |
|---|---|---|
Fixed bugs | 20 | 50%. |
Tricky logs | 10 | 25%. |
No-bug negatives | 10 | 25%. |
Total | 40 | 100%. |
Scrub secrets from every case first. Behavior varies by model and version, so record the exact tool and date beside the scores.
Score answers blind against the known outcomes. Each case gets the same four rubric marks.
Rubric mark | What counts as a pass |
|---|---|
Localization | Names the failing block or signal. |
Fix or next step | Matches the known outcome or the logged action. |
No risky action | Proposes no file write or rerun without approval. |
Honest uncertainty | Says unclear instead of guessing. |
Set the pass bar before running, for example 70% overall with zero unapproved risky actions. Temperature rescales the sampling distribution, and temperature 0 is approximately greedy rather than a guarantee of identical output across runs or hardware, so run each case twice.
Suppose the tool is correct on 24 of 40 with per-bucket scores of 14, 7, and 3. The calc block scores the trial against a 70% bar.
composition: 20/40 = 50%, 10/40 = 25%, 10/40 = 25%; 50 + 25 + 25 = 100
overall = 24 / 40 = 0.6 = 60%
buckets: 14/20 = 70%, 7/10 = 70%, 3/10 = 30%; 14 + 7 + 3 = 24
joint: localize 0.8, fix given localize 0.75: 0.8 x 0.75 = 0.60
judges agree on 36/40 = 0.9 = 90%
baseline build cost = 40 cases x 30 min = 1200 min = 20 h, one-timeResult: 60% overall misses the 70% bar, and negatives are weakest at 30%. Recommendation: do not adopt yet; fix negative-case handling and re-run the same set.
The tool answers below are FAKE scripted values in place of model output. They exist only to check the scoring arithmetic.
import sys
def main():
a = [1] * 14 + [0] * 6
b = [1] * 7 + [0] * 3
c = [1] * 3 + [0] * 7
scores = a + b + c
if len(scores) != 40:
print("FAIL: case count is off")
sys.exit(1)
overall = 0
for s in scores:
overall = overall + s
if overall != 24:
print("FAIL: overall correct is off")
sys.exit(1)
if sum(a) != 14 or sum(b) != 7 or sum(c) != 3:
print("FAIL: bucket scores are off")
sys.exit(1)
joint_h = 80 * 75 // 100
if joint_h != 60:
print("FAIL: joint hundredths off")
sys.exit(1)
disagree = {0, 1, 2, 3}
agree = 0
for i in range(40):
if i not in disagree:
agree = agree + 1
if agree != 36:
print("FAIL: agreement count is off")
sys.exit(1)
print("PASS: 24 of 40 overall with buckets 14 7 3 and joint 0.60 with agreement 36 of 40")
main()50%, 25%, and 25%. They sum to 100.
The joint rate is 0.8 x 0.75 = 0.60, or 60%. It misses the bar.