The harness below is a fixed-fixture arithmetic example. It makes no model calls, evaluates no actual outputs, and contains no observed timing measurements. Its expected values describe 20 invented task outcomes, invented minutes on two paths, and fictional tool spend; no execution of the example is claimed here.
The pass list holds 14 ones and 6 zeros. Each time-list position represents the same hypothetical task on both paths, including review and repair. Independent grading and real timing records would be needed to replace these fixtures with evidence about a tool.
import math
import sys
# synthetic fixture labels, not results from evaluating a real tool
# 1 represents meeting the done rule; 0 represents not meeting it
outcomes = [1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0]
# invented paired minutes per task, including review and fixes
base_min = [45, 30, 60, 25, 40, 35, 50, 30, 45, 25, 40, 35, 55, 30, 45, 25, 50, 35, 40, 30]
aid_min = [20, 15, 35, 20, 25, 20, 30, 15, 25, 20, 25, 15, 30, 25, 20, 15, 35, 20, 25, 20]
cost_units = 70.0 # stated example parameter for the assisted run
n = len(outcomes)
passes = sum(outcomes)
rate = passes / n
z = 1.96 # nominal two-sided 95% Wilson score interval
denom = 1.0 + z * z / n
center = (rate + z * z / (2.0 * n)) / denom
half_width = z * math.sqrt(rate * (1.0 - rate) / n + z * z / (4.0 * n * n)) / denom
lo = center - half_width
hi = center + half_width
base_total = sum(base_min)
aid_total = sum(aid_min)
saved = base_total - aid_total
saved_pct = 100.0 * saved / base_total
per_task = saved / n
cost_per = cost_units / passes
def bad(note):
print("FAIL: " + note)
sys.exit(1)
if n != 20:
bad("want 20 tasks")
if passes != 14:
bad("want 14 passes")
if abs(rate * 100.0 - 70.0) > 1e-9:
bad("rate sits off 70 pct")
if not lo < rate < hi:
bad("range must bracket the rate")
if not 48.0 < lo * 100.0 < 48.2:
bad("lower bound sits off its band")
if not 85.4 < hi * 100.0 < 85.6:
bad("upper bound sits off its band")
if base_total != 770:
bad("baseline total sits off 770")
if aid_total != 455:
bad("assisted total sits off 455")
if saved != 315:
bad("kept minutes sit off 315")
if abs(saved_pct - 31500.0 / 770) > 1e-9:
bad("kept share sits off its quotient")
if abs(per_task - 15.75) > 1e-9:
bad("per-task kept sits off 15.75")
if abs(cost_per - 5.0) > 1e-9:
bad("per-accepted spend sits off 5.0")
print("PASS: fixture arithmetic only; no AI performance measured; 14/20 = 70.0 pct; Wilson interval 48.1-85.5 pct; hypothetical saving 315/770 min; 5.0 units per accepted result")The fixture rate is 14 of 20, or 70.0%. On this finite list the count is exact. An interval for a broader task population is meaningful only with an appropriate sampling model; the following Wilson calculation illustrates independent Bernoulli trials with a common pass probability, not an empirical finding about any tool.
synthetic passes = 14, trials = 20
pass rate = 14 / 20 = 0.7 = 70.0%
z = 1.96; z squared = 3.8416
denominator = 1 + 3.8416 / 20 = 1.19208
center = (0.7 + 3.8416 / 40) / 1.19208 = about 0.667774
half-width = 1.96 x sqrt(0.7 x 0.3 / 20 + 3.8416 / 1600) / 1.19208
= about 0.186751
nominal 95% Wilson interval = about 0.481023 to 0.854525
rounded: 48.1% to 85.5%; it is not centered on the observed 70%Outcome | Count | Share |
|---|---|---|
Pass | 14 | 70% |
Not pass | 6 | 30% |
The nominal confidence level describes the interval procedure under its sampling assumptions, not a guarantee about the next task. Related tasks, tuning on the set, and a stratified task mix can require a different analysis. For the formula, see the NIST/SEMATECH e-Handbook, section 7.2.4.1, Wilson method.
Comparing two tools on the same tasks is paired: record which tasks both pass, neither passes, and only one passes. The discordant pairs matter for a comparison of pass rates. Separate interval overlap neither proves that a gap is noise nor settles a paired difference. Use an analysis appropriate to those paired outcomes and the sampling design; a small set may leave substantial uncertainty.
The invented minute lists pair the same 20 tasks under the same done rule. Both include review and repair. Their totals illustrate arithmetic only: baseline quality has not been supplied, and the six assisted failures are still failures. For an actual study, grade both paths independently, record timeouts and unfinished repairs, balance run order, and inspect per-task time differences alongside quality. Do not claim equivalent-quality productivity gains from these lists.
synthetic fixture arithmetic, not measured productivity:
baseline total = 770 min for 20 tasks
assisted total = 455 min for 20 tasks
kept = 770 - 455 = 315 min
kept share = 315 / 770 = 0.40909... = about 40.9%
per task = 315 / 20 = 15.75 min kept per task on averageA has two more observed passes, but the marginal totals alone do not establish a reliable advantage. Inspect discordant task pairs, uncertainty under the sampling design, severity, and review effort. Interval overlap does not prove noise. An unweighted binary rate on 20 tasks moves in 5-point steps, so 62% is impossible; ask whether the denominator, weighting, or metric differs.
Into the assisted minute list, all of it. Time counts every fix minute under the same done rule, or the comparison favours the tool.