Week one of the regression: coverage climbs from nothing to 80%. Week two: 90%. Week three: 91.8%. Week four: 92.0%. The project manager asks for a date for 100%, and someone proposes doubling the regression farm. The curve says that would buy perhaps a tenth of a percent. What remains is not a quantity problem.
What does a closure curve's shape tell you, and what do you do at the plateau?
The obvious response to a plateau is more of the same: more seeds, more tests, a bigger farm. That works only for bins that are merely rare. For very rare bins the expected wait is beyond any budget (lesson 4.5), and for unreachable bins it is infinite. Running longer at a plateau mostly confirms the plateau.
If a bin is hit with probability p per draw and each test makes D draws, its chance of being hit after T tests is 1 − (1 − p)^(DT) (fact M3); the curve is the sum of these over all bins. Common bins produce the steep rise, rare bins the bend, and very rare and unreachable bins the plateau.
The model has 80 common, 12 rare, 4 very rare and 4 unreachable bins, and makes 1,000 draws per test (illustrative). It prints the expected coverage, checks it with a Monte Carlo run of individual draws, and splits the remaining holes by cause.
"""Closure curves: why coverage climbs fast, then stops.
A coverage model has 100 bins. Each random test makes 1,000 draws, and each draw hits a bin
with a fixed probability (illustrative):
80 common bins p = 1e-3 per draw (about one hit per test)
12 rare bins p = 2e-5 per draw (about one test in 50)
4 very rare bins p = 2e-8 per draw (about one test in 50,000)
4 unreachable p = 0 (the design or the stimulus can never produce them)
The expected coverage after T tests is the sum over bins of 1 - (1 - p)^(1000 T). A Monte Carlo
run checks the formula at 50 tests. Then the plateau is split by cause.
"""
import random
DRAWS = 1000
bins = [1e-3] * 80 + [2e-5] * 12 + [2e-8] * 4 + [0.0] * 4
Ts = [1, 10, 50, 100, 200, 500, 1000, 5000]
def expected(T):
return sum(1 - (1 - p) ** (DRAWS * T) for p in bins)
print("tests: " + " ".join(f"{t:>6d}" for t in Ts))
cov = [expected(t) for t in Ts]
print("coverage (%): " + " ".join(f"{c:6.1f}" for c in cov))
rng = random.Random(4)
runs, T = 100, 50
edges, acc = [], 0.0
for p in bins: # one draw lands in bin k if u falls in [edges[k-1], edges[k])
acc += p
edges.append(acc)
sim = 0.0
for _ in range(runs):
seen = set()
for _ in range(DRAWS * T):
u = rng.random()
if u < acc:
seen.add(next(k for k, e in enumerate(edges) if u < e))
sim += len(seen) / runs
print(f"Monte Carlo at {T} tests ({runs} runs of {DRAWS * T:,} draws): {sim:.1f}% (formula {expected(T):.1f}%)")
def left(T, p):
return sum((1 - p) ** (DRAWS * T) for q in bins if q == p)
for T in (200, 1000):
print(f"holes after {T} tests: common {left(T, 1e-3):.1f}, rare {left(T, 2e-5):.1f}, "
f"very rare {left(T, 2e-8):.1f}, unreachable 4")
print(f"tests for a 50% chance at one very rare bin: {round(0.693 / (DRAWS * 2e-8)):,}")
assert abs(sim - expected(50)) < 1.0
assert 91.0 < cov[Ts.index(1000)] < 93.0 and cov[-1] < 93.0
print("PASS: coverage reaches 92% by about 200 tests and stays there; the last 8% is 4 very rare bins and 4 unreachable ones")It prints:
tests: 1 10 50 100 200 500 1000 5000
coverage (%): 50.8 82.2 87.6 90.4 91.8 92.0 92.1 92.4
Monte Carlo at 50 tests (100 runs of 50,000 draws): 87.6% (formula 87.6%)
holes after 200 tests: common 0.0, rare 0.2, very rare 4.0, unreachable 4
holes after 1000 tests: common 0.0, rare 0.0, very rare 3.9, unreachable 4
tests for a 50% chance at one very rare bin: 34,650
PASS: coverage reaches 92% by about 200 tests and stays there; the last 8% is 4 very rare bins and 4 unreachable onesHow to read it: by 200 tests every common bin and almost every rare bin is hit; coverage is 91.8%. From 200 to 5,000 tests, 25 times the work, it gains 0.6 points. The remaining 8% is four very rare bins (a 50% chance of hitting a given one of them needs about 35,000 tests) and four that no amount of testing can hit. The farm should not be doubled.
Cause of the hole | How to tell | Action |
|---|---|---|
Rare: stimulus reaches it slowly | Hit in some seeds or after long runs | Run longer, or bias the distributions (lesson 4.4) |
Very rare: needs a specific combination | Never hit; estimable probability is tiny | Directed test, coverage-driven stimulus (lesson 6.7), or formal |
Never generated: constraints exclude it | Probability zero under the current constraints | Fix the constraints or add a scenario (lesson 4.3) |
Unreachable in the design | No legal input can produce it | Prove it unreachable, then exclude it with a reason (lesson 6.6) |
Wrong bin: the model is wrong | The bin describes something impossible or irrelevant | Correct the model |
Practice | Purpose |
|---|---|
Coverage plotted per week, merged across the regression | The bend shows when random stimulus stops paying |
Holes triaged by cause at the plateau | Each gets an owner and an action, not more seeds |
Closure forecasts from the curve | Realistic dates, and early warning of a plateau (Part 9) |
The opening story is the model's curve. The illustrative team that doubled their farm at 92% gained a few tenths of a point in a month. Triage of the 8% found two constraints that excluded legal cases, three bins needing directed tests, and four bins describing illegal states that the model had marked as ordinary. A week of analysis closed what a month of extra simulation had not.
Everything reachable at ordinary probabilities is already hit; the remaining bins are either so rare that thousands more tests barely change their odds, or unreachable.
Very rare: write a directed test or steer stimulus toward it. Unreachable: prove it with formal and exclude it with a documented reason. (Also possible: the constraints never generate it, so fix them.)
A closure curve is the sum of its bins' hit probabilities: steep rise, bend, plateau.
At the plateau, more simulation buys almost nothing.
Classify holes by cause (rare, very rare, never generated, unreachable, wrong) and act on each.