Two weeks before tape-out, the chair of the review board asks the DV lead one question: "How many bugs are still in there?" The bug tracker shows 36 found. The coverage report shows 97%. Neither is an answer. The number of bugs not found is exactly what no report shows, and yet the whole decision depends on it.

Can you estimate the number of bugs you have not found, and how far should you trust the estimate?

The obvious answer, and why it breaks

The obvious answers are "we can't know" or "the bug rate has dropped, so few are left". The first is true but useless to someone who must decide. The second confuses the rate at which current methods find bugs with the number that exist: a regression that cannot reach a corner stops finding bugs there whether or not any remain. An estimate needs information about what the methods miss, and the only place that information appears is in how much two methods overlap.

The idea: overlap measures what is missed

Ecologists estimate fish in a lake by catching some, tagging them, releasing them, and catching again: if a quarter of the second catch is tagged, the first catch was about a quarter of the population. The same reasoning applies to two independent verification methods, A and B (fact M4):

• n1 bugs found by A, n2 by B, m found by both.

• If A finds a fraction f of all bugs, it should also find about a fraction f of B's bugs: m/n2 ≈ n1/N, which rearranges to N ≈ n1·n2/m.

• Remaining ≈ N̂ − (n1 + n2 − m).

• The key assumption: the two methods find bugs independently. A bug's chance of being found by A says nothing about B.

Build it: when the estimate works and when it fails

The model first works the example above, then simulates 2,000 projects, each with 60 real bugs. In one case the two methods find bugs independently; in the other, both methods are good at the same easy bugs and bad at the same hard ones.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

"""Estimating remaining bugs with capture-recapture.

Part 1, a worked example (illustrative numbers): method A (constrained-random simulation)
found 30 bugs, method B (formal) found 24, and 18 were found by both. Chapman's estimate of
the total is (n1 + 1)(n2 + 1)/(m + 1) - 1, and remaining = estimate - bugs found so far.

Part 2 tests the estimator on a simulated design with 60 bugs, over 2,000 projects:
  independent: each method finds each bug with its own chance (uniform 0.2 to 0.8), unrelated
               to the other method's chance; the estimator's assumption holds
  correlated:  both methods find the same "easy" bugs: one easiness e per bug (e = u^2, so
               many bugs are hard), and both methods find the bug with chance e
For each case it reports the bugs actually found, the mean estimate and the 10th-90th
percentile range. When the methods share blind spots, the estimate is far too low.
"""
import random

def chapman(n1, n2, m):
    return (n1 + 1) * (n2 + 1) / (m + 1) - 1

n1, n2, m = 30, 24, 18
est = chapman(n1, n2, m)
found = n1 + n2 - m
print(f"worked example: A found {n1}, B found {n2}, both {m}: estimate {est:.1f} bugs, {found} found, about {est - found:.0f} left")

TRUE, RUNS = 60, 2000
rng = random.Random(11)

def project(correlated):
    a_hits = b_hits = both = 0
    for _ in range(TRUE):
        if correlated:
            e = rng.random() ** 2
            pa = pb = e
        else:
            pa, pb = rng.uniform(0.2, 0.8), rng.uniform(0.2, 0.8)
        a, b = rng.random() < pa, rng.random() < pb
        a_hits += a; b_hits += b; both += a and b
    return a_hits + b_hits - both, chapman(a_hits, b_hits, both)

for case in ("independent", "correlated"):
    res = [project(case == "correlated") for _ in range(RUNS)]
    founds = sorted(r[0] for r in res)
    ests = sorted(r[1] for r in res)
    mean_f = sum(founds) / RUNS
    mean_e = sum(ests) / RUNS
    lo, hi = ests[RUNS // 10], ests[9 * RUNS // 10]
    print(f"{case:12s} true {TRUE}: found {mean_f:.1f} on average, estimate {mean_e:.1f} (10-90%: {lo:.1f} to {hi:.1f})")
    if case == "independent":
        ind = mean_e
    else:
        cor, cor_found = mean_e, mean_f

assert abs(est - 39.8) < 0.05
assert abs(ind - TRUE) < 3 and cor < TRUE - 15
print("PASS: with independent methods the estimate centres on the true 60; when both methods miss the same hard bugs it falls far short")

It prints:

worked example: A found 30, B found 24, both 18: estimate 39.8 bugs, 36 found, about 4 left
independent  true 60: found 45.0 on average, estimate 60.0 (10-90%: 50.3 to 70.5)
correlated   true 60: found 28.0 on average, estimate 33.3 (10-90%: 26.6 to 40.5)
PASS: with independent methods the estimate centres on the true 60; when both methods miss the same hard bugs it falls far short

How to read it: in the worked example the estimate is 39.8, so about 4 of the 40 bugs remain. With independent methods, the simulated projects found 45 of 60 bugs on average, and the estimate centres exactly on the true 60, with a 10–90% range of about 50 to 70: useful, but not precise. With correlated methods, the projects found only 28 bugs, and the estimate says 33: it reports about 5 bugs left when 32 remain. Shared blind spots make the overlap look large, and a large overlap reads as "nearly done".

Making the estimate trustworthy

In practice the estimate usually comes out low: most pairs of methods share some blind spots, which inflates the overlap. Methods with complementary strengths can push it high instead. Use it alongside other evidence, such as discovery curves (lesson 12.2) and coverage holes, never alone.

How real teams use it

What breaks

A DV lead (illustrative) estimated remaining bugs from two random regressions run by two engineers with different seeds, and got "fewer than three". Both regressions used the same stimulus generator, constraints and checkers, so they found the same bugs and missed the same ones; the overlap was nearly total. A formal pass on the block's control logic found nine more bugs the following week. The estimate had measured how similar the two regressions were, not how many bugs remained.

Key takeaways

• The overlap between two independent methods estimates the total: N ≈ n1·n2/m.

• Correlated methods share blind spots and make the estimate far too low.

• Report the estimate with its range and assumptions, alongside discovery curves and coverage, never alone.

Related topics

• Part 12 overview: Bug Economics