Week 1 of a block's random regression (illustrative) finds ten bugs. Week 9 finds less than two. The team runs more seeds, then more again. By week 16 random stimulus finds a bug every three weeks or so, while the plan still lists features random never reached. DCRouter's bug-catch matrices in Parts 4 to 6 showed the same thing in miniature: exactly which bugs random stimulus could not find. Running random longer was the obvious move. It was the wrong one.
When should you stop running random, and what should you do instead?
The obvious response to a slowing technique is more of it: more seeds, a bigger farm. Each technique has a pool of bugs it can reach; as the pool empties, each new bug costs more. Bugs outside the pool (unreachable states, unchecked rules, deep counters) are never found by that technique, however long it runs. The curve's flattening is information about the technique, not only about the design.
Cost per bug = effort spent ÷ bugs found, per week or per batch of runs.
When it climbs past a threshold, the technique's pool is nearly empty for this design.
Switch to a technique with a different pool: coverage-driven stimulus, directed tests from holes, formal on hard rules, emulation for long runs.
Keep the old technique running in regression at low cost: it still guards against regressions.
An illustrative design with 50 bugs random stimulus can reach and 15 it practically cannot. Plan A runs random for 16 weeks; plan B switches to targeted techniques when random finds fewer than 1.5 bugs in a week.
"""Diminishing returns: when to stop running random and change technique.
A design holds 50 bugs that random stimulus can reach and 15 that it practically cannot (deep
states, rare combinations, unchecked rules; illustrative). One engineer-week of random
regression finds 20% of the remaining random-reachable bugs. One engineer-week of a targeted
technique (directed tests from coverage holes, formal on the hard rules) finds 30% of the
remaining hard bugs and 5% of the remaining random-reachable ones.
plan A: random for all 16 weeks
plan B: random until it finds fewer than 1.5 bugs in a week, then the targeted technique
The model prints bugs found per week, engineer-weeks per bug, and totals (expected values).
"""
EASY, HARD, WEEKS = 50.0, 15.0, 16
def run(switch):
easy, hard, found, rows, mode = EASY, HARD, 0.0, [], "random"
for w in range(1, WEEKS + 1):
used = mode # the technique run this week
if mode == "random":
got = 0.20 * easy
easy -= got
if switch and got < 1.5:
mode = "targeted"
else:
g1, g2 = 0.05 * easy, 0.30 * hard
easy, hard, got = easy - g1, hard - g2, g1 + g2
found += got
rows.append((w, used, got, found))
return rows
a, b = run(False), run(True)
print("week plan A: found wk/bug cum | plan B: found wk/bug cum technique")
for (w, _, ga, ca), (_, mb, gb, cb) in zip(a, b):
print(f"{w:4d} {ga:13.1f} {1 / ga:6.1f} {ca:5.1f} | {gb:13.1f} {1 / gb:6.1f} {cb:5.1f} {mb}")
print(f"after {WEEKS} weeks: plan A found {a[-1][3]:.1f} of {EASY + HARD:.0f}, plan B found {b[-1][3]:.1f}")
assert b[-1][3] > a[-1][3] + 8
print("PASS: random's cost per bug climbs to almost three engineer-weeks per bug; switching technique when it slows finds the bugs random cannot reach")It prints (expected values):
week plan A: found wk/bug cum | plan B: found wk/bug cum technique
1 10.0 0.1 10.0 | 10.0 0.1 10.0 random
2 8.0 0.1 18.0 | 8.0 0.1 18.0 random
3 6.4 0.2 24.4 | 6.4 0.2 24.4 random
4 5.1 0.2 29.5 | 5.1 0.2 29.5 random
5 4.1 0.2 33.6 | 4.1 0.2 33.6 random
6 3.3 0.3 36.9 | 3.3 0.3 36.9 random
7 2.6 0.4 39.5 | 2.6 0.4 39.5 random
8 2.1 0.5 41.6 | 2.1 0.5 41.6 random
9 1.7 0.6 43.3 | 1.7 0.6 43.3 random
10 1.3 0.7 44.6 | 1.3 0.7 44.6 random
11 1.1 0.9 45.7 | 4.8 0.2 49.4 targeted
12 0.9 1.2 46.6 | 3.4 0.3 52.8 targeted
13 0.7 1.5 47.3 | 2.4 0.4 55.3 targeted
14 0.5 1.8 47.8 | 1.8 0.6 57.0 targeted
15 0.4 2.3 48.2 | 1.3 0.8 58.3 targeted
16 0.4 2.8 48.6 | 1.0 1.0 59.3 targeted
after 16 weeks: plan A found 48.6 of 65, plan B found 59.3
PASS: random's cost per bug climbs to almost three engineer-weeks per bug; switching technique when it slows finds the bugs random cannot reachHow to read it: random's cost per bug rises from 0.1 engineer-weeks in week 1 to 2.8 in week 16. Plan B switches after week 10, when random has found 44.6 bugs, and finds 14.7 more in six weeks, most of them from the pool random could not reach. Plan A spends the same six weeks finding 4.0. Same effort, different technique, nearly four times the bugs.
Technique added | What it found that the previous ones could not |
|---|---|
Error injection (lesson 4.7) | Error-path bugs: BUG_13, 14, 15, 19 |
Random timing (lesson 4.8) | Low-power timing bugs: BUG_20, 22 |
Arbitration checkers with hot spots (lesson 5.10) | Reached but unseen: BUG_5, 6, 7 |
Coverage-driven closing phase (lesson 6.9) | Never reached: BUG_16, 23 (and 15, 17, 18, 19 in one bench) |
Formal (lesson 7.9) | Proofs for every input; deep FIFO and credit bugs without stimulus |
Each column of the course's bug-catch matrix is a technique with its own pool. The course itself followed plan B.
Practice | Purpose |
|---|---|
Bugs found per technique per week tracked | Exhaustion detected early |
Planned technique transitions (random to coverage-driven to formal to emulation) | Effort moved to the next pool on time |
Low-cost regression of exhausted techniques | Protection kept, effort freed |
A team (illustrative) quadrupled its random regression over the last two months before tape-out, because the bug curve was falling and more seeds seemed the safe response. It found three bugs. A coverage review at the start of those two months would have shown a dozen holes in configuration and error modes that no seed could reach; two of them held the bugs found later in silicon.
They sit outside random stimulus's practical pool (unreachable states, unchecked rules); more seeds sample the same pool.
Adding the arbitration checker with hot-spot traffic (lesson 5.10): BUG_5, 6 and 7 were reached by random traffic all along.
Each technique finds its easy bugs first; cost per bug keeps rising.
A rising cost per bug signals an exhausted technique: switch to one with a different pool.
Keep exhausted techniques running cheaply as regression.