A team gives each of eight blocks the same ten weeks of verification, which seems fair. The reused UART and GPIO blocks, unchanged for three projects, find nothing in their ten weeks. The new cache controller, three times larger and far more complex, runs out of time with coverage still climbing and bugs still arriving every day. The effort was equal; the risk was not.

Where do bugs tend to hide, and how should that change where verification effort goes?

The obvious answer, and why it breaks

The obvious plan spreads effort evenly across blocks, or in proportion to their size. Both ignore what actually predicts bugs. A large block that was reused unchanged from a proven design may hold almost none; a small block that is new, intricate or heavily modified may hold many. Effort spread evenly is wasted on low-risk blocks and starved where the bugs are.

The idea: what predicts bugs

Build it: the same budget, two allocations

An illustrative design of eight blocks. Expected bugs grow with size, novelty and complexity; detection follows diminishing returns. The same 80 engineer-weeks are spread evenly, or in proportion to expected bugs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

"""Where bugs hide: spending test effort where bugs are likely.

A design of eight blocks (illustrative numbers). Each block's expected bugs grow with its
size, its novelty (reused 0.2, modified 0.6, new 1.0) and its complexity (1 to 3):
  expected bugs = 0.5 * size_klines * novelty * complexity
A test budget of 80 engineer-weeks is spread either uniformly (10 per block) or in
proportion to expected bugs. Each block's detection follows diminishing returns:
  fraction found = 1 - exp(-weeks / (2 * expected bugs ** 0.5))
The model prints expected bugs per block and the bugs found by each allocation.
"""
import math

blocks = [  # name, size in thousands of lines, novelty, complexity
    ("uart (reused)",        2, 0.2, 1),
    ("gpio (reused)",        1, 0.2, 1),
    ("dma (modified)",       8, 0.6, 2),
    ("cache ctrl (new)",    12, 1.0, 3),
    ("interconnect (new)",  10, 1.0, 3),
    ("timer (reused)",       1, 0.2, 1),
    ("crypto (modified)",    6, 0.6, 2),
    ("power ctrl (new)",     4, 1.0, 2),
]
expected = [0.5 * s * n * c for _, s, n, c in blocks]
total_exp = sum(expected)
BUDGET = 80

def found(weeks, e):
    return e * (1 - math.exp(-weeks / (2 * math.sqrt(e)))) if e > 0 else 0

uniform = [BUDGET / len(blocks)] * len(blocks)
risk = [BUDGET * e / total_exp for e in expected]
fu = sum(found(w, e) for w, e in zip(uniform, expected))
fr = sum(found(w, e) for w, e in zip(risk, expected))
print(f"{'block':20s} {'expected bugs':>13s} {'uniform weeks':>13s} {'risk weeks':>11s}")
for (name, *_), e, u, r in zip(blocks, expected, uniform, risk):
    print(f"{name:20s} {e:13.1f} {u:13.1f} {r:11.1f}")
new = sum(e for (name, *_), e in zip(blocks, expected) if "(new)" in name)
print(f"total expected bugs: {total_exp:.1f}; the three new blocks hold {100 * new / total_exp:.0f}%")
print(f"bugs found, uniform effort: {fu:.1f} ({100 * fu / total_exp:.0f}%)")
print(f"bugs found, effort by risk: {fr:.1f} ({100 * fr / total_exp:.0f}%)")
assert fr > fu * 1.15 and new / total_exp > 0.7
print("PASS: the same 80 weeks find many more bugs when spent where new, complex logic is")

It prints:

block                expected bugs uniform weeks  risk weeks
uart (reused)                  0.2          10.0         0.3
gpio (reused)                  0.1          10.0         0.2
dma (modified)                 4.8          10.0         8.4
cache ctrl (new)              18.0          10.0        31.4
interconnect (new)            15.0          10.0        26.2
timer (reused)                 0.1          10.0         0.2
crypto (modified)              3.6          10.0         6.3
power ctrl (new)               4.0          10.0         7.0
total expected bugs: 45.8; the three new blocks hold 81%
bugs found, uniform effort: 35.1 (77%)
bugs found, effort by risk: 42.5 (93%)
PASS: the same 80 weeks find many more bugs when spent where new, complex logic is

How to read it: the three new blocks (cache controller, interconnect, power controller) hold 81% of the expected bugs. Uniform effort gives them 30 of the 80 weeks and finds 77% of all bugs; effort by risk gives them about 65 weeks and finds 93%. The numbers are illustrative; the shape, a few blocks holding most of the risk, is typical.

Allocating effort by risk

• Rank blocks and features by novelty, complexity, change and history before planning (Part 9).

• Keep a floor for low-risk blocks: reused logic still needs integration checks and a regression.

• Re-rank when things change: a late change to a "done" block raises its risk at the worst moment.

• Watch where bugs are actually found: if a "low-risk" block keeps producing bugs, the estimate was wrong.

How real teams use it

What breaks

A project (illustrative) gave its "reused" Ethernet MAC the minimum effort. It had in fact been modified for a new bus width, a change nobody re-ranked. Most of the project's silicon escapes came from that block. Treating any changed block as partly new would have moved weeks of effort to it.

Key takeaways

• Bugs cluster in new, complex and changed logic and at boundaries between blocks and teams.

• The same effort finds many more bugs when allocated by risk instead of evenly.

• Keep a floor for low-risk blocks and re-rank whenever something changes.

Related topics

• Part 1 overview: What Can Go Wrong