The same bug can be fixed in a minute or cost a company millions. In a block testbench, a designer sees the failing assertion, edits one line and reruns. In silicon, the same mistake means debugging with almost no visibility, a new mask set and months of delay. In customers' hands, it means replacements, workarounds and reputation. Nothing about the bug changed; only when it was found.
Why does a bug cost so much more when it is found late, and what does that mean for how verification effort is spent?
The obvious view is to count bugs: a project that finds 200 bugs has done its job. Counting treats every bug as equal. They are not: the cost of a bug depends almost entirely on where it is found. Two projects can find the same number of bugs and differ fivefold in what verification cost them, because a handful found late outweigh hundreds found early.
Stage found | What finding and fixing costs |
|---|---|
Block testbench | Full visibility, fast reruns: an edit and a rerun |
SoC or subsystem test | Slower runs, more people involved, harder to localise |
Emulation or prototype | Long runs to reproduce, debug with limited visibility |
First silicon | Debug with almost no visibility, then a respin: new masks and months of schedule |
In the field | Replacements or recalls, software workarounds, lost customers |
Illustrative numbers: each project has 200 bugs, and each stage has a cost per bug, in thousands of dollars. Project "early" finds most bugs at block level; project "late" leaves more for later stages.
"""The cost of an escape: where bugs are found decides what verification costs.
Illustrative cost to find and fix one bug at each stage, in thousands of dollars:
block test 2, SoC test 10, emulation 40, first silicon 800 (a share of a respin and its
schedule slip), field 8,000 (replacement, recall, reputation). The numbers are
illustrative; the ratios between stages are what matter.
Two projects each have 200 bugs. Project "early" finds most of them at block level;
project "late" leaves more for SoC, emulation and silicon. The model prints the share of
bugs found at each stage, the cost of each stage and the totals.
"""
STAGES = ["block", "SoC", "emulation", "silicon", "field"]
COST_K = [2, 10, 40, 800, 8000]
BUGS = 200
projects = {
"early": [0.70, 0.20, 0.08, 0.015, 0.005],
"late": [0.40, 0.30, 0.20, 0.07, 0.03],
}
print("stage: " + " ".join(f"{s:>10s}" for s in STAGES) + " total")
print("cost per bug ($k):" + " ".join(f"{c:>10,d}" for c in COST_K))
totals = {}
for name, share in projects.items():
counts = [BUGS * f for f in share]
costs = [n * c for n, c in zip(counts, COST_K)]
totals[name] = sum(costs)
print(f"{name:6s} bugs: " + " ".join(f"{n:>10.0f}" for n in counts))
print(f"{name:6s} cost ($k): " + " ".join(f"{c:>10,.0f}" for c in costs) + f" {totals[name]:>10,.0f}")
late_tail = sum(BUGS * f * c for f, c in zip(projects["late"][3:], COST_K[3:]))
print(f"late project: silicon and field bugs are {100 * (projects['late'][3] + projects['late'][4]):.0f}% of bugs and {100 * late_tail / totals['late']:.0f}% of cost")
print(f"late / early total cost: {totals['late'] / totals['early']:.1f}x")
assert totals["late"] > 3 * totals["early"]
assert late_tail / totals["late"] > 0.9
print("PASS: the few bugs that reach silicon and the field dominate the cost; finding them earlier cuts it several times over")It prints:
stage: block SoC emulation silicon field total
cost per bug ($k): 2 10 40 800 8,000
early bugs: 140 40 16 3 1
early cost ($k): 280 400 640 2,400 8,000 11,720
late bugs: 80 60 40 14 6
late cost ($k): 160 600 1,600 11,200 48,000 61,560
late project: silicon and field bugs are 10% of bugs and 96% of cost
late / early total cost: 5.3x
PASS: the few bugs that reach silicon and the field dominate the cost; finding them earlier cuts it several times overHow to read it: the two projects found the same 200 bugs. The late project's 14 silicon bugs and 6 field bugs are 10% of its bugs and 96% of its cost, and it spends 5.3 times as much in total. The ratios between stages are what matter; the dollar figures are illustrative.
Shift left: every bug moved from silicon to block level saves orders of magnitude, so most verification effort goes into block and subsystem levels.
Spend more where escapes are expensive: features whose bugs cannot be worked around in software deserve the most rigour.
Count where bugs are found, not just how many: a rising share found late is a warning, even if the total looks healthy (Part 12).
Accept some cost: driving the late share to zero is impossible (lesson 0.4); the goal is to make it small and known.
Practice | Purpose |
|---|---|
Block-level verification with full checking before integration | Catch most bugs where they are cheapest |
Emulation and prototypes before tape-out | Find bugs that need long runs or software while they are still cheap compared with silicon |
Software workaround and patch paths | Lower the cost of the escapes that do happen |
Tracking the stage where each bug was found | Measure whether effort is landing early enough |
A team (illustrative) cut its block-level verification to meet a schedule, reasoning that SoC tests would catch whatever block tests missed. The SoC environment ran a thousand times slower and could not reach many block-level corners. Two of those corner bugs reached silicon; one needed a respin. The months saved on block verification cost more months of schedule than they saved.
3 in silicon: 3 × 800 = $2,400k, against 50 × 40 = $2,000k (illustrative numbers).
Not necessarily: bugs are being found later, where each costs more, which suggests block-level verification is missing them.
The cost of a bug grows by orders of magnitude with how late it is found.
A few late escapes dominate total cost; count where bugs are found, not just how many.
Verification effort pays most where it moves bugs earlier.