Five of dbglab's sixteen commits added a bug: a reset value, a deadlock, a counter that wraps, a hidden X and a missing isolation clamp. All five are now found. The project lead asks the question that matters for the next project: we had seeds, a scoreboard, protocol checks and a watchdog. Why did they miss all five?
"It didn't have enough tests" is not an answer. Which gate did each bug get through, and what in the plan changes?
Add a test for each bug. It pins that bug (the regression test that pins the bug), but says nothing about why the existing tests missed it, so the next bug of the kind gets through the same way.
"Run more seeds." Sometimes right, but without knowing why the seeds missed you cannot say how many, and for three of these five bugs (B1, B2 and B5, which already failed every night) more seeds would have changed nothing.
Analyse today's verification. The bench at HEAD is not the bench at the bad commit. An escape has to be replayed with the tests, seeds and checks that existed then.
For a bug to be caught, four things must all happen. An escape is the first one that did not:
Planned: some test or check is meant to cover the behaviour at all.
Reached: the stimulus actually drives the design into the state the bug needs, in the runs the regression makes.
Checked: something compares the result with the expected one, and fires when they differ, early enough to point at the cause.
Acted on: the failure reaches a person, and the commit does not land until it is understood.
Then, for each bug: check out the bad commit, run what existed, and show the gate with one run. A coverage count of zero shows a stimulus hole. A check added to a scratch copy that fires shows a checker hole. A regression diff that nobody read shows a process hole. Finally, write one change per gate into the plan, with a number someone can measure, and check it by putting the bug back.
Two stories. Three bugs were caught by verification and stopped by nobody, because the regression was already red: the process cause from the root-cause lesson. Two were never seen at all. The runs below prove both.
Check out each commit and run its own regression. tb/regress.sh arrives in commit 5 and defaults to seeds 1, 2 and 3 at every commit after it. A ten-line loop does the replay:
#!/bin/sh
# For each commit of dbglab: run tb/regress.sh icarus 1 2 3 and print the failing lines.
cd "$1" || exit 1
n=0
for c in $(git rev-list --reverse main); do
n=$((n+1)); git checkout -q $c
out=$(tb/regress.sh icarus 1 2 3)
printf '%2d %s %s\n' $n $(git log -1 --format=%h) "$(echo "$out" | tail -1)"
echo "$out" | grep FAIL | sed 's/^/ /'
done
git checkout -q main$ ../regress-by-commit.sh .
...
5 fa71142 0 failing run(s)
...
8 2e63466 0 failing run(s)
9 026f8bf 3 failing run(s)
irq_w1c seed 1 FAIL
...
11 c4a71b5 6 failing run(s)
...
dma_burst_backpressure seed 3 FAIL
12 bc2bb34 6 failing run(s)
...
13 03ebdc1 6 failing run(s)
...
14 b357bd8 6 failing run(s)
...
15 122e88a 9 failing run(s)
...
periph_power seed 3 FAIL
16 49c5f04 9 failing run(s)
...Commits 9, 11 and 15 each add three failing runs: B1, B2 and B5 were caught, by the tests written for them, on the day they landed. Their gate is the fourth one.
Commits 13 and 14 change nothing: 6 failing runs before, 6 after, the same six. B3 and B4 went into a design whose regression could not see them.
Both bugs live in the random test, which was planned to reach them: its generator makes 1 in 400 DMA runs 257 to 320 words long (B3 needs more than 255), and it sometimes zero-fills a buffer (B4 needs a zero fill before software has written FILL). Planned is not reached. A probe counts how often the runs actually get there:
// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// cov_probe: counts the DMA runs that reach the scenarios B3 and B4 need. It only reads the design.
// long : a run longer than 255 words (B3 loses the top bits of LEN)
// fill_unwr : a fill run started before software ever wrote FILL (B4 reads the reset value)
`timescale 1ns/1ps
module cov_probe;
int runs = 0, long_runs = 0, fills = 0, fill_unwr = 0;
bit fill_written = 0;
always @(posedge dbglab_tb.clk) begin
if (dbglab_tb.dut.u_csr.wr && dbglab_tb.dut.u_csr.off == 12'h024) fill_written = 1;
if (dbglab_tb.dut.u_dma.start && !dbglab_tb.dut.u_dma.busy) begin
runs++;
if (dbglab_tb.dut.u_dma.cfg_len > 255) long_runs++;
if (dbglab_tb.dut.u_dma.cfg_ctrl[4]) begin
fills++;
if (!fill_written) fill_unwr++;
end
end
end
final $display("COV seed=%0d dma_runs=%0d long(>255)=%0d fills=%0d fill_before_FILL_written=%0d",
dbglab_tb.seed, runs, long_runs, fills, fill_unwr);
endmodule$ git checkout -q b357bd8
$ for s in 1 2 3; do ../probes/run-probe.sh ../probes/cov_probe.sv random $s | grep COV; done
COV seed=1 dma_runs=131 long(>255)=0 fills=16 fill_before_FILL_written=0
COV seed=2 dma_runs=129 long(>255)=0 fills=11 fill_before_FILL_written=0
COV seed=3 dma_runs=122 long(>255)=0 fills=8 fill_before_FILL_written=0382 DMA runs, 35 fills, and not one of the two scenarios. The regression ran, passed, and tested neither bug. Run the same probe over seeds 1 to 60 at commit 12, where neither bug exists yet, and compare with the seeds that fail once each bug arrives:
$ ../cov-sum.sh ../runs/cov-commit12.txt
seeds 60 dma_runs 7459 long 15 seeds_long 12 fill_unwr 10 seeds_unwr 9
$ cat ../runs/random-sweep-12-13-14.txt
bc2bb34:
03ebdc1: 10 13 16 20 34 46 48 50 51 52 58 59
b357bd8: 10 13 16 17 19 20 23 29 31 34 41 43 46 48 50 51 52 58 59The 12 seeds with a long run are exactly the 12 seeds that fail at commit 13. The 9 seeds with an early zero fill (16 17 19 23 29 31 41 43 51) are exactly the ones that fail with an X at commit 14; two of them, 16 and 51, already failed at 13. The scoreboard was ready to catch both bugs. The seeds never gave it the chance.
# How likely is a regression of k fresh random seeds to reach each rare scenario?
# p = fraction of seeds 1-60 (commit 12, cov_probe) that reach it.
reach = {"B3 long run (>255 words)": 12 / 60, "B4 fill before FILL written": 9 / 60}
for name, p in reach.items():
print(f"{name}: p per seed = {p:.2f}")
for k in (3, 10, 30, 60):
print(f" {k:2d} fresh seeds: P(miss) = {(1 - p) ** k:.3f}")
n = 1
while (1 - p) ** (3 * n) > 0.05:
n += 1
print(f" 3 fresh seeds a night: {n} nights for a 95% chance of a hit")
print("3 fixed seeds (1 2 3), any number of nights: P(miss) = 1.000 for both")
assert round((1 - 12 / 60) ** 3, 3) == 0.512 and round((1 - 9 / 60) ** 3, 3) == 0.614
print("PASS: odds computed from the measured reach of seeds 1-60")$ python3 escape_odds.py
B3 long run (>255 words): p per seed = 0.20
3 fresh seeds: P(miss) = 0.512
10 fresh seeds: P(miss) = 0.107
30 fresh seeds: P(miss) = 0.001
60 fresh seeds: P(miss) = 0.000
3 fresh seeds a night: 5 nights for a 95% chance of a hit
B4 fill before FILL written: p per seed = 0.15
3 fresh seeds: P(miss) = 0.614
10 fresh seeds: P(miss) = 0.197
30 fresh seeds: P(miss) = 0.008
60 fresh seeds: P(miss) = 0.000
3 fresh seeds a night: 7 nights for a 95% chance of a hit
3 fixed seeds (1 2 3), any number of nights: P(miss) = 1.000 for both
PASS: odds computed from the measured reach of seeds 1-60The last line is the real escape: three fixed seeds replay the same 382 DMA runs every night. A random test with fixed seeds is a directed test that nobody chose (statistics of rare failures has the arithmetic).
B4 has a second hole. When a seed does reach it, the scoreboard notices only when someone reads the bad word, 7,461 cycles later for seed 17 (scoreboards report late). The bus checker would see the X the moment it is written, but it checks address and write, not write data. Add that one term in a scratch copy of the bench (dbglab itself is unchanged):
$ git diff
...
- if ($isunknown(valid) || (valid && ($isunknown(addr) || $isunknown(write))))
- violation("R1", $sformatf("valid=%b addr=%h write=%b", valid, addr, write));
+ if ($isunknown(valid) || (valid && ($isunknown(addr) || $isunknown(write) || (write && $isunknown(wdata)))))
+ violation("R1", $sformatf("valid=%b addr=%h write=%b wdata=%h", valid, addr, write, wdata));
...
$ tb/run.sh random 17
dbglab: test=random seed=17
FAIL bus_rules dma R1 at 4875000: valid=1 addr=00001ca8 write=1 wdata=xxxxxxxxTime 4875000 ps is cycle 488, the first X write, instead of cycle 7949. Over seeds 1 to 60 at commit 14 the new rule fires in exactly the 9 early-zero-fill seeds, and at commit 13 it adds no failure at all, so it costs no false alarms. But with seeds 1, 2 and 3 it passes: a check without the stimulus to reach the bug finds nothing, and stimulus without the check finds it late. You need both.
One more blind spot: the simulator. The same seed passes in Verilator's default two-state mode, because its flip-flops start at 0, the value the bench expects:
$ tb/run.sh random 17 verilator
dbglab: test=random seed=17
PASS test=random seed=17 cycles=18442 time=184416 ns
$ XRAND=1 tb/run.sh random 17 verilator
dbglab: test=random seed=17
FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)B3 had a check before any test: lint. The cast added in commit 13 silenced it, as fix the cause shows, and the bench builds Verilator with -Wno-lint anyway. Lint at HEAD was already noisy:
$ verilator --lint-only $(grep -v '^#' rtl/files.f) --top-module dbglab_top 2>&1 | grep '^%Warning'
%Warning-WIDTHEXPAND: rtl/async_fifo.sv:30:36: Operator AND expects 5 bits on the LHS, but LHS's VARREF 'winc' generates 1 bits.
...Eight warnings already stand at HEAD, all harmless widenings. In a list of eight, a ninth would have been hard to see; the cast made sure there was no ninth. The warning was reached and checked, then silenced by the change that introduced the bug.
Bug | Gate | Shown by | What would have caught it (checked) |
|---|---|---|---|
B1 reset value | acted on | regress 0 to 3 at commit 9 | a gate on new failing runs |
B2 deadlock | acted on | regress 3 to 6 at commit 11 | the same gate |
B3 wrap | reached (and lint silenced) | no long run in seeds 1-3 | fresh seeds or a coverage goal on LEN; lint with no new warnings and no unexplained narrowing cast |
B4 hidden X | reached, then checked late | no early zero fill in seeds 1-3; R1 blind to data | a coverage goal on fill-before-write; an X check on write data; a random-init run |
B5 isolation | acted on | regress 6 to 9 at commit 15 | the new-failure gate; an isolation check from the boundary list |
No. The bin 25-320 is hit by any run of 25 words or more, so it says nothing about the boundary at 256 where B3 lives. Coverage is only as good as its bins: split the range at the counter's width (255, 256, 257) and add the maximum. With those bins, seeds 1 to 3 show zero hits, which is the hole this lesson found.
It reaches B3 and B4 on the first night (the miss probabilities are effectively zero), so the scoreboard catches both, B4 late. It does nothing for B1, B2 and B5: those already failed every night and nobody acted. Without the new-failure gate, 60 seeds only make the red list longer.
dbglab rebuilt with dbglab/build-history.sh (HEAD 49c5f04), 2026-10-18, Icarus Verilog 13.0. tb/regress.sh icarus 1 2 3 at every commit from 5 (where the script appears; its seeds default to 1 2 3 at every commit): 0 failing runs at 5 to 8, 3 at 9 and 10, 6 at 11 to 14, 9 at 15 and 16. random seeds 1 to 60: no failures at bc2bb34 (commit 12); 12 at 03ebdc1; 19 at b357bd8, lists as shown.
cov_probe.sv compiled next to dbglab (iverilog -g2012 -Wno-timescale -I tb <rtl/files.f> tb/bus_rules.sv tb/dbglab_tb.sv cov_probe.sv, vvp -n ... +TEST=random +SEED=<s>): seeds 1 to 3 at b357bd8 reach neither scenario; at commit 12 over seeds 1 to 60, 15 long runs in 7,459 DMA runs, in 12 seeds, and 10 early zero fills in 9 seeds. Odds from escape_odds.py (Python 3.14.3) as shown.
A scratch clone with the write-data X check shown: at b357bd8 seeds 1 to 3 pass and seeds 1 to 60 fail R1 in exactly the 9 early-zero-fill seeds; at 03ebdc1 it adds no failure. Verilator 5.048: tb/run.sh random 17 verilator passes, with XRAND=1 it fails at cycle 7949. verilator --lint-only <rtl/files.f> --top-module dbglab_top at HEAD prints 8 warnings, none in dma_engine.sv; on a copy without the CW'() cast it adds WIDTHTRUNC at dma_engine.sv:89.
Limits: one small design with a teaching history, so each escape has one clear gate; real escapes often have two. The plan changes are checked by putting each bug back, not by running a full project's regression.