dbglab's random test fails seed 17 in the nightly Icarus run. A colleague reruns it in Verilator, sees PASS, and writes in the bug: simulator issue, Icarus only, low priority. Another open bug, the power test, fails in both simulators but with different messages. A third, a hang, fails with every FIFO depth anyone tried. Each of these is a fact about the configuration the failure needs. What do they say, and how do you collect them without drowning in runs?
The common reading of a configuration difference is the cheapest one: "the other simulator passes, so it is a tool problem", or "it only fails with a 16-deep FIFO, so make it 64". Both treat a pass as an answer. The other common move is to change several things at once to "see if it goes away": new tool version, different options, another seed. When it goes away, nobody knows which change mattered, and when it comes back nobody knows why.
Configurations are dimensions of the failure, like time, hierarchy, stimulus and history. Some of them are ordered and can be halved. All of them can be varied one at a time, and the pattern of passes and fails is evidence.
List the dimensions of the failing run: the commit, the simulator, its options (how unknown and uninitialised values start), the design's parameters, the test's knobs, the seed, the tool versions. Recording everything to replay is the list.
Start from one failing configuration and change one dimension at a time. A flip from fail to pass names a dimension the failure depends on; no flip rules it out.
Bisect the ordered dimensions. Commits (bisection in history), numeric parameters, tool versions: find the boundary value where the verdict changes.
Compare the unordered ones in a matrix. Simulators and option sets have no middle. Seeds have no order at all: you cannot bisect them, only sample and sort them.
Read the pattern, not the cells. Fails in every simulator the same way: real logic. Fails only where uninitialised state is not zero: an unreset register. Fails differently in each: a difference in how they model something (X, timing, power). The failure moves with a parameter: that parameter is on the failing path.
Distrust every pass. Before a passing configuration counts as evidence, check that the test still exercises the feature, and that the stimulus reaches the new limit.
Read it by rows. Three rows are flat: the bug is in the logic, and every simulator agrees on it to the cycle. B4 is the row that fooled the colleague: it passes only where uninitialised flip-flops start at zero, which happens to be the value the test expects. B5 fails everywhere but differently, because the two simulators model an unpowered domain differently.
#!/bin/sh
# sim_matrix.sh: one failing run per bug, in three simulator configurations. (run from dbglab)
for run in "irq_w1c 1" "dma_burst_backpressure 1" "random 10" "random 17" "periph_power 1"; do
set -- $run
for cfg in icarus verilator xrand; do
case $cfg in
icarus) out=$(tb/run.sh $1 $2 icarus) ;;
verilator) out=$(tb/run.sh $1 $2 verilator) ;;
xrand) out=$(XRAND=1 tb/run.sh $1 $2 verilator) ;;
esac
printf '%-26s %-9s %s\n' "$1 $2" "$cfg" "$(echo "$out" | grep -E '^(PASS|FAIL)' | cut -c1-120)"
done
done$ ../probes/sim_matrix.sh
irq_w1c 1 icarus FAIL test=irq_w1c seed=1 cycle=84: IRQ: irq=1 while CTRL=00 enables no interrupt
irq_w1c 1 verilator FAIL test=irq_w1c seed=1 cycle=84: IRQ: irq=1 while CTRL=00 enables no interrupt
irq_w1c 1 xrand FAIL test=irq_w1c seed=1 cycle=84: IRQ: irq=1 while CTRL=00 enables no interrupt
dma_burst_backpressure 1 icarus FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu
...
random 10 xrand FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last
random 17 icarus FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read xxxxxxxx, expected 00000000 (last
random 17 verilator PASS test=random seed=17 cycles=18442 time=184416 ns
random 17 xrand FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last
periph_power 1 icarus FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
periph_power 1 verilator FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 c
periph_power 1 xrand FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cFifteen runs. XRAND=1 builds Verilator with --x-initial unique and starts every flip-flop without a reset at a random value. Read the B4 lines together: Icarus reads xxxxxxxx at cycle 7949, default Verilator passes, and Verilator with random start values reads 5d88e2dd at the same cycle and the same word. The same cycle in all three failing runs means the stimulus and the design behave identically; only the value of something uninitialised differs. That is the signature of a missing reset, and it is the opposite of a simulator issue: default Verilator passes by luck, because its flops start at zero.
#!/bin/sh
# commit_sim.sh <test> <seed> <commit>...: the run at each commit in three simulator configurations,
# then back to main. (run from the dbglab folder)
t=$1 s=$2; shift 2
for c in "$@"; do
git checkout -q "$c"
for cfg in icarus verilator xrand; do
case $cfg in
icarus) o=$(tb/run.sh $t $s icarus) ;;
verilator) o=$(tb/run.sh $t $s verilator) ;;
xrand) o=$(XRAND=1 tb/run.sh $t $s verilator) ;;
esac
printf '%s %-9s %s\n' "$c" "$cfg" "$(echo "$o" | grep -E '^(PASS|FAIL)' | cut -c1-120)"
done
done
git checkout -q main$ ../probes/commit_sim.sh random 17 03ebdc1 b357bd8
03ebdc1 icarus PASS test=random seed=17
03ebdc1 verilator PASS test=random seed=17
03ebdc1 xrand PASS test=random seed=17
b357bd8 icarus FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read xxxxxxxx, expected 00000000 (last
b357bd8 verilator PASS test=random seed=17
b357bd8 xrand FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (lastTwo dimensions at once, still one change per cell. Commit 13 passes everywhere; commit 14, "csr: no reset on the DMA address and data registers", fails in every configuration that does not start flops at zero. The failure lives in one corner of the matrix, and the corner names both the change and the class. A bisect of history run in default Verilator would have found nothing at all: every commit passes there. Pick the configuration you bisect in from the matrix, not from habit.
B2 hangs under back-pressure. Is it a capacity problem, so that a deeper peripheral FIFO cures it? AW sets the FIFO's depth to 2^AW. A small script copies HEAD's RTL into a scratch folder, changes one parameter with sed, and runs one test:
#!/bin/sh
# param_run.sh <name> <file> <sed expression> <test> <seed>: copy dbglab's rtl and tb into
# ../var/<name>, change one parameter with sed, run one test in Icarus. (run from the dbglab folder)
name=$1 file=$2 expr=$3 test=$4 seed=$5
d=../var/$name
rm -rf "$d"; mkdir -p "$d"; cp -R rtl tb "$d"/
sed -i '' "$expr" "$d/$file"
(cd "$d" && tb/run.sh "$test" "$seed" icarus | grep -E '^(PASS|FAIL)')$ for aw in 2 3 4 5 6; do printf "AW=%s depth=%-3s " $aw $((1<<aw)); ../probes/param_run.sh aw$aw rtl/dbglab_top.sv "s/periph_slave #(.AW(4))/periph_slave #(.AW($aw))/" dma_burst_backpressure 1; done
AW=2 depth=4 FAIL test=dma_burst_backpressure seed=1 cycle=2069: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003004
AW=3 depth=8 FAIL test=dma_burst_backpressure seed=1 cycle=2101: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
AW=4 depth=16 FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
AW=5 depth=32 FAIL test=dma_burst_backpressure seed=1 cycle=2293: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
AW=6 depth=64 FAIL test=dma_burst_backpressure seed=1 cycle=787: the FIFO filled only 0 times: no back-pressure was testedRead it line by line. From 4 to 32 entries the hang is always there, with the CPU's request stuck (cpu_ready=0). What moves is when: 32, 64 and then 128 cycles later as the depth doubles, which is what you would expect if the hang begins when the FIFO first fills. Depth is on the path (it decides when), but it is not the cause (no depth cures it). A deeper FIFO hides this bug for a while; it does not fix it.
The last line is the trap: with 64 entries the 48-word test never fills the FIFO, and only the test's own guard stops it reading as a fix (one change at a time follows that patch to the end). The commit dimension tells the true story: the same test passes at commit 10 (27acd30) and hangs at commit 11 (c4a71b5), "dma: keep the bus for a whole chunk", which is where the case study picks it up.
$ for cw in 8 9 10; do for s in 10 34; do printf "CW=%-2s seed %s: " $cw $s; ../probes/param_run.sh cw$cw rtl/dma_engine.sv "s/localparam int CW = 8;/localparam int CW = $cw;/" random $s | cut -c1-120; done; done
CW=8 seed 10: FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last
CW=8 seed 34: FAIL test=random seed=34 cycle=11465: SCOREBOARD: RAM word 830 (address 00001cf8) read 9ef7e786, expected d2dc7f79 (last
CW=9 seed 10: PASS test=random seed=10 cycles=21581 time=215806 ns
CW=9 seed 34: PASS test=random seed=34 cycles=20107 time=201066 ns
CW=10 seed 10: PASS test=random seed=10 cycles=21581 time=215806 ns
CW=10 seed 34: PASS test=random seed=34 cycles=20107 time=201066 ns
$ for ch in 4 2 1; do printf "chunk=%s seed 10: " $ch; ../probes/param_run.sh ch$ch rtl/dma_engine.sv "s/(left_q >= CW'(4)) ? 3'd4 : left_q\[2:0\]/(left_q >= CW'($ch)) ? 3'd$ch : left_q[2:0]/" random 10 | cut -c1-120; done
chunk=4 seed 10: FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last
chunk=2 seed 10: FAIL test=random seed=10 cycle=4400: SCOREBOARD: RAM word 843 (address 00001d2c) read 17e2db95, expected 836624c9 (last
chunk=1 seed 10: FAIL test=random seed=10 cycle=4493: SCOREBOARD: RAM word 853 (address 00001d54) read 19f3a7da, expected eda56799 (lastTwo parameters of the DMA, two different answers. The chunk length (words per read-then-write burst) changes the cycle and the first bad word but never the verdict: chunking is ruled out. The beat counter's width flips the verdict between 8 and 9 bits: the counter is on the path, and the boundary is sharp. So is 9 bits a fix? The random test never asks for more than 320 words. Push the stimulus past the new limit with one directed 600-word pattern fill (DMA draw 28, destination word 100), replayed in three copies of the replay bench from bisection in stimulus with the counter at 8, 9 and 10 bits:
$ cat fill600.ops
ie 0
fill 5a5a0001
dma 28 600 0 100 00000000
$ ../probes/fill600.sh
CW=8 FAIL test=replay seed=1 cycle=433: SCOREBOARD (final sweep): RAM word 188 (address 000012f0) holds c2d6183e, expected 5a5a0001 (last written
CW=9 FAIL test=replay seed=1 cycle=433: SCOREBOARD (final sweep): RAM word 188 (address 000012f0) holds c2d6183e, expected 5a5a0001 (last written
CW=10 PASS test=replay seed=1 cycles=2737 time=27366 nsNine bits fails exactly like eight: the fill starts at word 100 and stops at word 188, after 88 words, because 600 is 2 × 256 + 88 and also 512 + 88. Only a counter as wide as the 10-bit length register is a fix. The random seeds passed at 9 bits because their stimulus never reached 512.
Dimension | Pattern | What it says |
|---|---|---|
Simulator, for B1, B2, B3 | same failure, same cycle, everywhere | logic in the design; the tool is not involved |
Simulator and start values, for B4 | fails only where unreset flops are not zero | uninitialised state; look for a missing reset |
Simulator, for B5 | fails everywhere, differently | a modelling difference (unknown versus noise) on the path: a power or X question |
Commit, for B4 and B2 | flips at one commit, in the right configuration | the change that introduced it |
FIFO depth, for B2 | never cures; moves when the hang starts | on the path, not the cause; not a capacity bug |
Counter width, for B3 | flips between 8 and 9 for runs up to 320 words, and between 9 and 10 for a 600-word fill | the counter is the cause; its width must match the length |
Chunk length, for B3 | never flips | ruled out |
None of this needed a waveform. Each row is a hypothesis with a prediction already tested, and together they cover all five bugs in dbglab.
random 17 in default Verilator and reports "every commit passes; the bug must be in the test environment". What went wrong?The bisect ran in a configuration where the bug cannot show: default Verilator starts unreset flops at zero, the value the test expects. The matrix says B4 needs a configuration with unknown or random start values. Bisect again in Icarus or with XRAND=1, after checking both ends there; it lands on commit 14.
The depth decides when the hang starts, so the FIFO filling is part of the trigger, but no depth removes it, so the cause is not capacity: it is what the design does when the FIFO is full, typically a resource held while waiting. Next, find the commit where the hang appeared, and build the who-waits-for-whom table at the moment of the hang (who is waiting for whom).
dbglab rebuilt with dbglab/build-history.sh (HEAD 49c5f04), Icarus Verilog 13.0, Verilator 5.048, git 2.54.0, 2026-10-17. Simulator matrix: sim_matrix.sh as shown (15 runs, 87 s, most of it Verilator rebuilding when XRAND changes). Commit by simulator: random 17 at 03ebdc1 and b357bd8 in each configuration; dma_burst_backpressure 1 at 27acd30 (PASS) and c4a71b5 (FAIL at cycle 2165).
Parameters, Icarus, each in a scratch copy of HEAD's rtl/ and tb/ with one sed (param_run.sh shown): periph_slave AW 2 to 6 for dma_burst_backpressure 1; DMA CW 8, 9, 10 for random 10 and 34; chunk length 4, 2, 1 for random 10. The 600-word fill (ie 0, fill 5a5a0001, dma 28 600 0 100) ran in Part 1's scratch replay bench with CW 8, 9 and 10.
Limits: one machine, one version of each tool; a third simulator would add a column, not change the method. Illustrative only: none. A matrix shows which dimensions a failure depends on; it does not show why.