dbglab's nightly now runs random in Verilator with random start values (XRAND=1). Nineteen of sixty seeds fail, and every message has the same shape:
$ for s in $(seq 1 60); do XRAND=1 tb/run.sh random $s verilator | grep FAIL; done
FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)
FAIL test=random seed=13 cycle=3701: SCOREBOARD: RAM word 250 (address 000013e8) read f42b063a, expected c6278de6 (last written by DMA run #22)
FAIL test=random seed=16 cycle=686: SCOREBOARD: RAM word 759 (address 00001bdc) read 600a6c51, expected 00000000 (last written by DMA run #3)
FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)
...
FAIL test=random seed=59 cycle=3217: SCOREBOARD: RAM word 377 (address 000015e4) read 913f438a, expected 3fec7961 (last written by DMA run #22)Last week the team found the beat counter that wraps at 256 words (the debug loop walks it through on seed 10). Its failures read exactly like these: a RAM word wrong, last written by a DMA run. The lead says: "It's the wrap bug. Merge the fix and these go away." Do they?
They merge the fix and rerun. Here is the real result:
$ for s in $(seq 1 60); do XRAND=1 tb/run.sh random $s verilator | grep FAIL; done # with CW = 10
FAIL test=random seed=16 cycle=686: SCOREBOARD: RAM word 759 (address 00001bdc) read 600a6c51, expected 00000000 (last written by DMA run #3)
FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)
FAIL test=random seed=19 cycle=415: SCOREBOARD: RAM word 984 (address 00001f60) read 4443c624, expected 00000000 (last written by DMA run #3)
FAIL test=random seed=23 cycle=2385: STREAM: word #41 = f0db2f1b, expected 00000000
...
FAIL test=random seed=51 cycle=1155: SCOREBOARD: RAM word 929 (address 00001e84) read efcb7c6f, expected 00000000 (last written by DMA run #7)Nineteen became nine. Inside one explanation, that reads as the fix is incomplete. The DMA has two more narrow counters (chunk_q and rd_q, three bits each), so the next two days go on those: widen them, rerun, still nine. Then the arbiter. The story is illustrative; the nine failures are real, and nothing about counters will ever touch them.
This is fixation. The first explanation becomes the lens, so new evidence is read through it instead of against it. A partial success makes it worse: "it went from 19 to 9, so we are on the right track." But a fix that removes ten failures confirms the hypothesis for those ten, and says nothing about the other nine.
Write at least two hypotheses for any group of failures, and make one of them "two causes at once". If you cannot think of a second, you have not looked at the evidence, only at the first explanation.
For each hypothesis, name a feature it predicts per failure, cheap enough to compute for every one: a run length, a register write order, a value, a signature in the message.
Score every failure against every hypothesis. A table, one row per failure. Failures that fit none go in a "neither" bucket, never into the nearest hypothesis.
Weigh only discriminating evidence. A clue that both hypotheses predict ("a RAM word is wrong") moves nothing.
Predict what each partial fix leaves behind, seed by seed, before running it. This is where one change at a time earns its keep.
Allow both. One failure can carry two causes; the first to fire hides the second.
Hypothesis | Mechanism | Feature it predicts in a failing seed |
|---|---|---|
W: counter wraps | a DMA run (copy or fill) longer than 255 words ends early and leaves stale words | a run with LEN over 255 before the failure |
F: FILL unreset | a zero fill writes FILL's power-up value | a zero fill before any write to FILL |
Each feature is something one hypothesis needs and the other does not care about, so it discriminates. Both come from the run_log probe shown in a hypothesis must predict, which flags each run it lists. One detail matters: at HEAD a test stops at its first failure, so a long run after an early fill failure would never be seen. Run the probe in a copy with both candidate fixes, where every test runs to the end:
$ for s in $(seq 1 60); do o=$(sh ../probes/run-probe.sh ../probes/run_log.sv random $s); f=""; echo "$o" | grep -q "longer than 255" && f="$f long"; echo "$o" | grep -q "FILL never written" && f="$f zfill"; echo "$o" | grep -q "^FAIL" && f="$f FAIL"; [ -n "$f" ] && echo "$s:$f"; done | tee ../logs/features-b34fix.txt
10: long
13: long
16: long zfill
17: zfill
19: zfill
20: long
23: zfill
29: zfill
31: zfill
34: long
41: zfill
43: zfill
46: long
48: long
50: long
51: long zfill
52: long
58: long
59: longRead it as a scoring table. Exactly nineteen seeds carry a feature, and they are exactly the nineteen that fail. Ten fit W only, seven fit F only, and seeds 16 and 51 fit both. No seed fails without a feature, so the "neither" bucket is empty, and no passing seed has one, so neither hypothesis predicts a failure that did not happen.
Now the hypotheses make sharp, separate predictions. Written before running:
Fix applied | Predicted to fail | Count |
|---|---|---|
W's fix only ( | the zfill seeds: 16 17 19 23 29 31 41 43 51 | 9 |
F's fix only (FILL reset) | the long seeds: 10 13 16 20 34 46 48 50 51 52 58 59 | 12 |
both | none | 0 |
$ for d in b3fix b4fix b34fix; do (cd $d; for s in $(seq 1 60); do XRAND=1 tb/run.sh random $s verilator | grep FAIL; done > ../logs/vlx-$d.txt); echo "$d: $(wc -l < logs/vlx-$d.txt) fails: $(sed 's/.*seed=\([0-9]*\).*/\1/' logs/vlx-$d.txt | tr '\n' ' ')"; done
b3fix: 9 fails: 16 17 19 23 29 31 41 43 51
b4fix: 12 fails: 10 13 16 20 34 46 48 50 51 52 58 59
b34fix: 0 fails:All three predictions hold, seed for seed (b3fix is the counter fix, b4fix the FILL fix). The nine that "survived the fix" were never the counter's.
Seeds 16 and 51 fail under either partial fix, and their message changes with the fix. Seed 51:
$ (cd b3fix; XRAND=1 tb/run.sh random 51 verilator) # counter fixed
dbglab: test=random seed=51
FAIL test=random seed=51 cycle=1155: SCOREBOARD: RAM word 929 (address 00001e84) read efcb7c6f, expected 00000000 (last written by DMA run #7)
$ (cd b4fix; XRAND=1 tb/run.sh random 51 verilator) # FILL fixed
dbglab: test=random seed=51
FAIL test=random seed=51 cycle=11722: SCOREBOARD: RAM word 322 (address 00001508) read 15d021ca, expected 00000000 (last written by DMA run #75)At HEAD the early zero fill fails first, at cycle 1155, and hides the long run that would fail at 11722. Fix FILL and the counter bug appears; fix the counter and the FILL bug is still there. Someone holding one hypothesis sees "the fix moved the failure" and suspects the fix. Someone holding two sees a seed with both features, and expects exactly this.
Two hypotheses, each with a feature the other does not need, explained all nineteen failures, predicted which would remain after each fix, and explained why two seeds changed signature. One hypothesis explained ten and turned the other nine into a two-day hunt in the wrong module.
Notice the clue that was in the messages all along: every FILL failure says expected 00000000. It is a good first sort, but not proof: seed 51's counter failure also expects zero, because its destination had been zero-filled earlier, and seed 20's counter failure reads zero. Use a cheap clue to form the groups; use a feature that one hypothesis forbids to confirm them.
expected 00000000. Your colleague files it under the FILL bug. What one check would you ask for?Run the run_log probe on it and look for a zero fill before any FILL write (F's feature) and a run longer than 255 words (W's feature). Only F's feature makes it a FILL duplicate; if it has neither, it is a third mechanism, and if it has both, expect it to keep failing after the FILL fix. The expected 00000000 alone does not decide it: seed 51's counter failure also expects zero.
At HEAD each test stops at its first failure, so a feature that occurs later in the stimulus is never reached: seed 51's long run comes after its fill failure at cycle 1155. With both fixes the tests run to the end, so the probe sees the whole stimulus. The general point: measure features where the first bug cannot hide the second.
dbglab at HEAD 49c5f04 (rebuilt with dbglab/build-history.sh), Verilator 5.048 with XRAND=1 (--x-initial unique, start-value seed = test seed), Icarus Verilog 13.0, 2026-10-17. for s in $(seq 1 60); do XRAND=1 tb/run.sh random $s verilator | grep FAIL; done gives 19 failures at HEAD, 9 with CW = 10 only, 12 with FILL in the reset block only, 0 with both (each on a copy of HEAD).
Features: run_log.sv (from the lesson on predictions) compiled into the copy with both fixes, so every test reaches its end, and run over seeds 1 to 60 with sh ../probes/run-probe.sh ../probes/run_log.sv random $s, grepping for "longer than 255" and "FILL never written". Flagged seeds: 12 and 9, overlapping in 16 and 51; no other seed is flagged.
Limits: two mechanisms in one design, in simulation. The cost of the fixation path in the scene is illustrative. The feature probe reads the stimulus of the fixed design, which matches HEAD up to each run's first failure.