The nightly regression now runs dbglab's random test in Verilator with random start values (XRAND=1), the closest a simulation gets to a chip powering up. Seed 29 fails:
$ XRAND=1 tb/run.sh random 29 verilator
dbglab: test=random seed=29
FAIL test=random seed=29 cycle=1820: SCOREBOARD: RAM word 542 (address 00001878) read a82c9528, expected 00000000 (last written by DMA run #8)The tracker already holds two open bugs in this corner: long DMA runs that end early, and the FILL register that lost its reset. Is seed 29 one of those, or a third bug? The meeting is in twenty minutes, and you have time to set up one experiment. Which one?
They run whatever is nearest. Rerun the command (it fails the same way, as every deterministic run does). Try seed 30. Open a waveform and scroll. Or they pick the one suspect they like and test only that: "let me check whether run #8 was long". If the answer is no, three suspects are left and the meeting starts.
None of those is wrong, but none was chosen. Guessing vs searching showed that the best yes-or-no run halves the suspects. Here the suspects are not a range you can bisect; they are four different mechanisms. So you need a different tool to find the halving run: a table.
Write the suspects as mechanisms. Three to five, each specific enough to predict something (see a hypothesis must predict). Include "the bench is wrong" when it is plausible.
List candidate runs, including cheap ones you would normally not think of: another simulator, another start-value seed, a known fix applied, an older commit.
Fill every cell: what does this run show if that suspect is the cause? Same failure, a different value, X, PASS.
Delete columns where every row agrees. They cannot change your mind.
Weigh the rest. Group the suspects by predicted result. The run is worth most when the groups carry about equal probability. Then divide by cost: a one-second run beats a one-hour build of nearly equal worth.
Check the suspects are exclusive. If two can be true at once, plan a second run that tests the survivor on its own.
For seed 29 the four suspects are:
Suspect | Mechanism |
|---|---|
A: the counter wraps (open bug) | an earlier long copy ended early, so word 542 kept stale data |
B: FILL has no reset (open bug) | run #8 is a zero fill that wrote FILL's power-up value |
C: a new fill datapath bug | the fill logic from commit 12 writes a wrong word (a logic bug, so the same word every run) |
D: a bench model bug | the design is right and the scoreboard's expected value is wrong |
Read it column by column. Rerun is grey all the way down: every suspect predicts the same failure, so it can teach nothing. Three columns (other start values, Icarus, two-state) single out B from the other three. Only both known fixes splits two against two.
A yes-or-no run whose outcome you give probability p is worth H(p) bits, as in guessing vs searching:
If all four suspects were equally likely, the 2:2 column would win: 1 bit against 0.81. But they are not. Two of them are open bugs that already explain similar failures; two are new and unexplained. Say A and B are 0.4 each and C and D 0.1 each (a judgement, written down so it can be argued with):
Run | Splits | Equal odds | A, B likelier (0.4, 0.4, 0.1, 0.1) | Cost |
|---|---|---|---|---|
Rerun | nothing | 0 bits | 0 bits | 1 s |
Icarus, four-state | B against A, C, D | 0.81 bits | 0.97 bits | 1 s |
Two-state Verilator | B against A, C, D | 0.81 bits | 0.97 bits | 1 s |
Both known fixes applied | A, B against C, D | 1.00 bit | 0.72 bits | a patched copy and a rebuild |
With realistic odds the Icarus run is worth more and costs a second. The 2:2 run answers "known or new?", which matters less when known is already likely. So run Icarus first, and keep the fix build for whatever survives.
Prediction, written before running: if B, the read value is X; if A, C or D, it is a82c9528 again, because none of them involves an unknown value.
$ tb/run.sh random 29
dbglab: test=random seed=29
FAIL test=random seed=29 cycle=1820: SCOREBOARD: RAM word 542 (address 00001878) read xxxxxxxx, expected 00000000 (last written by DMA run #8)X. A, C and D each predicted a known value, so all three are out in one second, and the result also says why the XRAND value looked random: it was FILL's random start value. The cheaper columns agree, as they should, since they ask the same question: with +verilator+seed+7 the same read gives a8c3ed4a instead of a82c9528, and two-state Verilator passes.
One suspect left, but do not stop at elimination: a suspect nobody listed would also have survived. Test B directly with a prediction that could fail: with only the FILL fix applied, seed 29 passes with XRAND on.
$ XRAND=1 tb/run.sh random 29 verilator # a copy of HEAD with FILL in the reset block
dbglab: test=random seed=29
PASS test=random seed=29 cycles=18522 time=185216 nsIt passes, in 18522 cycles, the same count as two-state Verilator at HEAD. Seed 29 is a duplicate of the FILL bug, and you walk into the meeting with that and the two commands that show it. For completeness: the 2:2 column, both fixes applied, also passes, and so does commit 12, which has fill mode but neither open bug.
Two runs settled four suspects, because the first was chosen from a table rather than from habit. Note what the table did beyond choosing: it made you write that a rerun was worthless before doing it, and it turned "Icarus shows X, Verilator does not" from a tool curiosity into the deciding measurement.
The tracker below holds two real dbglab failures, each with the suspects a team would name in the first five minutes. Each experiment button shows its cost and how many groups it would split the remaining suspects into. Aim for one suspect left at the lowest cost, then check that the last experiment you ran changed the design and passed: crossing out the others is not the same as proving the survivor.
Loading hypothesis tracker…
Rerun: same. Other start values: same (no unknown values involved). Icarus: the same known value, not X. Two-state Verilator: FAIL. Both known fixes: FAIL. So E sits with C and D in every column, and the Icarus run eliminates it as well. A new suspect costs nothing to add to the table, and you can see at once which existing run already answers it.
When the four suspects really are about equally likely, for example when nothing like this has been seen before, and when the 2:2 run is cheap. Here two suspects were open bugs that already explained similar failures, so a run separating those from each other was worth more, and it cost a second instead of a rebuild.
dbglab at HEAD 49c5f04 (rebuilt with dbglab/build-history.sh), 2026-10-17. Report: XRAND=1 tb/run.sh random 29 verilator (Verilator 5.048, --x-initial unique, start-value seed 29) fails at cycle 1820 reading a82c9528; with +verilator+seed+7 appended it reads a8c3ed4a, with +verilator+seed+8 9d9758b2. tb/run.sh random 29 (Icarus 13.0) fails at cycle 1820 with xxxxxxxx; tb/run.sh random 29 verilator (two-state) passes in 18522 cycles.
Fix builds (copies of HEAD): with both open fixes (FILL in the reset block, CW = 10) seed 29 passes with XRAND=1; with the FILL fix alone it passes; with CW = 10 alone it fails as at HEAD. Commit 12 (bc2bb34, a worktree) passes seed 29 in both modes. Seed 51 with the FILL fix alone and XRAND=1 fails at cycle 11722 reading 15d021ca.
The information figures are computed from the probabilities stated, not measured. Limits: one failure, four suspects chosen by hand; a fifth mechanism nobody listed would survive every run here.