dbglab's random test fails on seed 19: RAM word 984 reads xxxxxxxx at cycle 415, and the bench says DMA run #3 wrote it last. Two engineers add a line to the ticket.
One writes: the DMA corrupts data when the CPU is busy on the bus at the same time. The other writes: FILL has no reset, and run #3 is a zero fill that uses it before anyone writes it. Both sound reasonable. Which one can you test, and what would you run first?
Most first hypotheses name an area: the DMA, the arbiter, "timing", "traffic". They feel like progress because they point somewhere. But look at what happens when the first engineer tests theirs. Every run below is real:
$ tb/run.sh dma_copy 19
dbglab: test=dma_copy seed=19
PASS test=dma_copy seed=19 cycles=1182 time=11816 ns
$ tb/run.sh random 18
dbglab: test=random seed=18
PASS test=random seed=18 cycles=19229 time=192286 ns
$ tb/run.sh random 19 icarus +OPS=20
dbglab: test=random seed=19
FAIL test=random seed=19 cycle=400: SCOREBOARD (final sweep): RAM word 975 (address 00001f3c) holds xxxxxxxx, expected 00000000 (last written by DMA run #3)Run | Result | Read as | Had it gone the other way |
|---|---|---|---|
DMA alone ( | PASS | "it needs CPU traffic" | "the DMA is broken even alone" |
Another seed (18) | PASS | "it depends on the traffic pattern" | "traffic breaks it often" |
Seed 19, only 20 operations | FAIL | "even a little traffic does it" | "it needs heavy traffic" |
Read the last column. Every result, and its opposite, fits the hypothesis. Three runs later the engineer believes it more and knows nothing new. A hypothesis that forbids no outcome cannot be tested, so it cannot be wrong, so it cannot help. This is the trap guessing vs searching warned about, seen from the other side: the runs were not useless because they were cheap, but because nothing was riding on their results.
On the debug loop this is Hypothesise, and its output is not a suspect but a prediction. The rules:
Name the mechanism. Which signal has which wrong value, and why. "FILL is X because it has no reset, and a zero fill copies it" is a mechanism; "the DMA corrupts data" is a place.
Derive something you have not seen. A cycle, a seed, a value, a count of words. Predicting the failure you already have ("seed 19 will fail at 415") tests nothing: the hypothesis was built to fit it.
Write it down before the run. In the ticket or the debug log. After the run, memory bends towards the result.
Prefer the risky prediction. The best one would be surprising if the hypothesis were false. A precise number beats "it will still fail".
Say what would kill it. If you cannot name a result that proves it wrong, sharpen it until you can.
A failed prediction is progress. It removes a hypothesis or tells you which part of it is wrong. Change the hypothesis only together with a new prediction.
A hypothesis that predicts which seeds fail is easy to kill: one unpredicted X failure, or one predicted seed that passes, and it is wrong. That is what makes the match worth something.
The strong hypothesis, in full: commit 14 took FILL out of the reset block. The bench writes FILL only when its own model of FILL is not already 0, so a zero fill that comes before any write to FILL uses the register's power-up value. In Icarus that is X, and every word of the run is written as X. It came from reading the commit and the bench's zero-fill code, after walking back as in symptom, fault and cause. It predicts three things nobody has looked at yet.
A small probe compiled next to the design lists each DMA run and each write to FILL. It reads the stimulus only; it does not look for X.
// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// run_log: one line per DMA run (its number, mode and length) and one per write to the FILL
// register. It reads the design's ports and changes nothing. +RUN=<n> stops listing after run n.
`timescale 1ns/1ps
module run_log;
int unsigned run = 0, last;
bit fill_written = 0;
initial if (!$value$plusargs("RUN=%d", last)) last = 0; // 0: list every run
always @(negedge dbglab_tb.clk) begin // mid-cycle: values are settled
if (dbglab_tb.dut.u_csr.wr && dbglab_tb.dut.u_csr.off == dbglab_pkg::CSR_FILL
&& (last == 0 || run < last))
$display("cycle %0d: CPU writes FILL = %h", dbglab_tb.cycle, dbglab_tb.dut.u_csr.wdata);
if (dbglab_tb.dut.u_csr.wr && dbglab_tb.dut.u_csr.off == dbglab_pkg::CSR_FILL) fill_written = 1;
if (dbglab_tb.dut.u_dma.start && !dbglab_tb.dut.u_dma.busy) begin
run++;
if (last == 0 || run <= last)
$display("cycle %0d: DMA run #%0d starts: %0s LEN=%0d%0s", dbglab_tb.cycle, run,
dbglab_tb.dut.u_dma.cfg_ctrl[dbglab_pkg::CTRL_FILL] ? "fill" :
dbglab_tb.dut.u_dma.cfg_ctrl[dbglab_pkg::CTRL_PACED] ? "copy to peripheral" : "copy",
dbglab_tb.dut.u_dma.cfg_len,
dbglab_tb.dut.u_dma.cfg_ctrl[dbglab_pkg::CTRL_FILL] && !fill_written ? " (FILL never written)" :
dbglab_tb.dut.u_dma.cfg_len > 255 ? " (longer than 255)" : "");
end
end
endmodule$ sh ../probes/run-probe.sh ../probes/run_log.sv random 19 +RUN=3
dbglab: test=random seed=19
cycle 49: DMA run #1 starts: copy to peripheral LEN=5
cycle 128: DMA run #2 starts: copy LEN=16
cycle 283: DMA run #3 starts: fill LEN=23 (FILL never written)
FAIL test=random seed=19 cycle=415: SCOREBOARD: RAM word 984 (address 00001f60) read xxxxxxxx, expected 00000000 (last written by DMA run #3)Run #3 is a fill of 23 words, and no FILL write appears before it. Had the probe shown a copy, or a FILL write at cycle 200, the hypothesis would be dead.
If the mechanism is right, the stimulus alone tells you which seeds fail with X: exactly those that start a zero fill before any FILL write. Written in the ticket before the sweep: the X failures among seeds 1 to 60 are exactly the seeds the probe flags, and no others.
$ for s in $(seq 1 60); do sh ../probes/run-probe.sh ../probes/run_log.sv random $s | grep -q "FILL never written" && printf "%s " $s; done; echo
16 17 19 23 29 31 41 43 51
$ for s in $(seq 1 60); do tb/run.sh random $s | grep FAIL; done | grep xxxxxxxx | sed 's/.*seed=\([0-9]*\).*/\1/' | tr '\n' ' '; echo
16 17 19 23 29 31 41 43 51Nine predicted, nine failed with X, no X failure missed. The ten other failing seeds read a wrong known value; this hypothesis says nothing about them, and they are a different bug (the keeping several hypotheses alive lesson separates the two).
Two-state Verilator has no X: its FILL starts at 0, which is what the test expects, so it passes seed 19. If FILL's start value is the only thing wrong, then giving FILL a reset to 0 should make Icarus pass seed 19 in exactly the same number of cycles as two-state Verilator does today. Not roughly: the same number. Any other difference between the runs would show up as a different count.
$ git diff --stat # a copy of HEAD with FILL moved into the reset block
rtl/csr_block.sv | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
$ tb/run.sh random 19 # the copy, Icarus
dbglab: test=random seed=19
PASS test=random seed=19 cycles=16990 time=169896 ns
$ tb/run.sh random 19 verilator # HEAD, unchanged, two-state Verilator
dbglab: test=random seed=19
PASS test=random seed=19 cycles=16990 time=169896 ns16990 and 16990. Seed 17 gives the same kind of match (18442 in both). This is the riskiest of the three predictions, because it would catch any second difference, and it held.
The strong hypothesis made three predictions that could each have failed, and none did: a fact about run #3, a list of nine seeds out of sixty, and a cycle count shared by two simulators. That is far more convincing than a dozen runs that "fit", and it took five commands. The weak hypothesis made none, so its three runs, though real, moved nothing.
What it does not prove: that FILL is the only unreset register that matters, or anything about the ten wrong-value failures. Next on the loop is Test in earnest: the experiment that splits suspects when there are several.
Name what congestion would do and where: for example, "when the CPU holds the bus during a DMA write, the write data register is overwritten, so the word written is the CPU's data". That predicts the wrong word equals a value the CPU wrote nearby in time, and that runs with no CPU traffic (dma_copy) never fail. Here the read value is X, which no CPU write can produce, so the sharpened version dies on the first look. That is a good outcome: it took one minute.
That FILL's start value is not the only difference between the two runs. The fix may still be right, but something else (another unreset register, or a difference in how the simulators order events) changes the timing. You would not close the bug: you would find the first cycle where the two runs differ, as in comparing a good and a bad run.
dbglab at HEAD 49c5f04 (rebuilt with dbglab/build-history.sh), Icarus Verilog 13.0 and Verilator 5.048, 2026-10-17. Weak hypothesis runs: tb/run.sh dma_copy 19 (PASS), tb/run.sh random 18 (PASS), tb/run.sh random 19 icarus +OPS=20 (FAIL at cycle 400, final sweep); +OPS=30, 300 and 3000 fail at cycle 415 like the full run.
Prediction 1: sh ../probes/run-probe.sh ../probes/run_log.sv random 19 +RUN=3 (the probe shown, compiled next to dbglab with iverilog -g2012 -Wno-timescale -I tb <rtl/files.f> tb/bus_rules.sv tb/dbglab_tb.sv run_log.sv). Prediction 2: the same probe over seeds 1 to 60 flags 16 17 19 23 29 31 41 43 51; tb/run.sh random <seed> over 1 to 60 fails with xxxxxxxx in exactly those nine. Prediction 3: on a copy of HEAD with FILL moved into the reset block, tb/run.sh random 19 passes with cycles=16990, as does tb/run.sh random 19 verilator at HEAD; seed 17 gives 18442 in both.
Limits: one bug, one design, simulation only. The predictions confirm the mechanism for these 60 seeds; they do not show the design has no other X source, and the ten failures with a known wrong value are a different bug.