The nightly run of dbglab has one new red line: random, seed 23, a stream word that came out as xxxxxxxx. You have an hour before the triage meeting, where someone will ask whether this is real, whether it is new, and how anyone else can see it. Which checks do you run in that hour, and how do you know when Reproduce is done?
Most people rerun the failing command once, see it fail, and start reading waveforms. That skips most of what makes a repro useful. Nobody wrote down the commit or the tool, so a colleague who tries it on another simulator sees a pass and calls it a ghost. Nobody checked the commit before, so nobody knows if the bug is new. Nobody measured how often it fails, so later nobody can say whether a fix worked. Each gap costs a meeting.
Work down the table. Each row has a check, what finishes it, and the lesson that teaches it. Most rows take a minute; skip one only if you can say why it does not apply.
# | Check | Done when | Taught in |
|---|---|---|---|
1 | Write the failure as expected versus actual | the exact FAIL line, and its signature: what is wrong, where, which kind of message | |
2 | Run the same command twice | the two logs are identical, not just the last line | |
3 | Record what decides the run | commit and local edits, command, seed and plusargs, simulator and version, options and environment are saved | |
4 | Try the other simulator, and its options | you know whether the tool or an option is part of the repro | |
5 | Run the commit before | you know whether the bug is new, and have a good and a bad end | |
6 | Measure the rate if some seeds pass | a rate by signature over hundreds of seeds, and a stressed setting that fails most of them | |
7 | Check that the run ends | it fails with a FAIL line in bounded cycles, not a farm kill; a hang gives a watchdog message | |
8 | Shrink the stimulus | the smallest stimulus that still fails with the same signature | |
9 | Shrink the design, if it is large | the smallest design that still fails with the same signature | |
10 | If only the lab sees it | the differences between lab and bench are listed, and a model of the mechanism reproduces the symptom |
The output of the whole table is one line in the bug report: the command that fails, at a named commit, with a named tool, with its FAIL line. Everything else in this course starts from that line.
$ tb/run.sh random 23
dbglab: test=random seed=23
FAIL test=random seed=23 cycle=2385: STREAM: word #41 = xxxxxxxx, expected 00000000
$ cp build/icarus-random-23.log ../first.log
$ tb/run.sh random 23 > /dev/null
$ cmp build/icarus-random-23.log ../first.log && echo "logs identical"
logs identicalThe signature: the peripheral's output stream, word 41, is unknown where zero was expected. Not a RAM read, not a hang. The full log is identical on the second run, so the run is deterministic.
$ git log -1 --format='%h %s'
49c5f04 tb: report cycles and sim time in the PASS line
$ git status --short | wc -l
0
$ iverilog -V 2>&1 | head -1
Icarus Verilog version 13.0 (stable) (v13_0)Commit 49c5f04, no local edits, Icarus 13.0, tb/run.sh random 23 with no plusargs and no environment switches. That is the record.
$ tb/run.sh random 23 verilator
dbglab: test=random seed=23
PASS test=random seed=23 cycles=19235 time=192346 ns
$ XRAND=1 tb/run.sh random 23 verilator
dbglab: test=random seed=23
FAIL test=random seed=23 cycle=2385: STREAM: word #41 = f0db2f1b, expected 00000000Verilator passes by default and fails with XRAND=1, at the same cycle and word, with a random value instead of X. This row just paid for itself: the simulator and its options are part of the repro, and the pass is a strong hint that the value depends on how a simulator starts up state that has no reset. Write it in the report so nobody closes the bug from a Verilator run.
$ git checkout -q 03ebdc1 && tb/run.sh random 23 # commit 13
dbglab: test=random seed=23
PASS test=random seed=23
$ git checkout -q b357bd8 && tb/run.sh random 23 # commit 14
dbglab: test=random seed=23
FAIL test=random seed=23 cycle=2385: STREAM: word #41 = xxxxxxxx, expected 00000000New in commit 14, with the same line. You now have a good end and a bad end one commit apart, which is the cheapest localisation there is.
Seed 23 fails every time, but most seeds pass. The flaky test lesson measured this signature, an unknown value where zero was expected, at 100 failures in 1,000 seeds (measured with the counter bug fixed), and a biased setting that fails 177 seeds in 200. Row 7 is already true: the run stops with a FAIL line at cycle 2385, and /usr/bin/time -p says it takes 0.06 s.
$ tb/run.sh random 23 icarus +OPS=185
dbglab: test=random seed=23
FAIL test=random seed=23 cycle=2385: STREAM: word #41 = xxxxxxxx, expected 00000000
$ tb/run.sh random 23 icarus +OPS=184
dbglab: test=random seed=23
FAIL test=random seed=23 cycle=2341: SCOREBOARD (final sweep): RAM word 48 (address 000010c0) holds xxxxxxxx, expected 00000000 (last written by DMA run #16)185 operations instead of 1,500 still fail with the same line. At 184 the run still fails, but with a different message: an unknown RAM word found by the final sweep. It is probably the same bug seen from another angle, but "probably" is not a signature check, so the repro stays at 185. The shrinking lesson goes much further, automatically.
dbglab is small enough that row 9 can wait, and row 10 does not apply: the failure is in simulation. The finished line for the report:
random seed 23 at 49c5f04 (new in b357bd8), Icarus 13.0: tb/run.sh random 23 icarus +OPS=185
FAIL test=random seed=23 cycle=2385: STREAM: word #41 = xxxxxxxx, expected 00000000
Verilator 5.048 passes unless XRAND=1. Signature rate about 10% of seeds (1,000-seed sweep with the counter bug fixed).In under an hour the failure went from a red line to a repro that anyone can run, plus three facts that steer the next steps: it is new in commit 14, it depends on how a simulator initialises unreset state, and it fails one seed in ten. You have not looked at a waveform yet, and you already know where to look first. That is the point of finishing Reproduce before starting Observe.
Row 4 done halfway: they tried the other simulator but not its options. With XRAND=1, Verilator fails at the same cycle and word with a random value. The pass came from Verilator starting unreset flip-flops at zero, which happens to be the expected value here. Silicon gives no such promise, so the bug is real; it is a design bug that one tool's defaults hide.
+OPS=170, seed 23 fails at cycle 2302 with an X in RAM word 48, found by the final sweep. A colleague says 170 is a better repro than 185 because it is smaller. Is it?Not until you show it is the same bug. The message, the place and the cycle all changed, so by the signature rule it is a different failure. It may well have the same cause, an X written by a zero fill, but you would be assuming the answer. Keep 185, which fails with the original line, and note the other failure as a lead for Observe.
dbglab at HEAD 49c5f04 (rebuilt with dbglab/build-history.sh), Icarus Verilog 13.0 and Verilator 5.048, 2026-10-17. tb/run.sh random 23 twice: the two logs compare equal (cmp); /usr/bin/time -p tb/run.sh random 23: real 0.06 s. tb/run.sh random 23 verilator: PASS (cycles=19235); XRAND=1 tb/run.sh random 23 verilator: FAIL at cycle 2385 with f0db2f1b.
git checkout -q 03ebdc1 && tb/run.sh random 23 (commit 13) passes; at b357bd8 (commit 14) it fails with the same line. A bisection of +OPS between 100 (pass) and 200 (same failure) found 185 as the smallest value with the same STREAM line; +OPS=184 fails with a different message (final sweep, RAM word 48, DMA run #16). +OPS=150 passes and +OPS=170 fails at cycle 2302 on RAM word 48. The failure rate of this signature, 100 in 1,000 seeds, is from the flaky-test lesson's sweep.
Limits: one failure on one small design. The checklist is a minimum, not a guarantee; a failure that needs the lab goes through the last row and the bring-up course.