Row S5 of Monday's triage is the unknown value read from RAM in dbglab's random test. A colleague filing it writes: severity low. Three seeds of twenty, Icarus only, Verilator is clean. Probably X pessimism. The tracker sorts by severity, so the bug drops below a typo in a register description.
Is that right? And what should this record contain, so that whoever reads it in six months knows what it was, whether it was really fixed, and why nothing caught it sooner?
Most trackers start as a list with a title, an owner and a free-text priority. Three things go wrong as the list grows. One field does three jobs: "priority" is used to mean how bad it is, how often it happens and what to do next, so a frequent cosmetic failure outranks a rare corruption. Duplicates multiply: without a signature field, people search titles, and two titles for the same failure never match. States lie: "fixed" is set when the change merges, before anyone reruns the repro at the fix, and the regression that would have shown it still failing runs the default seeds that never saw it.
The bug report itself is covered in a bug report others can act on: symptom title, repro, expected versus actual, what was ruled out. This lesson is about the record around it: the fields a team sorts, searches and measures, and the states a bug moves through.
Field | What it holds | Who uses it |
|---|---|---|
Signature | The normalised failure message, one per simulator | triage, for finding duplicates |
Found by, stage | Test and seed, and the stage (RTL simulation, gate level, emulation, silicon) | metrics, escape analysis |
Introduced in | First bad commit, with the bisect log | the owner, the release manager |
Affected | Configurations, modes and versions that can reach it | severity, software teams |
Severity | Harm if it ships (scale below) | everyone; set at triage |
Frequency | n of m seeds, per simulator; how rare the path is | the owner, the proof |
Priority | Order of work, decided at triage | the owner's queue |
Root-cause class | The bug class from Part 5 | metrics, reviews |
Workaround | What software or a mode can do instead, and whether it was verified | software, the release decision |
Fixed in, verified by | The fix commit; the commands and sweeps that proved it | whoever closes it |
Could have been found | The earliest check or stage that would have caught it | escape analysis (Part 9) |
Severity asks one question: if this shipped in silicon, what is the worst a user who follows the documentation could see? A scale many hardware teams use looks like this (the exact words vary; these are illustrative):
Severity | Meaning | Typical examples |
|---|---|---|
S1 critical | Silent wrong data, a hang, or damage, with no acceptable workaround; would justify a respin | corrupted DMA copy, deadlock under legal traffic, an unclamped domain output |
S2 major | Wrong behaviour with a workaround that costs performance, power or software effort | an interrupt that must be masked in software, a mode that must be avoided |
S3 minor | Wrong only in a corner or a debug mode, with an easy workaround | a status bit wrong during reset, a debug counter that wraps |
S4 cosmetic | No effect on behaviour | a register name, a comment, a description |
Frequency is not severity. A test seeing it once in 500 runs says how hard it is to reproduce. In silicon, running billions of operations, a rare path is a daily event.
Priority is a decision, made in triage from severity, the schedule and who is free. It can change weekly; severity changes only when the facts do.
Accepted needs the owner's own repro run (see ownership and hand-offs).
Root-caused needs a hypothesis that predicted something and was tested (a hypothesis must predict).
Verified needs the original repro and a wider sweep passing at the fix commit, in every simulator the bug was seen in.
Duplicate needs a shared root cause, not a shared message. Cannot reproduce needs the replay record that was tried.
Start with facts for the frequency and severity fields: the random test over sixty seeds, in each simulator and mode the team has.
$ for s in $(seq 1 60); do tb/run.sh random $s | grep FAIL; done > sweep-icarus.txt
$ wc -l < sweep-icarus.txt; grep -c xxxxxxxx sweep-icarus.txt
19
9
$ for s in $(seq 1 60); do tb/run.sh random $s verilator | grep FAIL; done > sweep-ver.txt
$ wc -l < sweep-ver.txt; grep -c xxxxxxxx sweep-ver.txt
12
0
$ for s in $(seq 1 60); do XRAND=1 tb/run.sh random $s verilator | grep FAIL; done > sweep-verx.txt
$ wc -l < sweep-verx.txt
19
$ grep -E 'seed=(17|19) ' sweep-icarus.txt sweep-verx.txt
sweep-icarus.txt:FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read xxxxxxxx, expected 00000000 (last written by DMA run #4)
sweep-icarus.txt:FAIL test=random seed=19 cycle=415: SCOREBOARD: RAM word 984 (address 00001f60) read xxxxxxxx, expected 00000000 (last written by DMA run #3)
sweep-verx.txt:FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)
sweep-verx.txt:FAIL test=random seed=19 cycle=415: SCOREBOARD: RAM word 984 (address 00001f60) read 4443c624, expected 00000000 (last written by DMA run #3)Read it as a severity argument. Icarus shows the X on nine seeds of sixty. Verilator's default run shows none, because its two-state flops start at 0, which happens to be the value the test expects. With XRAND=1, which starts every unreset flop at a random value, as silicon does, Verilator fails on the same nine seeds at the same cycles, with random data instead of X. "Icarus only, probably pessimism" is wrong: the bug is real in both simulators, and one of them hides it by default.
The bisect for the repro points at b357bd8, csr: no reset on the DMA address and data registers. The FILL register lost its reset, and the zero-fill path relies on FILL's documented reset value of 0. On a chip, a DMA zero-fill issued before software has written FILL writes whatever FILL powered up with: silent wrong data in memory that software believes is zeroed, with no error and no interrupt. That is S1 by the scale above, however rare the path is in the random test.
The owner's fix, in a scratch copy of the tree, gives FILL its reset back:
$ git diff
...
if (!rst_n) begin
scratch_q <= SCRATCH_RST; ctrl_q <= CTRL_RST; done_q <= 1'b0;
- err_q <= 1'b0; err_addr_q <= '0;
+ err_q <= 1'b0; err_addr_q <= '0; fill_q <= FILL_RST;
...
CSR_CTRL: ctrl_q <= wdata[7:0];
+ CSR_FILL: fill_q <= wdata;
...
CSR_LEN: len_q <= wdata[9:0];
- CSR_FILL: fill_q <= wdata;
default: ;$ tb/run.sh random 19
dbglab: test=random seed=19
PASS test=random seed=19 cycles=16990 time=169896 ns
$ for s in $(seq 1 60); do tb/run.sh random $s | grep FAIL; done > sweep-b4fix.txt
$ wc -l < sweep-b4fix.txt; grep -c xxxxxxxx sweep-b4fix.txt
12
0
$ for s in $(seq 1 60); do XRAND=1 tb/run.sh random $s verilator | grep FAIL; done | wc -l
12
$ grep -E 'seed=(16|51) ' sweep-b4fix.txt
FAIL test=random seed=16 cycle=12811: STREAM: word #203 = 5484683d, expected 3caa023d
FAIL test=random seed=51 cycle=11722: SCOREBOARD: RAM word 322 (address 00001508) read 15d021ca, expected 00000000 (last written by DMA run #75)Every X is gone in Icarus, and the random-start Verilator sweep drops from 19 failures to 12. The twelve left are seeds 10 13 16 20 34 46 48 50 51 52 58 59 in both, exactly the twelve Verilator's default sweep fails at HEAD: the other random-test bug, the beat counter (row S4). Two of them, 16 and 51, were in this bug's list too: they carried both bugs, and with this one fixed they now fail later, on the other one. In the tracker that is a related link to S4, not a duplicate.
Field | Value |
|---|---|
Title | random: RAM reads unknown or random data after a DMA zero-fill that precedes any FILL write |
Signature | Icarus: |
Repro | dbglab 49c5f04, |
Found by, stage | Nightly |
Introduced in | b357bd8 csr: no reset on the DMA address and data registers (bisect log attached) |
Affected | Any DMA fill run (CTRL.fill set) that starts before software's first FILL write after reset |
Severity | S1: silent wrong data in silicon, no error raised |
Frequency | 9 of 60 seeds (Icarus and XRAND Verilator); 0 of 60 in default Verilator. About 1 DMA run in 100 is a zero-fill |
Priority | P1 (set at triage on Monday; reviewed weekly) |
Owner, state | CSR owner; Verified |
Root-cause class | X and reset: register without reset relied on for a documented value (X and reset bugs) |
Workaround | Software writes FILL once after reset. Proposed, not verified |
Fixed in, verified by | Reset restored for FILL (diff attached). Seeds 17 and 19 pass; 60-seed sweeps: 0 X in Icarus, XRAND Verilator failures 19 to 12, the 12 left are S4's seeds |
Related | S4 (beat counter wrap): seeds 16 and 51 carry both bugs |
Could have been found | A lint or review rule on registers without reset that feed datapath; a write-data X check on the bus (X in a waveform) |
Three sweeps set two fields a hurried filer would have got wrong: frequency per simulator instead of "Icarus only", and severity from what the design does on a chip, not from which tool complained. Repeated at the fix, the same sweeps are the verification: the bug's signature is gone everywhere it was seen, and the failures left belong, seed for seed, to a different ticket.
Compare the signature, not the verdict: seed 16 now fails on the stream (STREAM: word #203 ...), not with an X read, at a different cycle. That is the beat-counter bug, which the same seed also carries. The record stays verified, gains a related link to S4, and notes that seeds 16 and 51 carry both bugs.
The replay record that was tried: commit, command, seed, simulator and version, environment and local edits, from the original failure, and the result of replaying it in a fresh tree. If the replay differs from the original run, an input is missing and the bug is not "cannot reproduce" yet; it is "cannot replay", which is a fault of the record. See recording everything to replay.
dbglab at HEAD 49c5f04 (dbglab/build-history.sh), 2026-10-18. for s in $(seq 1 60); do tb/run.sh random $s | grep FAIL; done (Icarus Verilog 13.0): 19 FAIL lines, 9 with an X value (seeds 16 17 19 23 29 31 41 43 51). The same loop with verilator (Verilator 5.048): 12 FAIL lines, none with an X; with XRAND=1 ... verilator: 19, and the nine seeds above fail at the same cycles as in Icarus with random values.
Fix in a scratch copy of the tree: the two-line change to rtl/csr_block.sv shown in git diff. The Icarus sweep then gives 12 FAIL lines and 0 with X; the XRAND=1 Verilator sweep 12; the failing seeds are 10 13 16 20 34 46 48 50 51 52 58 59 in both, the same twelve as Verilator's default sweep at HEAD. tb/run.sh random 17 and random 19 pass. First bad commit b357bd8 from git bisect run tb/run.sh random 19.
Limits: the severity scale, the tracker fields and the record's people are illustrative (scales and field names vary by team). The software workaround in the record is proposed, not run: the bench cannot be changed for it without editing dbglab. A clean 60-seed sweep is evidence about this bug, not proof of its absence (see Part 8 on how many runs to trust).