Three months later, the same cast

A new engineer tidies a timer block. Two counters have different widths, so they give them one width, cast the wider input into it and watch the width warning disappear. Tests pass. It is dbglab's bug B3 again, in a different block, made by someone who joined after B3 was fixed.

B3 was found, fixed and proven. The ticket says fixed, random passes. What would the write-up have needed to say so that this engineer never made the change?

What people usually do, and why it fails

Bugs close in one of two ways. The one-liner, "counter too small, widened, tests pass", records that something happened, not what. The diary lists every hypothesis in the order it was tried, ten pages long; the facts are in there, but nobody rebuilds the story to find them.

Both miss the point. The bug report was written to get the bug fixed. The write-up comes after the proof, for people who were not there: the block's next owner, the reviewer of a similar change, the lead deciding what to verify next.

The method: a template that follows the loop

Each section answers one question a reader will ask, and most of them come straight from a step of the debug loop you already ran:

Rules for filling it in:

1. Summary first, under 80 words. Most readers stop there. It must stand alone.

2. Every claim has its command, copied with the real output, trimmed with .... The commands make the document checkable a year later.

3. Exposure is a list of facts, not an adjective. "Copies of 256 words or more" can be checked against a driver; "long transfers" cannot.

4. Separate what you showed from what you believe. The lengths you tested are facts; the lengths you did not are an inference, labelled.

5. End on one lesson that a reader could act on in code that does not exist yet.

The picture

Do it: the complete write-up for B3

Everything below is the write-up itself, as it would go in the ticket. The hunt for this bug is told step by step in the debug loop; here is only what a reader needs.

Title and summary

DMA copies of 256 words or more go wrong: the beat counter was narrowed to 8 bits (commit 03ebdc1)

Since commit 13 (03ebdc1, 2026-03-14), dma_engine loads the 10-bit LEN register into an 8-bit counter. A copy of 257 to 511 words writes only LEN minus 256 words, and a copy of exactly 256 words writes the right number of words with four of them wrong. Found by the random test, seed 10, in a 60-seed sweep. Fixed by sizing the counter from the port; proven on the failing seed in both simulators, a new length test and a 60-seed sweep.

Exposure

Measured with a length test (dma_len, below) on a scratch copy of the bench, one memory-to-memory copy per run:

$ for L in 255 256 257 300 511; do tb/run.sh dma_len 1 icarus +LEN=$L | grep -v '^dbglab'; done
PASS test=dma_len seed=1 cycles=2837 time=28366 ns
FAIL test=dma_len seed=1 cycle=2843: DMA_LEN: LEN=256 copied only 252 words
FAIL test=dma_len seed=1 cycle=813: DMA_LEN: LEN=257 copied only 1 words
FAIL test=dma_len seed=1 cycle=1283: DMA_LEN: LEN=300 copied only 44 words
FAIL test=dma_len seed=1 cycle=3605: DMA_LEN: LEN=511 copied only 255 words

• Affected: every DMA run with LEN of 256 or more, in every mode, at commits 13 to HEAD. Shown at 256, 257, 300 and 511; at 512 the run also copies only 252 words, like 256. Above that the same wrap is expected (LEN modulo 256, with multiples of 256 behaving like 256), not tested.

• Not affected: LEN of 255 or less, and the v0.1 release (commit 5), which predates the change.

• Silent: the DMA reports done with no error.

How it was found

$ tb/run.sh random 10
dbglab: test=random seed=10
FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)
$ tb/run.sh random 10 verilator
dbglab: test=random seed=10
FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)

Deterministic, identical in both simulators. At commit 13, 12 of seeds 1 to 60 fail this way (10 13 16 20 34 46 48 50 51 52 58 59); the nightly seeds 1 to 3 all pass.

Mechanism

$ ../probes/run-probe.sh ../probes/dma_beats.sv random 10 | grep -E "#29 |FAIL"
DMA run #29 start: LEN=308 left_q=52
DMA run #29 done: 52 write beats, 00001b1c..00001be8
FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)

Software asked for 308 words; the counter of words left, left_q, loaded 52 (308 minus 256) and the run ended after 52 writes. The word read at cycle 4895 was never written. The first wrong value is left_q the cycle after the load.

Root cause

$ git bisect start HEAD v0.1
...
$ git bisect run tb/run.sh random 10
...
03ebdc1fd778231df2f3dca283af1764db17f11e is the first bad commit
...
    dma: one width for the beat counters
    
    Name the counter width once and cast the length into it; this also
    silences the Verilator WIDTH warning on the load.
...
$ git show 03ebdc1
...
-  logic [9:0]  left_q;             // words still to write
+  localparam int CW = 8;           // beat counter width
+  logic [CW-1:0] left_q;           // words still to write
...
-          left_q  <= cfg_len;
+          left_q  <= CW'(cfg_len);
...

Three test runs over eleven commits. The commit was a tidy-up: one width for the counters. The width chosen, 8, is narrower than LEN's 10 bits, and the cast CW'(cfg_len) drops the top two bits. Without the cast, verilator --lint-only reports exactly this: %Warning-WIDTHTRUNC: rtl/dma_engine.sv:89:19: Operator ASSIGNDLY expects 8 bits on the Assign RHS, but Assign RHS's VARREF 'cfg_len' generates 10 bits. The commit message says the cast was added to silence that warning. The warning was right (fix the cause shows what removing the cast does).

Why it escaped

The random test makes a run longer than 255 words in 15 of 7,459 DMA runs, in 12 of 60 seeds, and the nightly regression runs seeds 1, 2 and 3, which contain none, every night. Lint would have caught it, but the cast silenced it and the bench builds with lint warnings off. Shown in escape analysis.

Fix

$ git diff rtl
...
-  localparam int CW = 8;           // beat counter width
+  localparam int CW = $bits(cfg_len); // beat counter width: holds any LEN
...

The counter now takes its width from the port it is loaded from, so the two cannot drift apart again. Rejected: CW = 10 (works, but repeats the number in a second place, which is how this bug started); removing the cast alone (brings the warning back without fixing anything); limiting LEN in software (hides the bug from one driver, not from the next).

Proof

$ tb/run.sh random 10
dbglab: test=random seed=10
PASS test=random seed=10 cycles=21581 time=215806 ns
$ tb/run.sh random 10 verilator
dbglab: test=random seed=10
PASS test=random seed=10 cycles=21581 time=215806 ns
$ for L in 255 256 257 300 511; do tb/run.sh dma_len 1 icarus +LEN=$L | grep -v '^dbglab'; done
PASS test=dma_len seed=1 cycles=2837 time=28366 ns
PASS test=dma_len seed=1 cycles=2847 time=28466 ns
PASS test=dma_len seed=1 cycles=2861 time=28606 ns
PASS test=dma_len seed=1 cycles=3331 time=33306 ns
PASS test=dma_len seed=1 cycles=5653 time=56526 ns
$ for s in $(seq 1 60); do tb/run.sh random $s | grep FAIL; done > ../runs/wu-fix-sweep.txt
$ wc -l < ../runs/wu-fix-sweep.txt; grep -c xxxxxxxx ../runs/wu-fix-sweep.txt
       9
9

The failing seed passes in both simulators, every tested length passes, and the 60-seed sweep has no wrong-value failure left. The 9 remaining failures all read an unknown value: that is B4, a separate bug with its own ticket. The default regression still has 9 failing runs, the same irq_w1c, dma_burst_backpressure and periph_power failures as before (B1, B2, B5), none of them new. How much proof is enough is in proving the fix.

Prevention

The length test, as added to the scratch bench, pins the boundary rather than seed 10:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

// not-run: a task inside dbglab's bench (tb/tests.svh); it uses the bench's run_copy, rd and model
// dma_len: one memory-to-memory copy of +LEN=<n> words from word 0 to word 512;
// prints how many destination words hold the source data, then checks them all.
task automatic t_dma_len();
  int len, ok = 0;
  logic [31:0] got;
  if (!$value$plusargs("LEN=%d", len)) len = 300;
  csr_wr(CSR_CTRL, 32'h0);
  run_copy(0, 512, len);
  for (int i = 0; i < len; i++) begin
    rd(RAM_BASE + 4 * (512 + i), got);
    if (got === m_ram[512 + i]) ok++;
    else if ($test$plusargs("SHOW")) $display("dma_len: word %0d read %h, expected %h", i, got, m_ram[512 + i]);
  end
  $display("dma_len: LEN=%0d words copied=%0d", len, ok);
  if (ok != len) fail($sformatf("DMA_LEN: LEN=%0d copied only %0d words", len, ok));
endtask

• Added: dma_len at LEN 255, 256, 257 and 511 in the regression, a per-length companion to dma_len_limits in fix the cause, which also runs 512 and 1023. Verified: fails at 256, 257 and 511 with the bug, passes with the fix. Why a boundary test beats a seed is in the regression test that pins the bug.

• Added and proven: the load property a_len_load (the counter equals LEN the cycle after start), PASSED by induction in proving the fix.

• Proposed: a coverage goal on LEN at 255, 256, 257 and 1023; fresh seeds every night.

• Proposed: lint in the regression with no new warnings; a narrowing cast needs a comment stating the range it assumes.

Lesson for the team

The weak version, for contrast

Read it as the engineer in the scene would, three months later:

• "Too small" for what? No LEN, no 256, so nobody can check their own driver or counter.

• "Random passes" with which seed? The nightly seeds passed before the fix too.

• CW = 10 repeats the width in a second place, the pattern that made the bug.

• No exposure, so the team cannot tell whether software already shipped long copies.

• No escape and no prevention, so the gap that hid it for days is still open.

• No lesson, so the cast in the timer block looks just as harmless as it did here.

Both came from the same debugging. The difference is under an hour of writing, mostly pasting commands already in the debug log; link that log and the bisect log rather than pasting the hunt.

What the evidence says

Every sentence in the good write-up can be checked by running something on the page, and the one that cannot (lengths above 512) says so. That keeps it trustworthy a year later, when the author has moved on. The lesson line is short so it survives being quoted in a review comment.

Traps

Related topics

• Part 9 · Fix, Prove, Prevent

• A bug report others can act on

• The debug loop (the hunt for this bug)

• Logic bugs: off-by-one, width and sign

• Part 11: debug as a team