"Covered by seed 17"

The X bug in dbglab, B4, is fixed: commit 14 had moved the FILL register out of the reset block, and the fix puts it back. The ticket's last line says Regression: added random seed 17 to the nightly list. Seed 17 failed before the fix and passes after it. Does that pin the bug? If someone removes the reset again next year, will the nightly run go red?

What people usually do, and why it fails

Adding the failing seed to the regression is the most common way to "pin" a bug, because it costs one line. It pins the seed, not the bug. Seed 17 reaches B4 only because, in this version of the random test, a zero fill happens to come before any write to FILL. Change the random test, and the seed draws something else. Run it in a simulator that cannot see the fault, and it passes on the broken design.

The other common test is named after the ticket, test_bug_1234, checks the exact symptom from the report and is never run on the commit that caused the bug. Nobody knows whether it can fail. A pin you have never seen fail is a hope.

The method: a test that states the condition

1. State the bug's condition directly. B4 needs a fill that starts before any write to FILL. A directed test does exactly that, first thing after reset, instead of waiting for random stimulus to stumble into it.

2. Check the fault, then what the user sees. Read the register (the first wrong value), then do what software relies on (a zero fill) and check the memory. The first check names the cause; the second keeps the test meaningful if the cause moves.

3. Prove the pin at three commits. Parent: pass. Guilty commit: fail, for this bug's reason. Fix: pass. Better, run it along the history: it should flip exactly once, at the guilty commit.

4. Run it where it can fail. An X only exists in a four-state simulator; in a two-state one the register starts at 0. Put the pin in a run that can see the fault: Icarus, or Verilator with random start values.

5. Name it for the rule, keep it cheap, keep it on. fill_after_reset, not bug_b4. A few hundred cycles, in the default test list, with the commit and ticket in a comment.

Do it: write the pin, then run it through history

The test

1
2
3
4
5
6
7
8
9
10
11
12
13

// not-run: a task for dbglab's bench (tb/tests.svh); it needs the bench's helpers
// fill_after_reset: FILL resets to 0 (register map), so a fill that starts before any write to
// FILL must write zeros. Pins b357bd8 (B4: FILL lost its reset). Directed: no seed needed.
task automatic t_fill_after_reset();
  logic [31:0] st;
  rd_expect(CSR_BASE + CSR_FILL, 32'h0, "FILL after reset");      // the register itself
  csr_wr(CSR_CTRL, 32'h1 << CTRL_FILL);                            // what software relies on
  dma_program(32'h0, RAM_BASE + 4 * 100, 16);
  dma_wait(st);
  dma_close(st, 1'b0, 32'h0);
  model_copy(0, 100, 16, 1'b1);
  for (int i = 0; i < 16; i++) ram_check(100 + i);
endtask

It is added to tb/tests.svh with one dispatch line in tb/dbglab_tb.sv. Nothing in it is random; the seed only changes the RAM's preload. It runs in 154 cycles.

Part 8's ordered test already reproduces B4 every time, but it fails at the symptom (cycle 79, a RAM word). fill_after_reset reads the register first and fails at the fault, at cycle 10, which is what a pin should do.

Parent, guilty commit, fix

$ c13/tb/run.sh fill_after_reset 1
dbglab: test=fill_after_reset seed=1
PASS test=fill_after_reset seed=1
$ c14/tb/run.sh fill_after_reset 1
dbglab: test=fill_after_reset seed=1
FAIL test=fill_after_reset seed=1 cycle=10: FILL after reset: read 00002024 = xxxxxxxx, expected 00000000
$ c14fix/tb/run.sh fill_after_reset 1
dbglab: test=fill_after_reset seed=1
PASS test=fill_after_reset seed=1

Commit 13 passes, commit 14 fails at cycle 10 on the register read, before the fill has even started, and the fix passes. Cycle 10 against cycle 7949 for the random seed: the pin fails at the fault, not at the symptom.

Where it can fail

$ c14/tb/run.sh fill_after_reset 1 verilator
dbglab: test=fill_after_reset seed=1
PASS test=fill_after_reset seed=1
$ XRAND=1 c14/tb/run.sh fill_after_reset 1 verilator
dbglab: test=fill_after_reset seed=1
FAIL test=fill_after_reset seed=1 cycle=10: FILL after reset: read 00002024 = 8f27a164, expected 00000000

In plain Verilator the test passes on the buggy commit: two-state flip-flops start at 0, which is exactly the value the register map promises, so the missing reset is invisible. With XRAND=1 every unreset flip-flop starts at a random value, as silicon does, and the test fails with 8f27a164. A pin for an X or reset bug belongs in the four-state run or the random-start run, never only in the default two-state one. (X and reset bugs explains why the simulators differ.)

Along the history

Commit 12 introduced fill mode, so the test can run from there. It flips exactly once, at the guilty commit, and stays failing through two later commits that changed other things. A test with that shape is also a perfect git bisect run script, as bisection in history shows.

The seeded pin, and how it goes blind

On this history seed 17 looks just as good: the bottom row has the same shape. The difference shows when the stimulus changes. Part 8 showed that writing FILL first turns the random seeds green. Here is what that does to each pin:

$ diff c14/tb/tests.svh c14init/tb/tests.svh
245a246
>   csr_wr(CSR_FILL, 32'h0);                                       // start from a known FILL
$ c14init/tb/run.sh random 17
dbglab: test=random seed=17
PASS test=random seed=17
$ c14init/tb/run.sh fill_after_reset 1
dbglab: test=fill_after_reset seed=1
FAIL test=fill_after_reset seed=1 cycle=10: FILL after reset: read 00002024 = xxxxxxxx, expected 00000000

This is still commit 14, with the bug in it. Seed 17 now passes, and so does every other seed: across seeds 1 to 60 the X failures go from 9 to 0. The random test can no longer reach the condition at all, and nothing tells you. The directed test fails exactly as before, because it creates the condition itself.

Not every change kills a seed. Adding an ID read to two of the random test's hundred operation codes kept all nine X seeds failing, but moved seed 17's failure from cycle 7949 to 8117. You cannot tell in advance which changes a seed survives, which is the point: determinism and seeds makes a seed replayable, not stable across versions of the test.

Keep it in the regression

$ diff c13/tb/regress.sh c14/tb/regress.sh
8c8
< TESTS="smoke dma_copy irq_w1c periph_stream dma_burst_backpressure periph_power random"
---
> TESTS="smoke dma_copy irq_w1c periph_stream dma_burst_backpressure periph_power random fill_after_reset"
$ c14/tb/regress.sh icarus 1 2 3 | grep -E "fill_after|failing"
fill_after_reset         seed 1      FAIL
fill_after_reset         seed 2      FAIL
fill_after_reset         seed 3      FAIL
9 failing run(s)
$ c14fix/tb/regress.sh icarus 1 2 3 | grep -E "fill_after|failing"
fill_after_reset         seed 1      PASS
fill_after_reset         seed 2      PASS
fill_after_reset         seed 3      PASS
6 failing run(s)

One word in the test list. On the buggy commit the regression has three more failures, on the fix it has none more (the other six are two other known bugs, B1 in irq_w1c and B2 in dma_burst_backpressure). In the default Verilator run the same list reports 6 failing runs at commit 14: the pin is silent there, as above. XRAND=1 tb/regress.sh verilator 1 2 3 reports 9.

What the evidence says

Keep seed 17 too if you like; it costs little and catches other things. But it is a sample, not a pin. The pin is the test that makes the condition happen on purpose.

Naming and keeping

• Name the rule, not the ticket. fill_after_reset tells a reader in two years what must hold. bug_b4 or seed17_regress tells them nothing, and gets deleted in the first clean-up.

• Put the history in the comment: the guilty commit and the ticket, so a failure points straight at the old analysis.

• Make the failure message a report. FILL after reset: read 00002024 = xxxxxxxx, expected 00000000 names the register, the value and the rule. A pin that prints "test failed" sends the next person back to the start of the loop.

• Keep the precondition in the test. If the condition can stop being reached (as the random test stopped reaching it), assert that it was reached: here the test creates it on its first line, so it cannot drift.

Traps

Related topics

• Part 9 · Fix, Prove, Prevent

• Next: the assertion that would have caught it

• X and reset bugs (a reset-value check for every register)

• Symptom, fault, cause (the same bug, walked back)

• Escape analysis: why the tests missed it