The report

Monday morning, two tickets from two nightly regressions of dbglab, filed by two people who have not spoken to each other:

TICKET A (Icarus nightly)
$ tb/run.sh periph_power 1
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown

TICKET B (Verilator nightly)
$ tb/run.sh periph_power 1 verilator
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008

Ticket A has a label, X on the stream pins, probably X pessimism. Ticket B has another, hang in the peripheral, Verilator only. Both are the same test, periph_power, which powers the stream sink down and up with words waiting in its FIFO. This is the whole test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// snippet: tb/tests.svh lines 151-169 at HEAD
task automatic t_periph_power();
  logic [7:0] EN, ISO, PWR;
  EN  = 8'h1 << PCTRL_SINK_EN;
  ISO = 8'h1 << PCTRL_ISO_EN;
  PWR = 8'h1 << PCTRL_PWR_ON;
  for (int i = 0; i < 4; i++) begin per_wait_space(); per_push(rnd()); end
  stream_drain(5000);
  per_ctrl_wr(PWR);                 // sink off
  repeat (10) tick();
  per_ctrl_wr(PWR | ISO);           // isolate
  per_ctrl_wr(ISO);                 // power off
  for (int i = 0; i < 6; i++) per_push(rnd());
  repeat (300) tick();
  per_ctrl_wr(PWR | ISO);           // power on, still isolated
  repeat (20) tick();
  per_ctrl_wr(PWR);                 // release isolation
  per_ctrl_wr(PWR | EN);            // sink on
  stream_drain(5000);
endtask

Four words go through with the power on, then six words are written while the sink is off, isolated and unpowered, and must all come out after it wakes. The test passed last week in both simulators.

Stop here and try it. You have the design (dbglab, with the power-intent lesson for the supply model) and the two lines above. The question is not only what is wrong but is this one bug or two? If it is one, why is one symptom an X at cycle 431 and the other a hang 2,000 cycles later?

The hunt

Reproduce: deterministic, and the seed does not matter

$ for s in 2 3; do tb/run.sh periph_power $s; tb/run.sh periph_power $s verilator; done
dbglab: test=periph_power seed=2
FAIL test=periph_power seed=2 cycle=431: STREAM: pstream_valid is unknown
dbglab: test=periph_power seed=2
FAIL test=periph_power seed=2 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008
dbglab: test=periph_power seed=3
FAIL test=periph_power seed=3 cycle=431: STREAM: pstream_valid is unknown
dbglab: test=periph_power seed=3
FAIL test=periph_power seed=3 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008

Each simulator fails the same way, at the same cycle, on every seed. In this test the seed only picks the data words; the power sequence never moves. So each ticket is perfectly reproducible, and the difference between them is the simulator, nothing else. That is the first piece of evidence that these could be one bug: two symptoms that split cleanly on one dimension usually come from one thing that dimension changes (bisection across configurations has the full matrix).

Observe: one probe, run in both simulators

Two waveforms from two simulators are hard to compare by eye. A probe that prints events is easy to compare: power and isolation changes, words pushed into the async FIFO, words popped from it, and anything on the stream pins. The same source runs in both simulators. Icarus compiles it as a second top-level module; Verilator needs it inside the bench, so a one-line file binds it in: bind dbglab_tb pd_timeline u_timeline();.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// not-run: needs dbglab's sources and bench; compile it next to them (commands in the Evidence box)
// pd_timeline: a read-only probe for periph_power. One line per event in the peripheral:
// a power or isolation change, a word pushed into the async FIFO, a word popped from it
// (or the first of a run of unknown pops), and each word or unknown value on the stream pins.
`timescale 1ns/1ps
module pd_timeline;
  logic [2:0] ctl_q = 3'b000;
  logic       pop, pop_q = 1'b0, vld_q = 1'b0;
  logic [4:0] wbin_q = '0;
  always @(posedge dbglab_tb.clk) begin
    #0.1;
    if (dbglab_tb.dut.u_per.u_fifo.wbin !== wbin_q)
      $display("cycle %4d  push   word into FIFO slot %0d", dbglab_tb.cycle, wbin_q);
    wbin_q = dbglab_tb.dut.u_per.u_fifo.wbin;
  end
  always @(negedge dbglab_tb.pclk) begin                  // mid pclk cycle: inputs to the next edge are stable
    if ({dbglab_tb.dut.u_per.u_pd.pwr_on, dbglab_tb.dut.u_per.u_pd.iso_en, dbglab_tb.dut.u_per.u_pd.en} !== ctl_q) begin
      ctl_q = {dbglab_tb.dut.u_per.u_pd.pwr_on, dbglab_tb.dut.u_per.u_pd.iso_en, dbglab_tb.dut.u_per.u_pd.en};
      $display("cycle %4d  power  pwr_on=%b iso_en=%b sink_en=%b", dbglab_tb.cycle, ctl_q[2], ctl_q[1], ctl_q[0]);
    end
    pop = dbglab_tb.dut.u_per.rinc & ~dbglab_tb.dut.u_per.rempty;
    if (pop === 1'b1 || pop !== pop_q)
      $display("cycle %4d  pop=%b  rinc=%b rempty=%b rbin=%0d pwr_on=%b", dbglab_tb.cycle, pop,
               dbglab_tb.dut.u_per.rinc, dbglab_tb.dut.u_per.rempty, dbglab_tb.dut.u_per.u_fifo.rbin,
               dbglab_tb.dut.u_per.u_pd.pwr_on);
    pop_q = pop;
    if (dbglab_tb.pstream_valid === 1'b1 || dbglab_tb.pstream_valid !== vld_q)
      $display("cycle %4d  pins   pstream_valid=%b", dbglab_tb.cycle, dbglab_tb.pstream_valid);
    vld_q = dbglab_tb.pstream_valid;
  end
endmodule

Here are the two runs from the moment the sink is switched off. The first 60 cycles, four words in and out with the power on, are the same in both and are trimmed.

$ sh ../probes/run-probe.sh . ../probes/pd_timeline.sv periph_power 1          # Icarus
dbglab: test=periph_power seed=1
...
cycle   63  power  pwr_on=1 iso_en=0 sink_en=0
cycle   70  power  pwr_on=1 iso_en=1 sink_en=0
cycle   74  power  pwr_on=0 iso_en=1 sink_en=0
cycle   76  push   word into FIFO slot 4
cycle   79  push   word into FIFO slot 5
cycle   82  push   word into FIFO slot 6
cycle   85  push   word into FIFO slot 7
cycle   85  pop=x  rinc=x rempty=0 rbin=4 pwr_on=0
cycle   88  push   word into FIFO slot 8
cycle   91  push   word into FIFO slot 9
cycle  396  power  pwr_on=1 iso_en=1 sink_en=0
cycle  396  pop=0  rinc=0 rempty=x rbin=x pwr_on=1
cycle  418  power  pwr_on=1 iso_en=0 sink_en=0
cycle  425  power  pwr_on=1 iso_en=0 sink_en=1
cycle  425  pop=x  rinc=x rempty=x rbin=x pwr_on=1
cycle  429  pop=0  rinc=0 rempty=x rbin=x pwr_on=1
cycle  429  pins   pstream_valid=x
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown

$ sh ../probes/run-probe-vl.sh . ../probes/pd_timeline.sv ../probes/bind_timeline.sv periph_power 1   # Verilator
dbglab: test=periph_power seed=1
...
cycle   63  power  pwr_on=1 iso_en=0 sink_en=0
cycle   70  power  pwr_on=1 iso_en=1 sink_en=0
cycle   74  power  pwr_on=0 iso_en=1 sink_en=0
cycle   76  push   word into FIFO slot 4
cycle   79  push   word into FIFO slot 5
cycle   82  push   word into FIFO slot 6
cycle   85  push   word into FIFO slot 7
cycle   88  push   word into FIFO slot 8
cycle   91  push   word into FIFO slot 9
cycle  126  pop=1  rinc=1 rempty=0 rbin=4 pwr_on=0
cycle  129  pop=1  rinc=1 rempty=0 rbin=5 pwr_on=0
cycle  133  pop=0  rinc=0 rempty=0 rbin=6 pwr_on=0
cycle  141  pop=1  rinc=1 rempty=0 rbin=6 pwr_on=0
cycle  144  pop=1  rinc=1 rempty=0 rbin=7 pwr_on=0
cycle  148  pop=0  rinc=0 rempty=0 rbin=8 pwr_on=0
cycle  155  pop=1  rinc=1 rempty=0 rbin=8 pwr_on=0
cycle  159  pop=1  rinc=1 rempty=0 rbin=9 pwr_on=0
cycle  163  pop=0  rinc=0 rempty=1 rbin=10 pwr_on=0
cycle  396  power  pwr_on=1 iso_en=1 sink_en=0
cycle  418  power  pwr_on=1 iso_en=0 sink_en=0
cycle  425  power  pwr_on=1 iso_en=0 sink_en=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008

Read the two side by side.

• Cycles 63 to 91, identical. The sink is switched off, isolated at 70 and unpowered at 74, in the documented order. The six words land in FIFO slots 4 to 9. So far the two simulators agree on everything.

• Cycle 85, Icarus. The FIFO is no longer empty (rempty=0: the first word has crossed the pointer synchroniser), and the read increment rinc is X. A pop that is X makes the read pointer X at the next pclk edge (cycle 87). Nothing on the pins shows it; they are clamped.

• Cycles 126 to 159, Verilator. No X exists here, so the same rinc is noise, and six times it is 1 while the FIFO has data. Six real pops, slots 4 to 9, with the supply off. At 163 the FIFO is empty. The words are gone, and again the clamped pins show nothing.

• Cycles 396 to 425, identical again. Power on, isolation released at 418, sink enabled at 425. Now the damage becomes visible: Icarus's sink reads an X rempty, and at 429 the pins carry X. Verilator's sink finds an empty FIFO and sends nothing at all.

One question is already answered. Both simulators show the FIFO's read side changing while the domain that drives it has no power. That is not two bugs; it is one wrong thing seen through two models of an unpowered wire.

Localise: two cycle numbers, one moment

Why does one ticket say 431 and the other 2421? The watchdog's message gives the cycle it gave up, not the cycle anything happened. It fires when nothing has moved for 2,000 cycles, so the interesting cycle is 2,000 earlier. A two-line probe asks the bench directly, in Verilator, at cycle 600:

$ sh ../probes/run-probe-vl.sh . ../probes/lastprog.sv ../probes/bind_lastprog.sv periph_power 1
dbglab: test=periph_power seed=1
cycle 600: last progress at cycle 420, stream words 4, words still expected 6
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008

Last progress at cycle 420 (the CPU's write that turns the sink on), and the bench still owes six words. So both tickets describe the same moment, the isolation release at 418 and the sink enable just after, 11 cycles apart in Icarus and 2,001 apart in Verilator only because of how each failure is detected. And both trace back to the same window, the 322 cycles with the power off.

Now localise the cause. While the domain is off, what is allowed to change the FIFO's read pointer? Only its read increment, rinc, which comes from the sink inside the switched domain. Isolation should hold every output of that domain at 0 while iso_en is high. Read the clamps:

$ sed -n 33,36p rtl/periph_pd.sv
  // Isolation: every output of the domain is clamped to 0 while iso_en is high.
  assign rinc      = rinc_p;          // internal: only feeds our own async FIFO
  assign out_valid = iso_en ? 1'b0  : valid_p;
  assign out_data  = iso_en ? 32'h0 : data_p;

Two outputs are clamped and one is not. The comment says rinc stays inside the peripheral, and that is true of the module; it is false of the power domain, because the FIFO's read side is always on. The systematic walk from the pins to this line is in cone-of-influence tracing; here the probe pointed straight at it.

Hypothesise and test: is the symptom in the RTL or in the model?

Hypothesis: one bug, the missing clamp on rinc; the simulators differ only in what an unpowered output is. A good hypothesis predicts things a wrong one would not. This one predicts three:

First the two bisects, one per ticket, in a fresh build of the history:

$ git bisect start HEAD v0.1 && git bisect run tb/run.sh periph_power 1 verilator
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[27acd3070be8abc6d2e39268f12788f0d15688ee] periph: deepen the async FIFO to 16 entries
running 'tb/run.sh' 'periph_power' '1' 'verilator'
...
PASS test=periph_power seed=1
...
[122e88a7c8e74f9e3b29db8fad03dbe3fd3b7e9b] periph_pd: isolate only the outputs that leave the peripheral
running 'tb/run.sh' 'periph_power' '1' 'verilator'
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008
...
122e88a7c8e74f9e3b29db8fad03dbe3fd3b7e9b is the first bad commit
commit 122e88a7c8e74f9e3b29db8fad03dbe3fd3b7e9b
Author: dbglab <dbglab@example.com>
Date:   Mon Mar 16 10:00:00 2026 +0000

    periph_pd: isolate only the outputs that leave the peripheral
    
    rinc only feeds our own async FIFO, so it does not need a clamp.
...
$ git bisect reset
...
$ git bisect start HEAD v0.1 && git bisect run tb/run.sh periph_power 1
...
122e88a7c8e74f9e3b29db8fad03dbe3fd3b7e9b is the first bad commit
...

Four runs each, the same commit, and its message is the comment from the line above. Then the experiment this page is really about. Scratch copies of periph_pd.sv change only what the supply model drives on an unpowered output: dbglab-zero drives 0, dbglab-ones drives 1, in both simulators. Same RTL, same bench, same seed:

$ sh supply_sweep.sh | head -6
dbglab       icarus    FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
dbglab       verilator FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008
dbglab-zero  icarus    PASS test=periph_power seed=1 cycles=469 time=4686 ns
dbglab-zero  verilator PASS test=periph_power seed=1 cycles=469 time=4686 ns
dbglab-ones  icarus    FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008
dbglab-ones  verilator FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008

Both predictions hold. With an unpowered output of 1, Icarus hangs at exactly Verilator's cycle: the X was never an Icarus problem, it was Icarus's way of drawing an unknown level. With 0, both simulators pass: a rinc stuck at 0 pops nothing, which is exactly what the missing clamp should have guaranteed. And with no supply model at all:

$ iverilog -g2012 -Wno-timescale -DSYNTHESIS -I tb -o build/nosupply.vvp $(grep -v '^#' rtl/files.f) tb/bus_rules.sv tb/dbglab_tb.sv
$ vvp -n build/nosupply.vvp +TEST=periph_power +SEED=1 | grep -E '^(PASS|FAIL)'
PASS test=periph_power seed=1 cycles=469 time=4686 ns

The SYNTHESIS branch has no supply model: the sink is held in reset while off, so its rinc is a clean 0 and the bug cannot show. The third prediction holds too. The symptom belongs to the supply model; the bug belongs to the RTL. A flow that simulates power-gated blocks without a supply model, or with one that drives 0, passes this design, and silicon, where an unpowered output floats to whatever level it likes, does not.

The fix and the lesson

Fix: clamp the output the reader needs clamped

// snippet: the fix, rtl/periph_pd.sv line 34
  assign rinc      = iso_en ? 1'b0  : rinc_p;

Clamp to the value its reader treats as idle: for a read increment, 0. The comment goes too; it was the bug in words.

Prove: under every value an unpowered output can take

The usual proof, the repro passing in both simulators, covers two of the four supply models. The experiment above showed that the symptom depends on the model, so the proof must cover all of them. The same sweep, with the clamp in each copy:

$ sh supply_sweep.sh | tail -6
fix          icarus    PASS test=periph_power seed=1 cycles=469 time=4686 ns
fix          verilator PASS test=periph_power seed=1 cycles=469 time=4686 ns
fix-zero     icarus    PASS test=periph_power seed=1 cycles=469 time=4686 ns
fix-zero     verilator PASS test=periph_power seed=1 cycles=469 time=4686 ns
fix-ones     icarus    PASS test=periph_power seed=1 cycles=469 time=4686 ns
fix-ones     verilator PASS test=periph_power seed=1 cycles=469 time=4686 ns
$ tb/regress.sh icarus 1 2 3 | grep failing          # in the fixed tree; the same in verilator
6 failing run(s)

Six passes in 469 cycles, the same count as the clean dbglab-zero runs, so the fix restores the behaviour of a domain whose outputs really are quiet. The six failing regression runs are irq_w1c and dma_burst_backpressure, two other known bugs with their own signatures. Varying the off value is the sweep that matters here; varying the seed is not, because the seed never touches the power sequence.

Prevent: make the model and the check independent of luck

• An always-on isolation check per switched domain, listing every output and its clamp value. The one in power-intent bugs fires at cycle 76 on this bug, 355 cycles before ticket A.

• Run power tests under at least two supply models, one of which is not 0: X in the four-state simulator and noise (or all ones) in the two-state one. dbglab already did this by accident, which is why there were two tickets. Make it a rule in the verification plan.

• The review question for every port of a switched domain: which supply powers its reader? "It is internal" is an answer about the module tree, not about power.

Which step won? Observe and Localise together: one probe in both simulators, and one subtraction, turned two tickets into one moment. Test made it certain, with an experiment that changed the symptom without touching the RTL.

Related topics

• Part 10 · Case Studies

• Power-intent bugs: the class and an always-on isolation check

• Bisection across configurations: the simulator matrix

• Who is waiting for whom: B5's hang as a chain that ends in nobody

• Escape analysis: why the regression missed B5

• Low-power verification: redundant or missing isolation

• RTL: power domains and isolation intent