The block that forgets after its first sleep

A sensor hub passes its whole regression. The first time the firmware team adds a sleep and wake to a test, three things go wrong at once: the hub stops sending samples, its mode register reads 00, and the always-on interrupt counter shows one event nobody sent. In Icarus a pin goes X; in Verilator the same test hangs. Nothing in the RTL changed. Where do you start?

What people usually do, and why it fails

Two habits: call it X pessimism and add a reset, so the X goes away and the bug ships; or debug the data path, where the failure was reported. Both miss what is special here: the logic is correct in every cycle where the supply is on. The bug is in power intent, the rules for what happens at a domain's edges when its supply goes off and comes back.

Plain RTL simulation has no supplies, so it cannot see these bugs. Real flows describe the intent in UPF and use a power-aware simulator; this page uses a hand-written supply model, as dbglab does: an off domain's outputs are X in a four-state simulator and noise in a two-state one.

The signature

Here is the retention case as a waveform. Two copies of the same domain sleep and wake in the same cycles; mode_good has a retention shadow, mode_noret does not, and raw_noret is the domain's own output before the clamp.

Read it as the power-state timeline first, then the data.

• Slot 1: software writes 5b. Slot 3: isolation goes on, so both clamped outputs read 00 from here.

• Slot 4: the domain is held in reset, so its own output drops to 0. Slots 5 to 7: the supply is off and the output inside the domain is X; outside, the clamp holds 00, which is what isolation is for.

• Slot 10: restore. Slot 11: isolation comes off. mode_good reads 5b again; mode_noret reads 00, its reset value, not X and not garbage.

The first three checks

On the debug loop these checks are Localise: they turn "something broke after a sleep" into one boundary signal or one register.

1. Line the first wrong value up with the power timeline. Print pwr_on, iso_en, the domain reset and save/restore around the first wrong cycle. Inside an off window points at isolation; just after a wake points at retention; at a transition edge points at order.

2. List every signal that leaves the switched domain, and who reads it. For each one whose reader is powered differently, check that it is clamped while iso_en is high. A wrong comment is not evidence; the reader's supply is.

3. For every state that must survive, check save, restore and order. Compare each value after the wake with its value before the sleep, and compare the sequence with the spec: save, isolate, reset, power off; then power on, release reset, restore, release isolation.

One real reproduction: a small power domain

The bench runs three copies of one domain through the same sleep and wake: correct, without retention on mode_q, and with a controller that releases isolation before the supply comes back. Two always-on checks watch each copy: the clamp must hold while the supply is off, and mode_o must read the same after a wake as before the sleep.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

// pd_core: a block in a switchable power domain. mode_q is its configuration (bit 0 enables a
// tick every fourth cycle). RET = 1 gives mode_q a retention shadow on the always-on supply:
// save copies it, restore puts it back. RET = 0 is the bug: nothing keeps mode_q across power-off.
// Supply model (simulation only): while pwr_on is low the outputs are not logic values, X in a
// four-state simulator and noise in a two-state one.
`timescale 1ns/1ps
module pd_core #(parameter bit RET = 1) (
  input  logic       clk, pwr_on, pd_rst_n, save, restore, wr,
  input  logic [7:0] wdata,
  output logic [7:0] mode_p,
  output logic       tick_p
);
  logic [7:0] mode_q, ret_q;
  logic [1:0] cnt_q;
  always_ff @(posedge clk or negedge pd_rst_n)
    if (!pd_rst_n)         begin mode_q <= 8'h00; cnt_q <= '0; end
    else begin
      if (restore && RET)  mode_q <= ret_q;
      else if (wr)         mode_q <= wdata;
      cnt_q <= cnt_q + 1'b1;
    end
  always_ff @(posedge clk) if (save) ret_q <= mode_q;          // always-on shadow
`ifdef VERILATOR
  logic [8:0] noise = 9'h1a5;
  always_ff @(posedge clk) noise <= {noise[7:0], noise[8] ^ noise[4]};
  assign {mode_p, tick_p} = pwr_on ? {mode_q, mode_q[0] && cnt_q == 2'd3} : noise;
`else
  assign {mode_p, tick_p} = pwr_on ? {mode_q, mode_q[0] && cnt_q == 2'd3} : 9'bx;
`endif
endmodule

// pmu: the always-on power controller. Down: save, isolate, reset, power off.
// Up (EARLY_ISO = 0): power on, release reset, restore, then release isolation.
// EARLY_ISO = 1 is the second bug: isolation is released first, while the supply is still off.
module pmu #(parameter bit EARLY_ISO = 0) (
  input  logic clk, rst_n, sleep_req, wake_req,
  output logic pwr_on, iso_en, pd_rst_n, save, restore, ready
);
  typedef enum logic [3:0] {BOOT, ON, SAVE, ISO, RST, OFF, UP, UNRST, RESTORE} st_t;
  st_t st;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) st <= BOOT;
    else case (st)
      BOOT:    st <= ON;
      ON:      if (sleep_req) st <= SAVE;
      SAVE:    st <= ISO;
      ISO:     st <= RST;
      RST:     st <= OFF;
      OFF:     if (wake_req) st <= UP;
      UP:      st <= UNRST;
      UNRST:   st <= RESTORE;
      default: st <= ON;
    endcase
  always_comb begin
    {pwr_on, iso_en, pd_rst_n, save, restore, ready} = 6'b101001;  // ON: powered, not isolated, ready
    case (st)
      BOOT:    {iso_en, pd_rst_n, ready} = 3'b100;
      SAVE:    {save, ready} = 2'b10;
      ISO:     {iso_en, ready} = 2'b10;
      RST:     {iso_en, pd_rst_n, ready} = 3'b100;
      OFF:     {pwr_on, iso_en, pd_rst_n, ready} = 4'b0100;
      UP:      {pwr_on, iso_en, pd_rst_n, ready} = EARLY_ISO ? 4'b0000 : 4'b1100;
      UNRST:   {iso_en, ready} = {!EARLY_ISO, 1'b0};
      RESTORE: {iso_en, restore, ready} = {!EARLY_ISO, 2'b10};
      default: ;
    endcase
    if (!rst_n) pd_rst_n = 1'b0;
  end
endmodule

// pd_unit: one domain, its controller and its isolation clamps (to 0) on the always-on side.
module pd_unit #(parameter bit RET = 1, parameter bit EARLY_ISO = 0) (
  input  logic       clk, rst_n, sleep_req, wake_req, wr,
  input  logic [7:0] wdata,
  output logic [7:0] mode_o,
  output logic       tick_o, ready, iso_en, pwr_on
);
  logic pd_rst_n, save, restore;
  logic [7:0] mode_p;
  logic       tick_p;
  pmu #(.EARLY_ISO(EARLY_ISO)) u_pmu (.clk, .rst_n, .sleep_req, .wake_req, .pwr_on, .iso_en,
                                      .pd_rst_n, .save, .restore, .ready);
  pd_core #(.RET(RET)) u_core (.clk, .pwr_on, .pd_rst_n, .save, .restore, .wr, .wdata, .mode_p, .tick_p);
  assign mode_o = iso_en ? 8'h00 : mode_p;
  assign tick_o = iso_en ? 1'b0  : tick_p;
endmodule

// pd_check: always-on checks. ISO: while the domain's supply is off, its outputs must reach the
// always-on side as the clamp value. RET: after a wake, mode_o must read what it read before the sleep.
module pd_check (
  input logic       clk, ready, iso_en, pwr_on,
  input logic [7:0] mode_o,
  input logic       tick_o,
  input logic [7:0] pre_mode
);
  int    cycle = 0, n_iso = 0, n_ret = 0;
  string first = "";
  bit    was_ready = 1'b0;
  always @(posedge clk) begin
    #1;
    if (!pwr_on && (mode_o !== 8'h00 || tick_o !== 1'b0)) begin
      n_iso++;
      if (first == "") first = $sformatf("cycle %0d: ISO: mode_o=%h tick_o=%b with the supply off (iso_en=%b)",
                                         cycle, mode_o, tick_o, iso_en);
    end
    if (ready && !was_ready && cycle > 2 && mode_o !== pre_mode) begin
      n_ret++;
      if (first == "") first = $sformatf("cycle %0d: RET: mode_o=%h after wake, %h before sleep", cycle, mode_o, pre_mode);
    end
    was_ready = ready;
    cycle++;
  end
endmodule

$ iverilog -g2012 -o pd.vvp pd_tb.sv && vvp -n pd.vvp
good      (retention, isolate until restored): clean
no_ret    (mode_q not retained):               cycle 23: RET: mode_o=00 after wake, 5b before sleep
early_iso (isolation released before power):   cycle 20: ISO: mode_o=xx tick_o=x with the supply off (iso_en=0)
after wake: mode_o good=5b no_ret=00 early_iso=5b
PASS: retention loss and early isolation release both caught; the good sequence is clean
...
$ verilator --binary --timing --assert -Wno-fatal --top-module pd_tb -Mdir vl pd_tb.sv && ./vl/Vpd_tb
...
good      (retention, isolate until restored): clean
no_ret    (mode_q not retained):               cycle 23: RET: mode_o=00 after wake, 5b before sleep
early_iso (isolation released before power):   cycle 20: ISO: mode_o=63 tick_o=1 with the supply off (iso_en=0)
after wake: mode_o good=5b no_ret=00 early_iso=5b
PASS: retention loss and early isolation release both caught; the good sequence is clean
...

Line by line. no_ret fails the retention check at cycle 23, the first cycle it is ready again, with 00: its reset value, which looks like a configuration bug and is not one. early_iso fails at cycle 20, the one cycle in which isolation is already off and the supply is not yet on. In Icarus the value is X. In Verilator there is no X, so the same cycle carries noise: tick_o=1, an event the domain never sent. A consumer that counts ticks would count one more, which is the phantom interrupt from the scene.

The same class in dbglab: B5, the missing clamp

dbglab's periph_power test powers the peripheral's sink down and up with words waiting in the async FIFO. At HEAD it fails differently in each simulator, which is the signature in the table above:

$ tb/run.sh periph_power 1
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
$ tb/run.sh periph_power 1 verilator
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008

Bisection in space and cone-of-influence tracing walk this failure back to one line, so here is only the class view. Check 2 asks for the domain's outputs and their readers. periph_pd has three: out_valid and out_data go to the pins, and rinc goes to the async FIFO's read side, which is always on. A check that states the isolation rule for all three, compiled next to the bench, finds the breach at its first cycle:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// iso_probe: the isolation rule as an always-on check. While iso_en is high, every output of the
// switched domain periph_pd must be at its clamp value (0). It only reads the design.
`timescale 1ns/1ps
module iso_probe;
  int n = 0;
  always @(posedge dbglab_tb.dut.u_per.u_pd.pclk) begin
    #0.1;
    if (dbglab_tb.dut.u_per.u_pd.iso_en &&
        (dbglab_tb.dut.u_per.u_pd.rinc !== 1'b0 || dbglab_tb.dut.u_per.u_pd.out_valid !== 1'b0
         || dbglab_tb.dut.u_per.u_pd.out_data !== 32'h0)) begin
      if (n == 0)
        $display("ISO-CHECK cycle %0d: iso_en=1 but rinc=%b out_valid=%b out_data=%h (pwr_on=%b)",
                 dbglab_tb.cycle, dbglab_tb.dut.u_per.u_pd.rinc, dbglab_tb.dut.u_per.u_pd.out_valid,
                 dbglab_tb.dut.u_per.u_pd.out_data, dbglab_tb.dut.u_per.u_pd.pwr_on);
      n++;
    end
  end
  final if (n) $display("ISO-CHECK: %0d pclk edges with an unclamped output", n);
endmodule

$ ../probes/run-probe.sh ../probes/iso_probe.sv periph_power 1
dbglab: test=periph_power seed=1
ISO-CHECK cycle 76: iso_en=1 but rinc=x out_valid=0 out_data=00000000 (pwr_on=0)
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
ISO-CHECK: 86 pclk edges with an unclamped output
$ grep -n "assign rinc" rtl/periph_pd.sv
34:  assign rinc      = rinc_p;          // internal: only feeds our own async FIFO

Cycle 76, with the supply off and isolation on: rinc is X while the two pin outputs are clamped. Cycle 76 is when this probe first looks, not when rinc went X. Cycles count the bus clock: the supply drops at 725 ns (cycle 73) and rinc goes X at that instant, the 73 that cone-of-influence tracing reads from the waveform; the falling-pclk monitor in bisection in space first sees it at 740 ns (cycle 74); this probe samples just after rising pclk edges, the first at 759 ns (cycle 76). Even that is 355 cycles before the bench's own check, in the right domain and on the right signal. The comment on line 34 is the mistake in words: rinc stays inside the peripheral, but its reader is not in the switched domain.

The fix pattern

• Missing isolation: clamp the signal at the domain boundary with the controller's iso_en, to the value its reader treats as idle. For B5: assign rinc = iso_en ? 1'b0 : rinc_p;. With it, periph_power passes in both simulators (cycles=469) and the probe is silent.

• Missing retention: make the state retained (a retention flop or a save/restore shadow), or make software reprogram it after every wake and say so in the spec. Do not paper over it with a different reset value.

• Wrong order: fix the controller's sequence, not the consumer. Making the reader ignore one bad cycle hides the bug until the timing changes.

Preventing it

• An always-on isolation check per domain, generated from the list of boundary ports, not typed by hand: while iso_en, every crossing output equals its clamp. The probe above is that check for one domain.

• A retention check after every wake that compares each state that must survive with its value before the sleep, as pd_check does.

• A power test with non-default settings. Most retention bugs survive because every test sleeps straight after reset, when the reset value and the saved value are the same.

• The review question: for each output of a switchable domain, which supply powers its reader? In a real flow, a static power-intent check answers it for every port; see the low-power verification course.

Traps

Related topics

• Part 5 · Bug Classes

• Low-power verification: state-loss scenarios

• Low-power verification: power-mode sequencing

• Low-power verification: corruption and retention in simulation

• RTL: power domains and isolation intent

• RTL: low-power design

• Part 10: case studies, including the missing isolation cell