The report

A sensor interface has an 8-bit sample-and-hold register: a one-cycle cap pulse captures the ADC value din, and the register keeps it while the ADC moves on. To save power, the register sits behind a hand-instantiated clock gate, so when cap is 0 it gets no clock at all. Block tests of the hold behaviour have passed for months. The full-chip boot test, which runs the boot-time memory self-test (BIST) first and then the same functional checks, fails:

$ iverilog -g2012 -o l.vvp leak_tb.sv && vvp -n l.vvp +SHOW=3
FAIL cycle 41: held=68, but 8c was captured at cycle 40 and cap has been 0 since
FATAL: leak_tb.sv:67: the sample-and-hold register did not hold
...

The register captured 8c and, one cycle later, holds 68, which is the ADC's next value. Nobody pulsed cap. The register's code is one line.

Stop here if you want to try it yourself. The design and bench are in the hunt. What can make a register take a new value when its only enable is off, and why would it need the boot sequence?

The hunt

Reproduce: the design, and a run without the self-test

The bench below has the clock gate, the self-test controller, the sampler with its test glue, and one functional test that runs with and without the boot self-test. The BUG parameter selects the glue; you are debugging rig 3, the design as built, after a boot self-test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

`timescale 1ns/1ps
// icg: a latch-based integrated clock gate, as the library cell behaves. The enable is captured
// while clk is low, so gclk has no glitches. te (test enable) forces the clock on: scan uses it to
// clock every flop during shift, and memory self-test uses it to keep the clocks running.
module icg (input logic clk, en, te, output logic gclk);
  logic en_l;
  always_latch if (!clk) en_l = en | te;
  assign gclk = clk & en_l;
endmodule

// bist_ctrl: a boot-time memory self-test engine. Software sets bist_en; the engine runs once
// (busy for 16 cycles), then sets done and stays idle until the next reset. Its programming
// guide says bist_en may be left set: the engine never restarts without a reset.
module bist_ctrl (input logic clk, rst_n, bist_en, output logic busy, done);
  logic [4:0] n;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin busy <= 1'b0; done <= 1'b0; n <= '0; end
    else if (busy) begin
      n <= n + 5'd1;
      if (n == 5'd15) begin busy <= 1'b0; done <= 1'b1; end
    end else if (bist_en && !done) busy <= 1'b1;
endmodule

// sampler: an 8-bit sample-and-hold register on a hand-instantiated clock gate. cap captures din;
// otherwise the gate is shut and held keeps its value. The test glue drives the gate's te.
//   BUG = 0: te = scan_mode | bist_busy   (on only while the self-test runs)
//   BUG = 1: te = scan_mode | bist_en     (the software enable bit, which may stay set)
module sampler #(parameter bit BUG = 0) (
  input  logic       clk, rst_n, scan_mode, bist_en, cap,
  input  logic [7:0] din,
  output logic [7:0] held,
  output logic       bist_busy, bist_done, te, gclk
);
  bist_ctrl u_bist (.clk, .rst_n, .bist_en, .busy(bist_busy), .done(bist_done));
  assign te = BUG ? (scan_mode | bist_en) : (scan_mode | bist_busy);
  icg u_cg (.clk, .en(cap), .te, .gclk);
  always_ff @(posedge gclk) held <= din;      // no enable here: the clock gate is the enable
endmodule

// rig: one sampler and one functional test. Reset, optionally the boot self-test, then the
// test proper: capture din once (cycle 40) and check that held keeps it for ten cycles while
// din keeps changing. A test-glue check states the functional-mode rule for te.
module rig #(parameter bit BUG = 0, parameter bit BOOT_BIST = 0, parameter int ID = 0) (
  input logic clk, input int show, trace_from, trace_to, input bit clear
);
  logic       rst_n = 1'b0, scan_mode = 1'b0, bist_en = 1'b0, cap = 1'b0;
  logic [7:0] din = 8'h00, held, want = 8'h00;
  logic       busy, done, te, gclk;
  int         cyc = 0, gedges = 0, win_edges = 0;
  string      fail = "", techeck = "";
  sampler #(.BUG(BUG)) u (.clk, .rst_n, .scan_mode, .bist_en, .cap, .din, .held,
    .bist_busy(busy), .bist_done(done), .te, .gclk);
  always @(posedge gclk) gedges++;
  always @(posedge clk) begin                  // after this edge's updates
    #1;
    cyc++;
    if (show == ID && cyc >= trace_from && cyc <= trace_to)
      $display("%5d      %b      %b     %b    %b    %b  %b   %h   %h    %0d", cyc, scan_mode, bist_en, busy, done, te,
               cap, din, held, gedges);
    // the functional-mode rule: te is on only in scan mode or while the self-test runs
    if (techeck == "" && te && !scan_mode && !busy)
      techeck = $sformatf("cycle %0d: te=1 with scan_mode=0 and bist_busy=0 (bist_en=%b done=%b)", cyc, bist_en, done);
    if (cyc > 40 && cyc <= 50 && held !== want && fail == "") begin
      fail = $sformatf("cycle %0d: held=%h, but %h was captured at cycle 40 and cap has been 0 since", cyc, held, want);
      if (show == ID) begin
        $display("FAIL %s", fail);
        $fatal(1, "the sample-and-hold register did not hold");
      end
    end
  end
  initial begin
    int unsigned r;
    r = 32'h2545_F491;
    repeat (2) @(negedge clk);
    rst_n = 1'b1;
    @(negedge clk) bist_en = BOOT_BIST;        // boot: start the self-test, or not
    if (clear) fork                            // +CLEAR=1: software clears bist_en once done
      begin wait (done); @(negedge clk) bist_en = 1'b0; end
    join_none
    while (cyc < 39) begin                     // an ADC that never stops moving
      @(negedge clk);
      r ^= r << 13; r ^= r >> 17; r ^= r << 5;
      din = 8'(r);
    end
    cap = 1'b1; want = din;                    // edge 40 captures din
    for (int i = 0; i < 10; i++) begin         // then cap is 0 and din keeps moving
      @(negedge clk);
      if (i == 0) begin cap = 1'b0; win_edges = gedges; end
      r ^= r << 13; r ^= r >> 17; r ^= r << 5;
      din = 8'(r);
    end
    @(negedge clk) win_edges = gedges - win_edges;
  end
endmodule

The failure is deterministic: cycle 41 in Icarus Verilog and in Verilator. The report already contains a differential, so use it (differential debugging): the same functional test passes in the block regression, which never runs the self-test. Rig 2 is the same design with the boot self-test skipped. Same ADC values, same capture:

$ vvp -n l.vvp +SHOW=2 +FROM=40 +TO=43
cycle  scan_mode bist_en busy done te cap din  held  gclk edges
   40      0      0     0    0    0  1   8c   8c    1
   41      0      0     0    0    0  0   68   8c    1
   42      0      0     0    0    0  0   7b   8c    1
   43      0      0     0    0    0  0   a2   8c    1
...

Without the self-test, held keeps 8c, and the last column, a count of rising edges on the gated clock, moves once: the capture. So the difference is not in the functional sequence. It is something the self-test left behind.

Observe: the register is being clocked

Now the failing run, around the capture and then around the end of the self-test:

$ vvp -n l.vvp +SHOW=3 +FROM=1 +TO=45
cycle  scan_mode bist_en busy done te cap din  held  gclk edges
    1      0      0     0    0    0  0   00   xx    0
...
    4      0      1     1    0    1  0   00   00    1
    5      0      1     1    0    1  0   3a   3a    2
...
   18      0      1     1    0    1  0   3e   3e    15
   19      0      1     1    0    1  0   bc   bc    16
   20      0      1     0    1    1  0   d2   d2    17
   21      0      1     0    1    1  0   ef   ef    18
...
   39      0      1     0    1    1  0   ed   ed    36
   40      0      1     0    1    1  1   8c   8c    37
   41      0      1     0    1    1  0   68   68    38
FAIL cycle 41: held=68, but 8c was captured at cycle 40 and cap has been 0 since
...

Two things stand out before you open any RTL. First, from cycle 4 on, held equals din on every line, not only at the capture. Second, from cycle 4 on, the gated-clock counter goes up by one on every cycle. The register is not being written by mistake; it is being clocked on every edge. The DFT-mode bugs signature table calls this a forced-open clock gate: activity with no enable.

Localise: walk the gated clock's cone

Take the cone of influence one stage at a time, but of the clock, not the data. held is clocked by gclk, the output of u_cg. The gate opens when its latched enable is 1, and that enable is cap | te. cap is 0 at cycle 41, so te must be 1, and the trace agrees. te is test glue: scan_mode | bist_en. scan_mode is 0. So the gate is open because `bist_en` is 1.

Now walk back in time along te:

• Cycle 4: software sets bist_en, the self-test starts (busy), and te goes to 1. That is intended: the memories and their wrappers need their clocks running during the self-test, which is what a clock gate's test enable is for. Scan uses the same pin to clock every flop during shift.

• Cycle 20, the first wrong cycle: the self-test finishes (busy 0, done 1). Nothing is testing any more, but te stays 1, because bist_en is still 1.

• Why is `bist_en` still 1? Because the controller's programming guide allows it: the engine runs once per reset and ignores the bit afterwards. Boot software, correctly, never clears it. The glue's author read bist_en as "the self-test is running". It means "software asked for a self-test".

The bad glue in a reduced copy (the same clock gate, a 3-cycle self-test) shows the whole mechanism in twelve slots:

Look for the slot where busy falls and te does not.

• Slot 2: software sets bist_en; the self-test starts and te rises with it, as intended.

• Slot 5: the self-test is over (bist_busy falls), but te stays high because it follows bist_en.

• Slot 6: cap captures 8c.

• Slots 7 to 10: cap is 0, yet held takes 68, 2f, d9 and 3e, every new din: the gate is still open.

Hypothesise and test: a prediction, and a workaround that hides it

Hypothesis: the clock gate is forced open by te, because te follows bist_en, which stays set after the self-test. It predicts something checkable without changing the RTL: if software clears bist_en as soon as the self-test is done, the hold test passes and the gated clock stops in the hold window. +CLEAR=1 makes the bench's software do exactly that:

$ vvp -n l.vvp +CLEAR=1
good glue, no boot BIST: hold test clean; 0 gated-clock edges in the hold window
good glue, boot BIST:    hold test clean; 0 gated-clock edges in the hold window
bad glue,  no boot BIST: hold test clean; 0 gated-clock edges in the hold window
bad glue,  boot BIST:    hold test clean; 0 gated-clock edges in the hold window
te check, good glue, boot BIST: clean
te check, bad glue,  boot BIST: cycle 20: te=1 with scan_mode=0 and bist_busy=0 (bist_en=1 done=1)
END: experiment finished (software clears bist_en after the self-test)
...

The prediction holds: the hold test passes and the gated clock is quiet. But read the sixth line. Even with the software change, the te check (in the bench, explained below) still fires at cycle 20, because software can only clear the bit after it sees done, so te stays on for at least that cycle. The software change hides the symptom; the glue still leaks test mode into function for as long as software takes to react, and on any boot path that forgets. It is a useful workaround for silicon that already exists, and a way to confirm the hypothesis. It is not the fix. See fix the cause, not the symptom.

The fix and the lesson

Fix: drive te from what the test logic is doing

// snippet: sampler's test glue
  assign te = scan_mode | bist_busy;   // was: scan_mode | bist_en

bist_busy is high only while the engine runs, so te returns to 0 the cycle the self-test ends, whatever software does with the enable bit.

Prove: all four rigs and the te check

$ vvp -n l.vvp
good glue, no boot BIST: hold test clean; 0 gated-clock edges in the hold window
good glue, boot BIST:    hold test clean; 0 gated-clock edges in the hold window
bad glue,  no boot BIST: hold test clean; 0 gated-clock edges in the hold window
bad glue,  boot BIST:    hold test cycle 41: held=68, but 8c was captured at cycle 40 and cap has been 0 since; 10 gated-clock edges in the hold window
te check, good glue, boot BIST: clean
te check, bad glue,  boot BIST: cycle 20: te=1 with scan_mode=0 and bist_busy=0 (bist_en=1 done=1)
PASS: only the bad glue after a boot self-test fails the hold test; the te check names it first
...

• Line 2, the fix: with the corrected glue and the boot self-test, the register holds and the gated clock is silent for the ten hold cycles.

• Line 4, the bug: ten gated-clock edges in ten cycles. On silicon that is the power symptom of the same bug: every clock gate driven by this te stays open after boot.

• Line 3: the bad glue passes without the self-test. That is why the block regression never saw it, and why a differential run found it so quickly.

• Lines 5 and 6: the te check is silent on the fix and names the bug at cycle 20.

Verilator 5.048 prints the same lines.

Prevent: check the test control, not the victim

The te check is the functional-mode rule from DFT-mode bugs, written for this glue: outside scan mode, `te` is 1 only while the self-test is running. It fired at cycle 20, 21 cycles before the hold test, and it would fire in any test that runs the boot self-test, whatever the function is doing. The hold test needed a capture, a moving input and a check in the right window; the te check needs none of that. Run it in every full-chip test, and ask in review, for every test control: what is its value in mission mode after each test feature has run? The test-mode integration checklist is the place for that question.

Which step won this hunt? Localise, twice: the differential run cut the suspects to what the self-test leaves behind, and following the clock, not the data, led from the register to te in three hops.

Related topics

• Part 10 · Case Studies

• Part 5: DFT-mode bugs

• RTL: the ICG cell

• RTL: test-mode integration checklist

• Counters and performance probes

• Cone-of-influence tracing