A sensor that is wrong only when it matters

A sensor block averages two samples and raises an alarm when an offset passes a limit. Its directed tests use readings like 10, 20 and 40, and they all pass. In the lab, the average jumps from 127 to 0 as the input rises, and the alarm fires the moment the offset goes below zero.

Nobody has opened a waveform yet. Can the numbers alone tell you what kind of bug this is?

What people usually do, and why it fails

They add a test for each wrong case (an average of 200 and 100, an offset of -5) and patch each one, until the next report shows another value. Every case comes from the same two lines. The pattern of wrong values is the evidence, and it is thrown away one case at a time.

The signature

Logic bugs of this class are deterministic: the same inputs give the same wrong answer in every simulator and every seed. What gives them away is which inputs fail and by how much.

On the debug loop this is Localise and Hypothesise: the failing set and the size of the error give you a hypothesis before you have read a line of RTL.

The first three checks

1. Sort the failures by input. Find the smallest failing value and the largest passing one. A sharp edge at 128, 256 or a range end points straight at this class.

2. Subtract. Compute expected minus actual for every failure. One constant that is a power of two means a dropped top bit; plus or minus one means an off-by-one.

3. Read the widths and signs on the path. For each operator between the input and the wrong output, write down its width and signedness. Then run verilator --lint-only -Wall and read every WIDTH warning on that path.

One real reproduction: the sensor block

Here is the block from the scene. The FIX parameter selects the buggy lines (0) or the corrected ones (1, the default). The bench tries every input pair and compares each output with integer arithmetic.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

// widthsign: two classic logic bugs in a sensor block. mean is the average of two unsigned 8-bit
// samples; alarm is high when a signed offset is above an unsigned limit. FIX = 0 builds the
// buggy version; FIX = 1 (the default) the corrected one. The bench checks every input pair.
`timescale 1ns/1ps
module sensor #(parameter bit FIX = 1) (
  input  logic [7:0]        a, b,       // unsigned samples
  input  logic signed [7:0] ofs,        // signed offset, -128 to 127
  input  logic [6:0]        limit,      // unsigned limit, 0 to 127
  output logic [7:0]        mean,
  output logic              alarm);
  if (FIX) begin : g_good
    logic [8:0] sum;
    assign sum   = a + b;                                // 9 bits: keeps the carry
    assign mean  = sum[8:1];
    assign alarm = ofs > $signed({1'b0, limit});         // both operands signed
  end else begin : g_bad
    assign mean  = (a + b) >> 1;                         // bad: the sum is 8 bits wide here
    assign alarm = ofs > limit;                          // bad: one unsigned operand makes it unsigned
  end
endmodule

module tb;
  parameter bit FIX = 1;
  logic [7:0] a, b, mean;
  logic signed [7:0] ofs;
  logic [6:0] limit;
  logic alarm;
  int bad_mean = 0, bad_alarm = 0, d, short_min = 999, short_max = -999, sum_min = 999, ofs_max = -999;
  sensor #(.FIX(FIX)) u (.a, .b, .ofs, .limit, .mean, .alarm);

  initial begin
    for (int i = 0; i < 256; i++)
      for (int j = 0; j < 256; j++) begin
        a = i; b = j; #1;
        if (mean !== (i + j) / 2) begin
          if (bad_mean == 0) $display("first wrong mean: a=%0d b=%0d mean=%0d, expected %0d", i, j, mean, (i + j) / 2);
          bad_mean++;
          d = (i + j) / 2 - int'(mean);          // int'(): keep the arithmetic signed
          if (d < short_min) short_min = d;
          if (d > short_max) short_max = d;
          if (i + j < sum_min) sum_min = i + j;
        end
      end
    for (int o = -128; o < 128; o++)
      for (int l = 0; l < 128; l++) begin
        ofs = o; limit = l; #1;
        if (alarm !== (o > l)) begin
          if (bad_alarm == 0) $display("first wrong alarm: ofs=%0d limit=%0d alarm=%b, expected %b", o, l, alarm, o > l);
          bad_alarm++;
          if (o > ofs_max) ofs_max = o;
        end
      end
    $display("mean wrong for %0d of 65536 pairs, alarm wrong for %0d of 32768 pairs", bad_mean, bad_alarm);
    if (bad_mean)  $display("  every wrong mean has a+b >= %0d and is short by %0d to %0d", sum_min, short_min, short_max);
    if (bad_alarm) $display("  every wrong alarm has ofs <= %0d", ofs_max);
    if (bad_mean || bad_alarm) $fatal(1, "sensor outputs do not match the reference");
    $display("PASS: mean and alarm correct for every input (FIX=%0d)", FIX);
    $finish;
  end
endmodule

$ iverilog -g2012 -P tb.FIX=0 -o w0.vvp widthsign.sv && vvp -n w0.vvp
first wrong mean: a=1 b=255 mean=0, expected 128
first wrong alarm: ofs=-128 limit=0 alarm=1, expected 0
mean wrong for 32640 of 65536 pairs, alarm wrong for 16384 of 32768 pairs
  every wrong mean has a+b >= 256 and is short by 128 to 128
  every wrong alarm has ofs <= -1
FATAL: widthsign.sv:56: sensor outputs do not match the reference
...
$ iverilog -g2012 -P tb.FIX=1 -o w1.vvp widthsign.sv && vvp -n w1.vvp
mean wrong for 0 of 65536 pairs, alarm wrong for 0 of 32768 pairs
PASS: mean and alarm correct for every input (FIX=1)

Read the summary as the three checks would. The mean fails for 32,640 pairs, which is exactly the number of pairs whose sum is 256 or more, and every one is short by exactly 128. That is the carry of an 8-bit sum, halved. The alarm fails for 16,384 pairs: all 128 negative offsets against all 128 limits, and no others. Verilator prints the same five lines. Two hypotheses, one suspect line each, and no waveform.

Why the lines are wrong: in (a + b) >> 1 the sum takes its width from the widest operand and the target, all 8 bits, so the carry is gone before the shift. In ofs > limit one operand is unsigned, so Verilog compares both as unsigned and -1 becomes 255. The RTL course's signed arithmetic pattern gives the full rules.

The same class in dbglab: B3, the counter that wraps

The Debug Loop walked bug B3 from report to fix: the DMA's 8-bit beat counter left_q takes a 10-bit length, and runs longer than 255 words stop early. It has this class's signature exactly: it fails only for long runs, and short by exactly 256. Here it is caught by a load check instead of the scoreboard. The check compares left_q with the length the cycle after START:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// wrap_check: the cycle after the DMA accepts START, its beat counter must hold the full length.
// Prints the first violation. It only reads the design.
`timescale 1ns/1ps
module wrap_check;
  int unsigned want = 0;
  bit armed = 0, seen = 0;
  always @(posedge dbglab_tb.clk) begin
    #0.5;                                                   // just after the edge
    if (armed && !seen && 32'(dbglab_tb.dut.u_dma.left_q) != want) begin
      $display("WRAPCHK cycle=%0d: LEN=%0d loaded as left_q=%0d", dbglab_tb.cycle, want,
               dbglab_tb.dut.u_dma.left_q);
      seen = 1;
    end
    armed = 0;
    if (dbglab_tb.dut.u_dma.start && dbglab_tb.dut.u_dma.state == 0 && dbglab_tb.dut.u_dma.cfg_len != 0) begin
      want = dbglab_tb.dut.u_dma.cfg_len; armed = 1;
    end
  end
endmodule

$ ../probes/run-probe.sh ../probes/wrap_check.sv random 10
dbglab: test=random seed=10
WRAPCHK cycle=4443: LEN=308 loaded as left_q=52
FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)

The check fires at cycle 4443, the cycle the counter loads, 452 cycles before the scoreboard notices. The Debug Loop already swept all sixty seeds with a probe; what is new here is the form: a load check is an assertion you can leave in the regression, so the next truncation fails at its cause. The nine seeds that fail with an X are a different class, the subject of X and reset bugs.

The fix pattern

• Width: size the intermediate for the largest result (an N-bit sum needs N+1 bits) and take the bits you want from it explicitly, as sum[8:1] does.

• Sign: make both operands signed, widening the unsigned one with a zero first ($signed({1'b0, limit})), or keep both unsigned and handle the sign yourself.

• Off-by-one: write down what the counter means at the moment of the compare ("words still to write"), and derive the end condition from that sentence. Size counters from the range they count ($clog2(MAX+1)), not from a neighbour's width.

Preventing it

• Lint as a gate. verilator --lint-only -Wall flags WIDTHTRUNC and WIDTHEXPAND. On the sensor it flagged the alarm line (Operator GT expects 8 bits on the RHS, but RHS's VARREF 'limit' generates 7 bits) but said nothing about the mean, whose operands are all 8 bits. Lint catches width mismatches between names, not lost carries.

• Boundary tests. For every counter and length, test the values either side of each power of two and the maximum: 255, 256, 257 and 1023 for a 10-bit length.

• Load checks. An assertion that a register holds what it was loaded with, the cycle after the load, turns a late scoreboard failure into a failure at the cause.

• Review question: "Where can this value be wider, or negative, than the thing it is stored in or compared with?"

Traps

Related topics

• Part 5 · Bug Classes

• RTL: signed arithmetic pattern

• RTL: overflow detection patterns

• The debug loop, walked on B3