The total is right, and software still sees nonsense

A saturating accumulator sums signed samples into a 12-bit total that must stick at +2047 and -2048. Its regression is green: after 600 samples the total equals the model's. Then the firmware team, which polls the total while samples stream in, reports that it sometimes jumps from about -2000 to about +2000 in one sample. The bench has a reference model. Why did it not see this, and how do you make it point at the sample that went wrong?

What people usually do, and why it fails

The usual first move is to stare at the failing run and reason about what each signal should have been: a reference model in your head, slow, and wrong exactly where your understanding is. The second is to trust the existing check: "the model agrees at the end, so the firmware must be wrong". A check that compares once per test can say whether the end state matches, never when the design left the path.

The third habit: diff everything against a second simulator or an older release, get hundreds of differences, and give up. A reference that differs in many legitimate ways buries the one that matters.

The method: a reference, its allowed differences, and the first surprise

On the debug loop this is Localise: the reference supplies the expected value of every compared signal, so the first disagreement is the first wrong cycle without walking back by hand.

1. Pick the reference that differs from the failing run in the fewest ways. The same stimulus and seed always; then as little else as possible.

2. Write down how it is allowed to differ. A model has no pipeline latency; another simulator has no X, or orders races differently; an older release lacks some features. Anything on the list is noise.

3. Compare as often as the reference allows. Every beat or transaction beats every test; every register beats every output.

4. Read the first difference not on the list, with the inputs at that moment. That is your localisation.

5. If design and reference disagree, the spec decides who is right. A reference can be wrong too; see when the bug is in the test.

The picture

For 428 of 600 beats the design was wrong, and the last compare happens in the 83 beats where it was right again. Saturation forgets history: once both totals stick at +2047, the bug is erased from the state the end check reads.

Do it, part 1: a reference model on every beat

The bench below drives 600 samples, drifting down for 300 and up for 300. ref_add is the reference: plain integers and a clamp, written from the spec, sharing no code with the RTL. One checker compares once at the end; the other compares after every beat and prints the first mismatch with the state before it. The bench runs the shipped design and the fixed one side by side and checks that each checker behaves as this page says.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

// sat_acc adds a signed 8-bit sample to a 12-bit signed total on every valid beat and must
// saturate at +2047 and -2048. acc_o is visible every cycle (software polls it).
// sat_acc_bad is the version that shipped; sat_acc is the fix. The bench runs both.
`timescale 1ns/1ps
module sat_acc_bad (
  input  logic               clk, rst_n, valid,
  input  logic signed [7:0]  s,
  output logic signed [11:0] acc_o
);
  logic signed [12:0] sum;
  assign sum = acc_o + s;                          // one extra bit holds any single overflow
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n)                 acc_o <= '0;
    else if (valid) begin
      if (sum > 13'sd2047)      acc_o <= 12'sd2047;  // saturate high
      else                      acc_o <= sum[11:0];  // bug: no low limit, -2049 wraps to +2047
    end
endmodule

module sat_acc (
  input  logic               clk, rst_n, valid,
  input  logic signed [7:0]  s,
  output logic signed [11:0] acc_o
);
  logic signed [12:0] sum;
  assign sum = acc_o + s;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n)                 acc_o <= '0;
    else if (valid) begin
      if (sum > 13'sd2047)      acc_o <= 12'sd2047;
      else if (sum < -13'sd2048) acc_o <= -12'sd2048; // the fix: saturate low too
      else                      acc_o <= sum[11:0];
    end
endmodule

$ iverilog -g2012 -o sat.vvp sat_acc_tb.sv && vvp -n sat.vvp
lockstep, sat_acc_bad: first difference at beat 89: acc_o=2034 model=-2048 (before: acc_o=-2023 model=-2023, s=-39)
end of test, sat_acc_bad: acc_o=2008 model=2008 (equal)
end of test, sat_acc:     acc_o=2008 model=2008; lockstep differences: none
PASS: end-of-test compare missed the bug; lockstep compare found it at beat 89
sat_acc_tb.sv:82: $finish called at 6020000 (1ps)

Verilator prints the same four lines. Read them in order:

• Lockstep, beat 89. Before the beat both agree at -2023. The sample is -39, so the true sum is -2062, below the floor. The model clamps to -2048; the design shows +2034, which is -2062 plus 4096. That is a wrap, and it happens exactly when the sum crosses the low limit. One line has localised the bug to one branch of the saturation logic.

• End of test, buggy design: equal. Both read 2008. The end check would have shipped this design.

• End of test, fixed design: the same total and no lockstep difference in 600 beats. The fix adds the missing low branch, one line.

Nobody had to work out, beat by beat, what the total should be; the reference did, for five lines of checker code.

Do it, part 2: dbglab's five bugs against their references

dbglab's HEAD has five failing tests, each from a different bug. Before reading any waveform, run each against references that differ in one known way. The simulator and start-value matrix for all five bugs is in bisection across configurations: three bugs fail identically everywhere (logic), B4 splits on initial values (an unreset register), and B5 splits on how unknown values are modelled. The fourth reference, the commit before each bug, costs one run each:

$ ../tools/before.sh
## 2e63466 Fix comments in the fabric and the DMA; README register map
PASS test=irq_w1c seed=1
## 27acd30 periph: deepen the async FIFO to 16 entries
PASS test=dma_burst_backpressure seed=1
## bc2bb34 dma: fill mode
PASS test=random seed=10
## 03ebdc1 dma: one width for the beat counters
PASS test=random seed=17
## b357bd8 csr: no reset on the DMA address and data registers
PASS test=periph_power seed=1

• The commit before passes every one. Each failure has a good end one commit away, so comparing that pair of runs will find its first difference with almost no noise: the only allowed difference is one commit.

What the evidence says

A reference localises in two ways. Compared often, it gives the first wrong cycle and the inputs there: beat 89, sample -39, a sum just under the floor. Compared across kinds, it says which class of cause to look in before you look at anything: logic, initial state, or unknown values. Both took minutes. The next step is Hypothesise: "the low saturation branch is missing" predicts that every first mismatch follows a sum below -2048, which the lockstep line already shows for one case and a sweep of seeds could test.

Traps

Related topics

• Part 3 · Localise

• Comparing a good and a bad run: the previous release as the reference

• Cone-of-influence tracing: B5 from the stream pins to the cause

• When the bug is in the test

• UVM: debugging the testbench and its scoreboards