The FIFO that returns stale data

A 4-entry FIFO's scoreboard reports pop read 19, expected 1d. The data are sequence numbers, so the FIFO seems to have handed back an old word. The designer says the FIFO is fine and the test is wrong. The verification engineer says the test has passed for weeks and the FIFO changed yesterday. Both are sure. Who is right, and how would you know without taking sides?

What people usually do, and why it fails

Two habits, both bad. The first treats every FAIL as a design bug, and the designer loses a day explaining a waveform the bench misread. The second treats every inconvenient FAIL as a bench bug and edits the expected value until the test is green. That one is worse: it ships the bug and leaves a check that agrees with it.

Both skip the same question. A FAIL line says the design and the bench disagree. It does not say which side is wrong. Only the spec can, and only at a cycle you can point to.

The method: check the design, then check the checker

On the debug loop this is still Localise: the bench is part of the search space. Work through it in order:

1. Find the first wrong cycle from the checker's report, as for any failure.

2. Ask what the spec requires there. Read the spec, the register map or the protocol rule, not the RTL and not the checker's code.

3. Check the design against it. If the design did what the spec requires, the design is cleared for this failure.

4. Then check the checker. Where did the expected value come from? Which inputs did the model see, and when? A checker that records the provenance of each expected value answers this in one line.

5. Fix the side that is wrong, and prove the checker on a broken design. Run the fixed checker against a design with a deliberate, real bug (a mutant). It must fail. If it passes, the fix blinded it.

Most bench bugs that look like design bugs fall into a few shapes:

The picture

Do it, part 1: one FIFO, three scoreboards

The FIFO's spec says a push while full is dropped and sets a sticky ovf flag. The bench below runs the same random traffic into two FIFOs: the correct one, and a mutant whose full flag rises one entry early (a real bug: it drops words it had room for). Three scoreboards watch each, identical except for when they accept a push into their model. Each records where every expected word came from.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

// fifo4: a 4-entry FIFO. Spec: a push while full is dropped, the contents are kept, and the
// sticky flag ovf is set. A pop while empty is ignored. dout shows the oldest entry.
// FULL_AT is the count at which full asserts: 4 is correct; 3 makes a mutant with a real bug.
`timescale 1ns/1ps
module fifo4 #(parameter int FULL_AT = 4) (
  input  logic       clk, rst_n, push, pop,
  input  logic [7:0] din,
  output logic [7:0] dout,
  output logic       full, empty, ovf
);
  logic [7:0] mem [4];
  logic [1:0] wp, rp;
  logic [2:0] count;
  assign full  = (count == FULL_AT);
  assign empty = (count == 0);
  assign dout  = mem[rp];
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin wp <= '0; rp <= '0; count <= '0; ovf <= 1'b0; end
    else begin
      if (push && !full) begin mem[wp] <= din; wp <= wp + 1'b1; end
      if (push && full)  ovf <= 1'b1;
      if (pop && !empty) rp <= rp + 1'b1;
      count <= count + (push && !full) - (pop && !empty);
    end
endmodule

// sb: a scoreboard for fifo4. MODE says when it accepts a push into its model:
//   0 (A): on every push, as the bench was first written
//   1 (B): when the design's full flag is low, borrowing the design's opinion
//   2 (C): when its own model has room, the spec computed independently
module sb #(parameter int MODE = 0) (
  input logic       clk, rst_n, push, pop, full,
  input logic [7:0] din, dout
);
  logic [7:0] q [$];
  string      why [$];                              // where each expected word came from
  logic [7:0] exp;
  string      from, msg;
  bit         accept;
  int         held, cycle = 0, first = -1;
  always @(posedge clk) if (rst_n) begin            // inputs are stable here: the bench drives at negedge
    held   = q.size();
    accept = (MODE == 0) ? push : (MODE == 1) ? push && !full : push && held < 4;
    if (pop && held > 0) begin
      exp  = q.pop_front();
      from = why.pop_front();
      if (dout !== exp && first < 0) begin
        first = cycle;
        msg = $sformatf("cycle %0d: pop read %h, expected %h (%s)", cycle, dout, exp, from);
      end
    end
    if (accept) begin
      q.push_back(din);
      why.push_back($sformatf("pushed at cycle %0d with %0d held, full=%b", cycle, held, full));
    end
    cycle++;
  end
endmodule

$ iverilog -g2012 -o f.vvp fifo_sb_tb.sv && vvp -n f.vvp
good design,   scoreboard A: cycle 55: pop read 19, expected 1d (pushed at cycle 51 with 4 held, full=1)
good design,   scoreboard B: clean
good design,   scoreboard C: clean
mutant design, scoreboard A: cycle 22: pop read 0c, expected 0b (pushed at cycle 19 with 3 held, full=1)
mutant design, scoreboard B: clean
mutant design, scoreboard C: cycle 22: pop read 0c, expected 0b (pushed at cycle 19 with 3 held, full=1)
good design: ovf=1; mutant: ovf=1
PASS: A blames a correct design, B misses a real bug, C gets both right
...

Read it as the design engineer would, then as the checker's reviewer

The first line is the report from the scene. Taken at face value it says the FIFO returned word 19 when 1d was due: stale data, a read-pointer bug. Now apply the method. The expected word 1d was pushed at cycle 51, when the FIFO held 4 words and `full` was 1. The spec says that push is dropped and ovf is set; ovf is 1. The design did exactly what the spec requires. Scoreboard A accepted a word the design was right to refuse, and when the FIFO later ran empty, A still expected 1d and compared it with whatever dout showed. Test bug.

Now the fifth line, scoreboard C on the mutant. Its message has the same shape: a pop read a different word from the one expected. But its provenance reads differently: word 0b was pushed when the FIFO held 3 words, with full already 1. The spec says a 4-entry FIFO has room for a fourth word, so full at 3 is wrong and the drop is a lost word. Design bug. Same symptom, opposite verdict, and the spec decided both in one line each.

The middle rows show the other two ways a checker fails:

• Scoreboard A on the mutant also fails, at the same cycle as C. A broken checker can be right by accident; that is no reason to trust it.

• Scoreboard B passes the mutant. It accepts a push only when the design's full is low, so it agrees with the design about exactly the signal that is wrong. Every design signal a checker reads is a signal it no longer checks.

• Scoreboard C computes fullness from its own model and the spec, so it passes the correct design and catches the mutant. That last check, a fixed checker failing on a real bug, is step 5 of the method.

Do it, part 2: dbglab's own bench on trial

dbglab's irq_w1c test fails at HEAD with a message that names the bench's own model:

$ tb/run.sh irq_w1c 1
dbglab: test=irq_w1c seed=1
FAIL test=irq_w1c seed=1 cycle=84: IRQ: irq=1 while CTRL=00 enables no interrupt
$ grep -n "done_ie" README.md
55:| 0x008 | CTRL | RW | [0] done_ie, [1] err_ie, [2] paced, [3] dst_fixed, [4] fill. Reset 0 |
$ grep -n "CTRL_RST    =" rtl/dbglab_pkg.sv
39:  localparam logic [7:0]  CTRL_RST    = 8'h01;
$ grep -n "m_ctrl    = 8'h00" tb/dbglab_tb.sv
68:  logic [7:0]  m_ctrl    = 8'h00;

CTRL=00 is the bench's model of the register, not the register. The bench and the RTL disagree about CTRL's reset value: the model says 00, the RTL says 01. Here the spec is the register map, and it says Reset 0. The design is wrong and the bench is right (bisection in history finds the commit that changed it).

Now see why the bench was right. Its header says The bench keeps its own copy of every reset value: it must not share them with the RTL. Break that rule in a scratch copy and the bug disappears:

$ git diff
...
-  logic [7:0]  m_ctrl    = 8'h00;
+  logic [7:0]  m_ctrl    = CTRL_RST;     // shared with the RTL
...
$ tb/run.sh irq_w1c 1
dbglab: test=irq_w1c seed=1
PASS test=irq_w1c seed=1 cycles=316 time=3156 ns
$ tb/run.sh irq_w1c 1 verilator
dbglab: test=irq_w1c seed=1
PASS test=irq_w1c seed=1 cycles=316 time=3156 ns

Green in both simulators, with the design still wrong. A checker that reads its expected values from the design's own package checks nothing about them. This is scoreboard B's flaw in another form, and it is exactly the edit a hurried engineer makes to "fix the test".

What the evidence says

In both parts the question that settled it was the same: what does the spec say at this cycle, and where did the checker's expected value come from? The FIFO's correct design passed its spec at cycle 51, so the bench was wrong; dbglab's design failed its register map at reset, so the design was wrong. Neither verdict needed anyone's confidence. The next step for a bench bug is the same loop in the bench: hypothesise and test, fix, and prove the fix on a mutant.

Traps

Related topics

• Part 3 · Localise

• Testbench bugs that look like design bugs

• UVM: debugging the testbench

• Case study: the testbench that was wrong

• Differential debugging: references and their limits