The report

A programmable delay line has been in use for months at its default delay of 8. It keeps the last 16 samples in a small two-port RAM used as a ring: every cycle it writes the new sample and reads the one written dly cycles ago. A new product needs the maximum delay, 16, so last week the bench's delay loop was extended from 15 to 16. Two engineers ran it, one in each simulator:

$ iverilog -g2012 -o orig.vvp orig.sv && vvp -n orig.vvp
dly=16  31 of 64 samples wrong, first at sample 16
FATAL: orig.sv:56: delay line wrong in 31 samples
...
$ verilator --binary --timing --assert -Wno-fatal --top-module delay_tb -Mdir vl_orig orig.sv > /dev/null 2>&1; vl_orig/Vdelay_tb
dly=16  63 of 64 samples wrong, first at sample 16
[13120000] %Fatal: orig.sv:56: Assertion failed in delay_tb: delay line wrong in 63 samples
...

Delays 1 to 15 pass in both. At 16, Icarus gets about half the samples wrong and Verilator nearly all of them. The thread under the bug is already long: "Icarus is flaky, half right is not a result", "Verilator must be optimising something away", "try a third simulator and take the majority". Here is the design:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// bad-example: the delay line as it was (orig.sv); the bench is the one in the full listing at the end
// delay_line: a programmable delay of 1 to 16 samples, built on a 16-entry two-port RAM used as a
// ring. Every cycle it writes the new sample at wptr and reads the one written DLY cycles ago, so
// dout after edge k is din at edge k - DLY. At DLY = 16 the read and the write hit the same entry:
// the spec says the RAM is read-first (a read returns the value from before the write).
`timescale 1ns/1ps
module delay_line (
  input  logic       clk, rst_n,
  input  logic [4:0] dly,                    // 1 to 16
  input  logic [7:0] din,
  output logic [7:0] dout
);
  logic [7:0] mem [16];
  logic [3:0] wptr, raddr;
  assign raddr = wptr - dly[3:0];            // dly = 16 gives raddr = wptr
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) wptr <= '0;
    else        wptr <= wptr + 1'b1;
  always @(posedge clk) dout <= mem[raddr];  // read port
  always @(posedge clk) mem[wptr] = din;     // write port
endmodule

Stop here and try it. Which simulator is right, and is the design right? Do not vote; find something that decides it.

The hunt

Reproduce: each simulator is deterministic, and they disagree

Rerunning gives the same counts in each simulator, every time. So this is not flakiness in the usual sense: each tool is perfectly repeatable and they simply give different answers to the same question. That pattern has a short list of causes: X or uninitialised state handled differently, or a race, two processes whose order the language leaves open. Bisection across configurations sorts them; here the RAM starts empty in both and no X appears in any output, so the race is the first suspect.

Observe: name the wrong value

A count of wrong samples says little. The bench's +TRACE=16 prints seven samples at the delay that fails, with what came in, what came out and what should have come out:

$ vvp -n orig.vvp +TRACE=16 | head -7
dly=16 sample 16: din=e3 dout=e3 expected 44 WRONG
dly=16 sample 17: din=ed dout=b1 expected b1 ok
dly=16 sample 18: din=53 dout=53 expected 40 WRONG
dly=16 sample 19: din=ef dout=cd expected cd ok
dly=16 sample 20: din=ae dout=ae expected ef WRONG
dly=16 sample 21: din=90 dout=9c expected 9c ok
dly=16 sample 22: din=b4 dout=b4 expected c8 WRONG
$ vl_orig/Vdelay_tb +TRACE=16 | head -7
dly=16 sample 16: din=e3 dout=e3 expected 44 WRONG
dly=16 sample 17: din=ed dout=ed expected b1 WRONG
dly=16 sample 18: din=53 dout=53 expected 40 WRONG
dly=16 sample 19: din=ef dout=ef expected cd WRONG
dly=16 sample 20: din=ae dout=ae expected ef WRONG
dly=16 sample 21: din=90 dout=90 expected 9c WRONG
dly=16 sample 22: din=b4 dout=b4 expected c8 WRONG

Two things jump out, and neither needs a waveform.

• Every wrong value is the current input. dout equals din of the same sample: e3, 53, ae, b4. The delay line is not producing garbage; it is producing a delay of 0 where 16 was asked for. The output read the entry after this edge's write had already replaced it.

• Icarus alternates. Wrong, right, wrong, right, on consecutive edges of the same design, with the same inputs. Hardware does not do that. Something in the simulation is changing from one edge to the next, and it is not the design.

Localise: what is special about 16

The comment says it: at dly = 16, raddr = wptr - 0 = wptr. The read port and the write port address the same entry on the same edge. At every other delay they never meet, which is why 1 to 15 pass in both simulators and why the bug lived for months at the default of 8. The spec answers what should happen when they meet: the RAM is read-first, so the read returns the old value, the sample from 16 edges ago.

Now read how the two ports are written. They are two separate processes on the same edge. The read port uses a non-blocking assignment to dout, but it reads mem immediately. The write port updates mem with a blocking assignment, immediately. So the value the read port sees depends on which process the simulator runs first, and the language does not say which. That is a race.

Hypothesise and test: predict what a race would do

Hypothesis: a race between the two port processes on one shared variable, mem. If that is right, three things follow that a real logic bug would not do:

First, the order. A copy of orig.sv adds one $display at the top of each port process, for four edges at dly = 16:

$ vvp -n oo.vvp +TRACE=16 | head -10                 # Icarus
12465 ns  write port: mem[14] = c5
12465 ns  read  port: mem[14] = c5
12475 ns  read  port: mem[15] = aa
12475 ns  write port: mem[15] = 11
12485 ns  write port: mem[0] = e3
12485 ns  read  port: mem[0] = e3
dly=16 sample 16: din=e3 dout=e3 expected 44 WRONG
12495 ns  read  port: mem[1] = b1
12495 ns  write port: mem[1] = ed
dly=16 sample 17: din=ed dout=b1 expected b1 ok

Icarus runs the two processes in an order that flips on every edge. Where the write ran first (12485 ns) the sample is wrong; where the read ran first (12495 ns) it is right. The alternation in the trace is the scheduler, not the design. Now the same copy in Verilator:

$ vl_oo/Vdelay_tb +TRACE=16 | head -8                # Verilator
12465 ns  read  port: mem[14] = b4
12465 ns  write port: mem[14] = c5
12475 ns  read  port: mem[15] = aa
12475 ns  write port: mem[15] = 11
12485 ns  read  port: mem[0] = 44
12485 ns  write port: mem[0] = e3
dly=16 sample 16: din=e3 dout=e3 expected 44 WRONG
12495 ns  read  port: mem[1] = b1

Read this one slowly. The print in the read port says it saw 44, the right value, and yet dout became e3. The print and the assignment one line below it did not see the same memory. In a race the simulator owes you no particular order, not even a consistent one between a $display and the statement after it, so a print inside a racing process tells you about the print. It is still useful here, because Icarus's half of the experiment is clear, but it is not evidence of what the register received.

Second, move one line. orig_swapped.sv puts the write port above the read port and changes nothing else:

$ iverilog -g2012 -o orig_swapped.vvp orig_swapped.sv && vvp -n orig_swapped.vvp | head -1
dly=16  32 of 64 samples wrong, first at sample 17
$ vl_orig_swapped/Vdelay_tb | head -1
dly=16  63 of 64 samples wrong, first at sample 16

Icarus's answer moves (32 wrong, starting at sample 17 instead of 16; it has shifted the alternation by one edge). Verilator's does not. A design whose behaviour depends on the order of lines in the file is not describing hardware. That settles it as a race.

Third, the hardware's own answer. The design alone through Yosys, then the netlist with the same bench:

$ sed '/^module delay_tb/,$d' orig.sv > orig_design.sv
$ yosys -q -p "read_verilog -sv orig_design.sv; synth -flatten -top delay_line; rename delay_line delay_line_net; write_verilog -noattr orig_net.v"
$ iverilog -g2012 -o on.vvp orig_net.v orig_wrap.sv orig_tb.sv && vvp -n on.vvp +TRACE=16 | head -8
dly=16 sample 16: din=e3 dout=44 expected 44 ok
dly=16 sample 17: din=ed dout=b1 expected b1 ok
dly=16 sample 18: din=53 dout=40 expected 40 ok
dly=16 sample 19: din=ef dout=cd expected cd ok
dly=16 sample 20: din=ae dout=ef expected ef ok
dly=16 sample 21: din=90 dout=9c expected 9c ok
dly=16 sample 22: din=b4 dout=c8 expected c8 ok
PASS: delays 1 to 16, every sample checked

Three tools, three answers to one RTL file: half wrong, nearly all wrong, all right. Synthesis has no processes to order, so it builds a RAM whose read sees the old contents, and it gives no warning. The chip would have worked; every simulation of it would have failed, differently. That is the worst kind of mismatch to leave in place, because the next bug in this block will be debugged in a simulator that does not match the silicon. (Races and simulation-synthesis mismatches shows the opposite case, where the netlist is the one that is wrong.)

Lint had said so all along:

$ verilator --lint-only -Wall --top-module delay_line orig_design.sv
...
%Warning-BLKSEQ: orig_design.sv:19:35: Blocking assignment '=' in sequential logic process
                                     : ... note: In instance 'delay_line'
                                     : ... Suggest using delayed assignment '<='
...

The fix and the lesson

Fix: one character, and the order stops mattering

// snippet: the fix, the write port
  always @(posedge clk) mem[wptr] <= din;    // non-blocking: written after every process has read

With a non-blocking write, mem changes only after every process triggered by the edge has run, so the read port always sees the old contents, whatever the order. That is the read-first behaviour the spec asks for and the netlist already had. Here is the full listing, with FIX choosing the original write port (0) or the corrected one (1, the default):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

// delay_line: a programmable delay of 1 to 16 samples, built on a 16-entry two-port RAM used as a
// ring. Every cycle it writes the new sample at wptr and reads the one written DLY cycles ago, so
// dout after edge k is din at edge k - DLY. At DLY = 16 the read and the write hit the same entry:
// the spec says the RAM is read-first (a read returns the value from before the write).
// FIX = 0 is the original: the RAM write uses a blocking assignment in its own always block.
`timescale 1ns/1ps
module delay_line #(parameter bit FIX = 1) (
  input  logic       clk, rst_n,
  input  logic [4:0] dly,                    // 1 to 16
  input  logic [7:0] din,
  output logic [7:0] dout
);
  logic [7:0] mem [16];
  logic [3:0] wptr, raddr;
  assign raddr = wptr - dly[3:0];            // dly = 16 gives raddr = wptr
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) wptr <= '0;
    else        wptr <= wptr + 1'b1;
  always @(posedge clk) dout <= mem[raddr];  // read port
  if (FIX) begin : g_fix
    always @(posedge clk) mem[wptr] <= din;  // write port, non-blocking
  end else begin : g_orig
    always @(posedge clk) mem[wptr] = din;   // write port, blocking
  end
endmodule

Prove: all three tools agree

$ iverilog -g2012 -o d1.vvp delay_tb.sv && vvp -n d1.vvp
PASS: delays 1 to 16, every sample checked
...
$ verilator --binary --timing --assert -Wno-fatal --top-module delay_tb -Mdir vl1 delay_tb.sv > /dev/null 2>&1; vl1/Vdelay_tb
PASS: delays 1 to 16, every sample checked
...
$ vvp -n n1.vvp | head -1                            # the FIX = 1 netlist, built as above
PASS: delays 1 to 16, every sample checked

Both simulators pass every delay, the netlist still passes, and the +TRACE=16 output is now identical in all three. Agreement between the tools is the real proof here: a race fix that makes one simulator pass and leaves another failing has not removed the race. The lint warning is gone too.

Prevent: catch the race before the corner does

• Make BLKSEQ an error in the lint run of every block: a blocking assignment in a clocked process is almost never what the author meant, and here it was the bug.

• Run the regression in two simulators and treat any disagreement as a bug, not as noise. Here the two tools disagreed on the first run that reached the corner; a single-simulator flow would have shown "31 of 64" and invited a guess.

• Test the corner the structure creates. A ring of 16 and a delay of up to 16 means read and write meet at the maximum. The bench stopped at 15 because the product used 8; derive the test list from the design's limits, not from today's use.

• The review question for every memory model: when a read and a write hit the same address on one edge, what does the spec say, and which line of the RTL guarantees it?

Which step won? Hypothesise and Test: the race was a guess after Localise, and it became a fact when moving one line changed one simulator's answer and not the design. Observe made the guess easy, by naming the wrong value: this edge's input, a delay of 0.

Related topics

• Part 10 · Case Studies

• Races and simulation-synthesis mismatches

• Testbench bugs: a drive race on the design's edge

• Blaming the tool, checked properly

• RTL: blocking vs non-blocking, race-aware reasoning

• RTL: inferring flip-flops and RAM

• SystemVerilog: anatomy of a race