"Icarus is broken"

A message in the team chat: "burst_fsm times out in Icarus and passes in Verilator. The FSM is trivial, so it's an Icarus bug. Can we drop Icarus from the regression?" Attached, two real runs of the same file:

$ iverilog -g2012 -o naive.vvp naive.sv && vvp -n naive.vvp
FATAL: naive.sv:38: TIMEOUT: no done 100 cycles after start (state=xx)
       Time: 1040000  Scope: tb
$ verilator --binary --timing -Wno-fatal --top-module tb -Mdir vl naive.sv > vl.log 2>&1 && vl/Vtb
PASS: done 8 cycles after start
...

The author has evidence for their view: one tool passes, the FSM "looks right", and Icarus does lack some SystemVerilog features. What would you need to see before agreeing?

What people usually do, and why it fails

They collect support. The passing run is support. The code review ("three states, what could go wrong?") is support. A forum post about an Icarus limitation is support. Nobody runs the experiment that could show the claim is wrong, because the claim already feels proven. This is confirmation bias: we look for, notice and remember evidence that fits what we already believe, and we explain away the rest.

In debugging it has two common forms. Blaming the tool when two tools disagree. And seeing the expected bug in a result that is really a different one. Both end the same way: the hypothesis is never tested, because every result was read as agreeing with it.

The method: try to break your own idea

1. State the claim so it can fail. "Icarus mis-simulates this case statement" predicts that a minimal case statement gives a result the language forbids.

2. Ask what you would see if you were wrong, and run that first. For a tool claim: if the design is at fault, a different model of the same chip (random start values) should fail too.

3. Shrink until it fits on a screen. A tool difference that survives shrinking to ten lines is easy to judge; one that vanishes when you shrink was the design.

4. Name the rule. Which sentence of the language does the tool break? "It should obviously do X" is not a rule.

5. Distinguish freedom from error. Two-state and four-state simulators are allowed to differ on X; race-prone code is allowed to give different orders. A difference inside the language's freedom is your bug.

6. Only then file it, with the program, both outputs, the versions and the rule.

The picture: is it the tool?

Do it: the FSM that "only fails in Icarus"

The design as written

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

// bad-example: burst_fsm with the bench its designer wrote. It passes in Verilator and times out in Icarus.
`timescale 1ns/1ps
module burst_fsm (
  input  logic clk, rst_n, start,
  output logic busy, done
);
  localparam logic [1:0] IDLE = 2'd0, RUN = 2'd1, DONE = 2'd2;
  logic [1:0] state;
  logic [2:0] beat;
  // No reset on state: "IDLE is 0, and flip-flops come up at 0".
  always @(posedge clk)
    case (state)
      IDLE:    if (start) state <= RUN;
      RUN:     if (beat == 3'd7) state <= DONE;
      DONE:    state <= IDLE;
      default: ;                                  // "unreachable"
    endcase
  always @(posedge clk or negedge rst_n)
    if (!rst_n)            beat <= '0;
    else if (state == RUN) beat <= beat + 3'd1;
  assign busy = (state == RUN);
  assign done = (state == DONE);
endmodule

module tb;
  logic clk = 1'b0, rst_n = 1'b0, start = 1'b0, busy, done;
  int   cyc = 0;
  always #5 clk = ~clk;
  burst_fsm dut (.*);
  initial begin
    repeat (2) @(posedge clk);
    rst_n = 1'b1;
    @(negedge clk) start = 1'b1;
    @(negedge clk) start = 1'b0;
    while (done !== 1'b1) begin
      @(negedge clk);
      cyc++;
      if (cyc > 100) $fatal(1, "TIMEOUT: no done 100 cycles after start (state=%b)", dut.state);
    end
    $display("PASS: done %0d cycles after start", cyc);
    $finish;
  end
endmodule

The Icarus run already said something the chat message skipped: state=xx. The state register is unknown 100 cycles after start.

Shrink it until it fits on a screen

Ten lines that keep only the suspected construct, a case on an unknown selector:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

// case_x_tb: the smallest program that shows the difference. A two-bit selector that is
// unknown, a case with three items and a default. Which branch runs?
`timescale 1ns/1ps
module case_x_tb;
  logic [1:0] s;                 // never assigned: X in a four-state simulator
  initial begin
    #1;
    case (s)
      2'd0:    $display("s=%b took item 2'd0", s);
      2'd1:    $display("s=%b took item 2'd1", s);
      2'd2:    $display("s=%b took item 2'd2", s);
      default: $display("s=%b took the default", s);
    endcase
    $display("PASS: case on s ran one branch");
    $finish;
  end
endmodule

$ iverilog -g2012 -o cx.vvp casex_min.sv && vvp -n cx.vvp
s=xx took the default
PASS: case on s ran one branch
...
$ verilator --binary --timing -Wno-fatal --top-module case_x_tb -Mdir vlc casex_min.sv > vlc.log 2>&1 && vlc/Vcase_x_tb
s=00 took item 2'd0
PASS: case on s ran one branch
...

The difference survives shrinking, so check the rule. A case compares the selector with each item using case equality, where X and Z bits must match exactly. xx matches none of 00, 01, 10, so the default runs. That is what Icarus did. Verilator is a two-state simulator by design: it has no X, and an unassigned variable starts at zero unless you ask for random values. That is what Verilator did. Both tools followed their rules. The difference is inside the language's freedom about unknown values, which by the flow above means: suspect the design.

The run that could prove the design innocent

If the design were fine and Icarus wrong, then a two-state simulator that powers the state register up at random values, as a chip does, should always pass. That is the disconfirming experiment the chat message never ran:

$ verilator --binary --timing -Wno-fatal --x-assign unique --x-initial unique --top-module tb -Mdir vlx naive.sv > vlx.log 2>&1
$ for n in $(seq 1 20); do printf "seed %2d: " $n; vlx/Vtb +verilator+rand+reset+2 +verilator+seed+$n 2>&1 | grep -E "PASS|TIMEOUT" | head -1; done
seed  1: PASS: done 8 cycles after start
seed  2: [1040000] %Fatal: naive.sv:38: Assertion failed in tb: TIMEOUT: no done 100 cycles after start (state=11)
...
seed  6: PASS: done 6 cycles after start
...
seed 14: [1040000] %Fatal: naive.sv:38: Assertion failed in tb: TIMEOUT: no done 100 cycles after start (state=11)
...

Four of twenty power-ups hang in state 11, the encoding no item names, where default: ; holds it for ever. Four more finish in 6 cycles instead of 8: they powered up in RUN, were already counting a burst nobody asked for, and swallowed the real start. Twelve pass. The chip would hang on some power-ups. Icarus showed it as X; the passing Verilator run was the one hiding it.

Prove it in one bench that passes in both simulators

Rather than rely on random start values, power the design up in each of its four states on purpose and check what it does. The same bench checks the fix: a reset on state, and a default that returns to IDLE.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

// powerup_tb: is it the simulator or the design? Power burst_fsm up in each of its four
// possible states (what a chip can do) and watch whether it finishes a burst. The same
// check runs on the fix: a reset on state and a default that recovers.
`timescale 1ns/1ps
module burst_fsm (                                  // as written: no reset on state
  input  logic clk, rst_n, start,
  output logic busy, done
);
  localparam logic [1:0] IDLE = 2'd0, RUN = 2'd1, DONE = 2'd2;
  logic [1:0] state;
  logic [2:0] beat;
  always @(posedge clk)
    case (state)
      IDLE:    if (start) state <= RUN;
      RUN:     if (beat == 3'd7) state <= DONE;
      DONE:    state <= IDLE;
      default: ;                                    // "unreachable"
    endcase
  always @(posedge clk or negedge rst_n)
    if (!rst_n)            beat <= '0;
    else if (state == RUN) beat <= beat + 3'd1;
  assign busy = (state == RUN);
  assign done = (state == DONE);
endmodule

module burst_fsm_fixed (                            // reset on state; default recovers
  input  logic clk, rst_n, start,
  output logic busy, done
);
  localparam logic [1:0] IDLE = 2'd0, RUN = 2'd1, DONE = 2'd2;
  logic [1:0] state;
  logic [2:0] beat;
  always @(posedge clk or negedge rst_n)
    if (!rst_n) state <= IDLE;
    else
      case (state)
        IDLE:    if (start) state <= RUN;
        RUN:     if (beat == 3'd7) state <= DONE;
        DONE:    state <= IDLE;
        default: state <= IDLE;
      endcase
  always @(posedge clk or negedge rst_n)
    if (!rst_n)            beat <= '0;
    else if (state == RUN) beat <= beat + 3'd1;
  assign busy = (state == RUN);
  assign done = (state == DONE);
endmodule

$ iverilog -g2012 -o pu.vvp powerup.sv && vvp -n pu.vvp
power-up state 0: as written, done after 8 cycles; fixed, after 8
power-up state 1: as written, done after 6 cycles; fixed, after 8
power-up state 2: as written, done after 8 cycles; fixed, after 8
power-up state 3: as written, done after -1 cycles; fixed, after 8
PASS: as written, stuck from state 3 and early from state 1; fixed, 8 cycles from every state
...
$ verilator --binary --timing --assert -Wno-fatal --top-module powerup_tb -Mdir vlp powerup.sv > vlp.log 2>&1 && vlp/Vpowerup_tb
power-up state 0: as written, done after 8 cycles; fixed, after 8
...
power-up state 3: as written, done after -1 cycles; fixed, after 8
PASS: as written, stuck from state 3 and early from state 1; fixed, 8 cycles from every state
...

Same lines in both simulators. Once the power-up value is fixed by the bench, the tools agree completely, which is the strongest sign the original difference was never theirs. The bug is the missing reset, made worse by a default that holds an illegal state instead of leaving it.

Seeing the bug you expect: two simulators, two bugs

The second form of bias is quieter. dbglab's random test on seed 16 fails in both simulators, differently:

$ tb/run.sh random 16
dbglab: test=random seed=16
FAIL test=random seed=16 cycle=686: SCOREBOARD: RAM word 759 (address 00001bdc) read xxxxxxxx, expected 00000000 (last written by DMA run #3)
$ tb/run.sh random 16 verilator
dbglab: test=random seed=16
FAIL test=random seed=16 cycle=12811: STREAM: word #203 = 5484683d, expected 3caa023d

An engineer who knows the FILL bug and knows that Verilator hides X writes the obvious story: same bug; two-state Verilator turns the X into a value, so the damage shows up later, on the stream. It fits. What would you see if it were wrong? The FILL fix would not change the Verilator failure. So run that:

The Verilator failure ignores the FILL fix and vanishes with the counter fix. The two simulators were showing two different bugs: Icarus the FILL bug, which fires first, and two-state Verilator, where FILL starts at the right value, the counter bug further on. The expected story would have closed the counter failure as a duplicate. The run took a minute because the story was turned into a prediction that could fail.

If it really is the tool

Tools do have bugs, and Icarus 13 does reject or mis-handle some SystemVerilog constructs. When the flow above ends at "rule found", the report is short and complete:

A report like that gets fixed. "My 40,000-line design behaves differently in your simulator" gets closed.

Traps

Related topics

• Part 4 · Hypothesise and Test

• X in a waveform: optimism, pessimism and two-state simulators

• Bisection across configurations: the simulator matrix

• RTL: FSM bug stories

• Next: one change at a time