Wrong data, one seed in nine

A small fabric connects one master to two slaves and keeps up to two reads in flight. The master needs its responses in the order it asked. In the random test, about one seed in nine reports a response for read #5 while read #4 was still expected. Every directed test passes. Two engineers have each "fixed" it by changing a slave latency, and both times it came back.

The failure needs something to happen in a particular order. Which two events, and can you make them happen in that order every time?

What people usually do, and why it fails

The usual moves are to rerun random seeds and stare at the failing ones, or to tune latencies until the failure stops. The first is slow, because the order that fails is rare. The second is worse: changing a latency moves the window, so the bug still exists and now fails in a different seed, or only in silicon where the latencies are different again.

Another common move is to blame the checker: "the scoreboard is too strict about order". Sometimes that is right; the testbench bugs lesson covers it. Here the master really does need its responses in order, so the design is wrong.

The method: list the events, enumerate the orders

1. Name the two events whose order seems to matter. The failure message usually points at them: here a response for read #5 arriving before the response for read #4.

2. List the orders they can come in: A before B, the same cycle, B before A, and by how many cycles. With latencies, the order is set by the issue times and the latencies together.

3. Make each one a knob. Give the bench plusargs that set each event's timing directly, so one command produces one order.

4. Enumerate. Run every order in a small grid. The cells that fail are the bug's specification: they say exactly when it happens.

5. Predict the boundary from the mechanism and check that the grid agrees, cell for cell.

6. Prove the fix in every cell, then keep the random test: it finds the orders you did not think to list.

This works for the design's own events and for the bench's. Two bench processes acting in the same simulation time step are an ordering problem the simulator decides, which is a race; the races lesson owns that case. The rule for benches is the same: drive and sample away from the clock edge, so there is only one possible order.

The picture: one response overtakes another

Read it left to right, one slot per cycle.

• Slots 0 and 1: read #0 goes to slave 0, read #1 to slave 1, and the fabric accepts both (req_ready is high). This is the bug: the second read targets a different slave while the first is in flight.

• Slot 2: slave 1 is quick and answers with tag 1, while want is still 0. This is the overtaking response the master cannot accept.

• Slot 4: slave 0 answers #0, two cycles too late to be in order.

Do it: from one seed in nine to every time

The fabric below counts reads in flight. The bug lets a read to the other slave through while one is pending. The fix makes such a read wait until the bus is idle; reads to the same slave still overlap, since each slave answers in order. The bench plays the master and both slaves. +SEED runs ten random pairs of reads where each slave is usually quick and once in 150 reads slow; +LA and +LB run one directed pair with chosen latencies; with no plusargs it runs both and checks them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// fab2: one master, two slaves, up to two reads in flight. The master needs its responses in the
// order it asked, and each slave answers in order. The bug: a read to the other slave is let
// through while one is still in flight, so if the second slave is faster its response overtakes.
// The fix: a read to a different slave waits until the bus is idle; reads to the same slave
// still overlap.
module fab2 #(parameter bit FIX = 0) (
  input  logic clk, rst_n,
  input  logic req_valid, req_slave,          // the master's read and the slave it targets
  output logic req_ready,
  input  logic rsp_valid,                     // a response from either slave (one per cycle)
  output logic busy
);
  logic [1:0] outs;                           // reads in flight
  logic       last_q;                         // the slave of the reads in flight
  assign req_ready = (outs < 2'd2) && (!FIX || outs == 2'd0 || req_slave == last_q);
  assign busy = (outs != 2'd0);
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin outs <= '0; last_q <= 1'b0; end
    else begin
      outs <= outs + 2'(req_valid && req_ready) - 2'(rsp_valid);
      if (req_valid && req_ready) last_q <= req_slave;
    end
endmodule

// order_tb: the bench is the master and both slaves. Slave s answers a read after its latency;
// if both answer in the same cycle, slave 0 goes first. The master checks that responses come
// back in request order. +LA=<n> +LB=<n>: one read to slave 0 with latency LA, then one to
// slave 1 with latency LB, the next cycle. +SEED=<n>: 10 random pairs, where each slave is
// usually quick (1-2 cycles) and once in 150 reads slow (6). +FIX=1 uses the fixed fabric.
// With no plusargs it runs the lesson's random sweep and every ordering, and checks them.
`timescale 1ns/1ps

Random: rare and different every time

$ for s in $(seq 1 40); do vvp -n o.vvp +SEED=$s | grep FAIL; done
FAIL seed=1 fix=0: cycle 18: response for read #5 arrived while read #4 was still expected
FAIL seed=4 fix=0: cycle 42: response for read #14 arrived while read #13 was still expected
FAIL seed=6 fix=0: cycle 52: response for read #15 arrived while read #14 was still expected
FAIL seed=7 fix=0: cycle 7: response for read #1 arrived while read #0 was still expected
FAIL seed=14 fix=0: cycle 40: response for read #13 arrived while read #12 was still expected

Five failing seeds in forty, each at a different read and cycle. Every message has the same shape: a later read's response arrives while an earlier one is still expected. That shape names the two events: a response for read n and a response for read n + 1, arriving in the wrong order.

Directed: every order, once

Read #0 goes to slave 0 with latency LA; read #1 goes to slave 1 one cycle later with latency LB. The responses arrive at cycles LA and 1 + LB. Prediction: the run fails exactly when 1 + LB < LA, and passes on a tie because slave 0 wins the response bus.

$ iverilog -g2012 -o o.vvp order_tb.sv && vvp -n o.vvp
random: 23 of 200 seeds fail
every ordering, one read each (ok: in order, OOO: out of order):
      LB=1 LB=2 LB=3 LB=4 LB=5 LB=6
LA=1:   ok   ok   ok   ok   ok   ok
LA=2:   ok   ok   ok   ok   ok   ok
LA=3:  OOO   ok   ok   ok   ok   ok
LA=4:  OOO  OOO   ok   ok   ok   ok
LA=5:  OOO  OOO  OOO   ok   ok   ok
LA=6:  OOO  OOO  OOO  OOO   ok   ok
PASS: the bug needs slave 1's answer to overtake slave 0's; the fix holds in every order
$ vvp -n o.vvp +LA=4 +LB=1
FAIL la=4 lb=1 fix=0: cycle 6: response for read #1 arrived while read #0 was still expected
$ vvp -n o.vvp +LA=4 +LB=1 +FIX=1
PASS la=4 lb=1 fix=1: both responses in order

The random sweep fails 23 seeds in 200, 11.5%. The grid shows OOO (out of order) in ten cells, exactly the triangle below the line 1 + LB = LA, and in no other: LA = 3 with LB = 1 fails, LA = 3 with LB = 2 ties and passes. The prediction held in all 36 cells, so the mechanism is confirmed. +LA=4 +LB=1 now fails at cycle 6 every time; it is the repro. The fixed fabric passes all 36 cells, and all 200 random seeds when run with +FIX=1. Verilator 5.048 printed the same lines.

The same idea in dbglab

dbglab's hidden X (B4) is an ordering bug in the stimulus. Making a flaky test fail found that it needs a zero fill to come before any pattern fill, and raised its rate to 88.5% by biasing the weights. Controlling the order instead of biasing it takes the rate to 100% and 0%. In a scratch copy with a two-fill directed test, the zero fill first fails every seed tried, and the pattern fill first passes every one:

$ for o in 0 1; do for s in 1 2 3; do tb/run.sh fill_order $s icarus +ORDER=$o | tail -1; done; done
FAIL test=fill_order seed=1 cycle=79: SCOREBOARD: RAM word 42 (address 000010a8) read xxxxxxxx, expected 00000000 (last written by DMA run #1)
FAIL test=fill_order seed=2 cycle=79: SCOREBOARD: RAM word 281 (address 00001464) read xxxxxxxx, expected 00000000 (last written by DMA run #1)
FAIL test=fill_order seed=3 cycle=79: SCOREBOARD: RAM word 286 (address 00001478) read xxxxxxxx, expected 00000000 (last written by DMA run #1)
PASS test=fill_order seed=1 cycles=202 time=2016 ns
PASS test=fill_order seed=2 cycles=202 time=2016 ns
PASS test=fill_order seed=3 cycles=202 time=2016 ns

The last lesson of this part shows the test and uses it as the repro for the whole hunt.

What the evidence says

Two events, one order. Once the order is a knob, an 11.5% intermittent is a deterministic failure with a known boundary, and the fix has been checked against every order in the grid rather than against the orders the random test happened to draw. The grid also tells the designer what the fix must guarantee: no response may overtake an earlier one, whatever the latencies.

Traps

Related topics

• Part 8 · Intermittent Failures

• Next: heisenbugs

• Protocol violations

• UVM: debugging the testbench