The report

A sensor block writes numbered samples, 01, 02, 03 and so on, into an eight-entry dual-clock FIFO at 100 MHz. A filter on a 137 MHz clock reads them in short bursts: it reads every word the FIFO's level says is there, up to four at a time. The prototype team's note, paraphrased (illustrative): every few minutes the filter's sequence check fires; it got 77 right after 7e. RTL regression is green.

The regression really is green. This is the block's own bench with no options, and the line for the design as built:

$ iverilog -g2012 -o p.vvp ptr_tb.sv && vvp -n p.vvp
binary, no skew : 6474 reads, 0 errors
...

6,474 words read at random rates, every one in order. Yet the lab sees 77 after 7e, and 77 is a number the writer sent a while ago, not next.

Stop here if you want to try it yourself. The design is below, in the hunt. What would make a correct-looking FIFO hand back an old word, and why can simulation not see it?

The hunt

Reproduce: why plain RTL simulation cannot fail

In RTL simulation, when the writer's pointer goes from 1101 to 1110, all four bits change at the same instant, and the reader's flop sees either the old value or the new one. On silicon or an FPGA, each bit travels its own wire. Clock-crossing paths are normally excluded from timing analysis, so nothing makes those wires equal. Each bit can arrive at a different time, and a read clock edge can land between them. Metastability, covered in CDC bugs, is a second way for bits of one word to disagree; this case uses the simpler one.

No seed can produce that, so the first job is a model that can, as in when simulation cannot reproduce it. skew_wires sits between the domains and delays bit i of each crossing pointer by (3 − i) × 0.4 ns: bit 3 arrives first, bit 0 last, 1.2 ns apart. That is all it does. It does not model metastability, it does not claim any real layout has these delays, and it cannot give a failure rate. It answers one question: if the bits arrive at different times, does this design break?

The bench runs five copies of the FIFO side by side. Rig 2 is the design as built with the skew on; the others are controls the hunt uses later.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119

`timescale 1ns/1ps
// ptr_fifo: an 8-entry dual-clock FIFO. The write side (wclk) sends its write pointer to the read
// side, and the read side sends its read pointer back, each through two flops in the other domain.
// The reader works in bursts: it reads every word the level says is there, up to 4 in a row.
//   FIX = 0: the pointers cross in binary, one synchroniser per bit.
//   FIX = 1: the pointers cross in Gray code (one bit changes per step) and are converted back.
// The crossing pointers leave on *_x and come back on *_x_in, so the bench can put wires between.
module ptr_fifo #(parameter bit FIX = 0) (
  input  logic       wclk, rclk,
  input  logic       wr,
  input  logic [7:0] wdata,
  output logic [7:0] rdata,
  output logic       full, rd,
  output logic [3:0] level,                   // words the reader believes are there
  output logic [3:0] wptr_x, rptr_x,          // to the other domain
  input  logic [3:0] wptr_x_in, rptr_x_in     // after the wires
);
  logic [7:0] mem [8];
  logic [3:0] wptr = '0, rptr = '0, ws1 = '0, ws2 = '0, rs1 = '0, rs2 = '0, rptr_a;
  function automatic logic [3:0] bin2gray(input logic [3:0] b); return b ^ (b >> 1); endfunction
  function automatic logic [3:0] gray2bin(input logic [3:0] g);
    logic [3:0] b;
    b[3] = g[3];
    for (int i = 2; i >= 0; i--) b[i] = b[i + 1] ^ g[i];
    return b;
  endfunction
  // write side
  assign wptr_x = FIX ? bin2gray(wptr) : wptr;
  always @(posedge wclk) begin
    {rs2, rs1} <= {rs1, rptr_x_in};           // two flops per bit
    if (wr && !full) begin mem[wptr[2:0]] <= wdata; wptr <= wptr + 4'd1; end
  end
  assign rptr_a = FIX ? gray2bin(rs2) : rs2;
  assign full   = (wptr == {~rptr_a[3], rptr_a[2:0]});
  // read side
  logic [3:0] wptr_b;
  logic [2:0] left = '0;                      // words still to read in this burst
  assign rptr_x = FIX ? bin2gray(rptr) : rptr;
  assign wptr_b = FIX ? gray2bin(ws2) : ws2;  // the write pointer as the read side sees it
  assign level  = wptr_b - rptr;
  assign rd     = (left != 0);
  assign rdata  = mem[rptr[2:0]];
  always @(posedge rclk) begin
    {ws2, ws1} <= {ws1, wptr_x_in};           // two flops per bit
    if (rd) begin rptr <= rptr + 4'd1; left <= left - 3'd1; end
    else if (level != 4'd0)                   // start a burst: every word it sees, at most 4
      left <= (level > 4'd4) ? 3'd4 : level[2:0];
  end
endmodule

// skew_wires: the routing between the domains. Bit i arrives (3 - i) * SKEW ns after it leaves:
// bit 3 first, bit 0 last. This is a model of unequal wire delays only. It is not a model of
// metastability, and SKEW = 0 is what plain RTL simulation does.
module skew_wires #(parameter real SKEW = 0.0) (input logic [3:0] a, output logic [3:0] y);
  for (genvar i = 0; i < 4; i++) begin : g
    if (i == 3) assign y[i] = a[i];
    else        assign #((3 - i) * SKEW) y[i] = a[i];
  end
endmodule

// rig: one FIFO, its wires, a writer that sends 1, 2, 3, ... and a check on every word read.
module rig #(parameter bit FIX = 0, parameter real SKEW = 0.0, parameter int ID = 0) (
  input logic wclk, rclk, input logic [31:0] seed, input int show, trace_from, trace_to
);
  logic       wr = 1'b0, full, rd;
  logic [7:0] wdata = 8'd1, rdata, expect_v = 8'd1;
  logic [3:0] wx, rx, wx_in, rx_in, level;
  int unsigned errors = 0, reads = 0, rcyc = 0, rng = 0;
  string first = "";
  ptr_fifo #(.FIX(FIX)) u (.wclk, .rclk, .wr, .wdata, .rdata, .full, .rd, .level,
    .wptr_x(wx), .rptr_x(rx), .wptr_x_in(wx_in), .rptr_x_in(rx_in));
  skew_wires #(.SKEW(SKEW)) w_w (.a(wx), .y(wx_in));
  skew_wires #(.SKEW(SKEW)) w_r (.a(rx), .y(rx_in));
  int unsigned run = 0, gap = 0;
  always @(posedge wclk) begin                 // the writer: packets of 1 to 6 words, 4 to 20 cycles apart
    if (rng == 0) rng = seed * 32'h9E37_79B9 + 32'(ID) * 32'h85EB_CA6B + 32'h7F4A_7C16;
    if (wr && !full) begin wdata <= wdata + 8'd1; run--; end
    if (run == 0 && gap == 0) begin
      rng ^= rng << 13; rng ^= rng >> 17; rng ^= rng << 5;
      run = 1 + rng % 6;
      gap = 4 + (rng >> 8) % 17;
    end
    if (run == 0) gap--;
    wr <= (run != 0);
  end
  // the rule a crossing pointer must keep, checked where it leaves: at most one bit changes per
  // write clock. It needs no skew model, so it can run in every plain RTL simulation.
  logic [3:0] wx_prev = '0, chg;
  string      onebit = "";
  always @(posedge wclk) begin
    chg = wx ^ wx_prev;
    if (onebit == "" && (chg & (chg - 4'd1)) != 4'd0)   // more than one bit changed
      onebit = $sformatf("%0.2f ns: the crossing write pointer went %b -> %b", $realtime, wx_prev, wx);
    wx_prev = wx;
  end
  always @(posedge rclk) begin                 // values just before the edge
    rcyc++;
    if (trace_to > 0 && show == ID && rcyc >= trace_from && rcyc <= trace_to) begin
      string got;
      got = "";
      if (rd) got = $sformatf("   %h", rdata);
      $display("%5d %8.2f   %b   %b  %b  %b  %2d    %0d%s", rcyc, $realtime, u.wptr, wx_in, u.ws2, u.rptr,
               level, u.left, got);
    end
    if (rd) begin                              // check each word as it is read
      reads++;
      if (rdata !== expect_v) begin
        errors++;
        if (first == "")
          first = $sformatf("read %0d at %0.2f ns: got %h, expected %h", reads, $realtime, rdata, expect_v);
        if (show == ID) begin
          $display("FAIL time=%0.2f ns: read %0d got %h, expected %h", $realtime, reads, rdata, expect_v);
          $fatal(1, "the reader got a word that is not the next one sent");
        end
      end
      expect_v = rdata + 8'd1;                 // carry on from the word just read
    end
  end
endmodule

$ vvp -n p.vvp +SHOW=2
FAIL time=5975.05 ns: read 127 got 77, expected 7f
FATAL: ptr_tb.sv:113: the reader got a word that is not the next one sent
       Time: 5975050  Scope: ptr_tb.r2

The lab's symptom on demand, at 5975.05 ns, with exactly the lab's numbers: 77 where 7f belonged. Verilator 5.048 fails at the same time with the same line. The model has earned a first piece of trust: with skew off (rig 0) the same design passed.

Observe: the word is eight writes old

7f − 77 = 8, and the FIFO has eight entries. Slot 6 held 77 from the previous lap and was about to receive 7f, but the writer had not written it yet. So the reader read a slot the writer had not filled. It does that only when the level says a word is there. The level is wptr_b - rptr, and wptr_b is the write pointer as the read side sees it, after it has crossed. The question becomes: what did the read side see, and when?

Localise: a pointer value nobody sent

Trace the read side cycle by cycle before the failure. The columns are the writer's real pointer, the pointer as it arrives on the wires, the synchronised pointer, the read pointer, the level the reader computes, how many words of the burst are left, and the word read. Each line shows the values just before that read edge.

$ vvp -n p.vvp +SHOW=2 +FROM=812 +TO=830
 rcyc  time ns   wptr wires sync rptr level left read
  812  5923.95   1101   1101  1101  1101   0    0
...
  814  5938.55   1101   1101  1101  1101   0    0
  815  5945.85   1110   1111  1101  1101   0    0
  816  5953.15   1110   1110  1101  1101   0    0
  817  5960.45   1110   1110  1111  1101   2    0
  818  5967.75   1110   1110  1110  1101   1    2   7e
  819  5975.05   1110   1110  1110  1110   0    1   77
FAIL time=5975.05 ns: read 127 got 77, expected 7f
...

• Up to 814: the FIFO is empty and quiet. Every column agrees: 1101.

• 815, the first wrong cycle: the writer wrote 7e at 5945 ns and its pointer went 1101 → 1110. At the read edge 0.85 ns later the wires show 1111. The writer has never had that value in this lap; it went from 1101 straight to 1110. The first synchroniser flop captures it cleanly, because nothing is metastable here; the value itself is wrong.

• 817: 1111 reaches the synchronised output. The reader computes a level of 1111 − 1101 = 2, but one word is there.

• 818: the reader starts a burst of two and reads 7e, which is correct.

• 819: it reads the next slot, which still holds 77 from the previous lap. The sequence check fires four read cycles after the bad capture, which scoreboards report late predicts.

The write changed two bits. Bit 1 rose at 5945.80 ns; bit 0 fell at 5946.20 ns. The read edge at 5945.85 ns came between them and saw the new bit 1 with the old bit 0: 1111, the binary pointer jumping two places ahead for one cycle. Every bit was captured correctly. The word was not.

Hypothesise and test: three predictions

Hypothesis: the pointers cross as binary words, one synchroniser per bit, so any write that changes two or more bits can be captured as a mix of old and new bits. If that is the whole story, three things follow, and each could prove it wrong:

• With no skew, the same design passes (rig 0). Already seen.

• A pointer where one bit changes per write (Gray code) never shows a value nobody sent, so with the same 0.4 ns skew it passes (rig 3).

• Gray code fails too when the skew is longer than a write clock, because then two writes are in flight on the wires at once and two bits can disagree (rig 4: 4.0 ns a bit, a 12 ns spread against a 10 ns write clock).

$ vvp -n p.vvp
binary, no skew : 6474 reads, 0 errors
Gray,   no skew : 6655 reads, 0 errors
binary, skew 0.4: 7626 reads, 132 errors  first: read 127 at 5975.05 ns: got 77, expected 7f
Gray,   skew 0.4: 6641 reads, 0 errors
Gray,   skew 4.0: 7456 reads, 94 errors  (skew window 12 ns, longer than a write clock)
one-bit check, binary, no skew: 35.00 ns: the crossing write pointer went 0001 -> 0010
one-bit check, Gray,   no skew: silent
PASS: without skew both pass; with 0.4 ns skew only the binary crossing fails; the one-bit check needs no skew
...

All three predictions hold, and on seeds 1 to 20 as well: binary with skew fails every seed (92 to 166 bad words), Gray with the same skew never does, Gray with 4.0 ns fails every seed. Verilator 5.048 prints the same lines. The third prediction matters most: it tells you the fix has a condition, and a hypothesis that only explains the failure you already have would not have predicted it.

The fix and the lesson

Fix: Gray-coded pointers

The fix is the standard asynchronous FIFO: convert each pointer to Gray code before it leaves its domain, synchronise it, and convert back on the other side. Successive Gray values differ in one bit, so a capture between arrivals sees the old value or the new one, never a third.

// snippet: the fix in ptr_fifo (FIX = 1), write pointer shown; the read pointer is the same
  assign wptr_x = bin2gray(wptr);            // was: assign wptr_x = wptr;
  assign wptr_b = gray2bin(ws2);             // was: assign wptr_b = ws2;

Prove: the fix, the sweep and its limit

$ for s in $(seq 1 20); do echo "seed $s: $(vvp -n p.vvp +SEED=$s | grep -E 'skew 0.4' | sed -E 's/ reads,/,/; s/  first:.*//' | tr '\n' '|')"; done
seed 1: binary, skew 0.4: 7626, 132 errors|Gray,   skew 0.4: 6641, 0 errors|
seed 2: binary, skew 0.4: 7788, 144 errors|Gray,   skew 0.4: 6649, 0 errors|
seed 3: binary, skew 0.4: 7586, 128 errors|Gray,   skew 0.4: 6639, 0 errors|
...
seed 20: binary, skew 0.4: 7643, 140 errors|Gray,   skew 0.4: 6559, 0 errors|

Twenty seeds, the bug in every one, the fix clean in every one. The fix also comes with a condition you must hand to the people who own timing: the skew between the bits of a Gray pointer must stay below one write clock, or rig 4 is what you get. That is a constraint on the crossing (a maximum delay or skew on those paths), not something RTL can promise. Where Gray fails covers it in depth.

Prevent: a check that needs no model

The skew model found the bug, but no regression should depend on someone thinking to add one. The rule the design broke can be checked where the value leaves its domain: a crossing pointer changes at most one bit per clock. The bench's one-bit check does that. It fired at 35 ns on the binary design with no skew at all, when the pointer went 0001 → 0010, and stayed silent on Gray. In a real project the same rule is a CDC tool's job: a bus with one synchroniser per bit is a standard finding, and it should never be waived as "passes simulation".

Which step won this hunt? Reproduce. Until simulation could fail, every experiment was a guess. Localise then turned "an old word" into "a pointer value nobody sent", which named the crossing.

Related topics

• Part 10 · Case Studies

• CDC/RDC: debug and case studies

• Part 5: CDC bugs

• CDC/RDC: skew and the capture window

• RTL: pointers and Gray conversion

• The first wrong cycle