A block takes start commands from a controller on another clock. On the bench board, roughly once an hour, a command is acknowledged but never executed, and once a day the block's state register reads 01, a value its FSM has no state for. Every simulation passes, including ten thousand seeds over the weekend.
The failure rate changed when the lab moved the controller's clock from 100 MHz to 97 MHz. Why can simulation not see a bug that the clock frequency can move?
They run more seeds, which cannot help: in zero-delay simulation a flop captures a changing input cleanly, and every bit of a bus changes at the same instant. Then they blame the board, add a firmware retry, or add a third flop to every synchroniser. None of these fixes a crossing with no synchroniser, two synchronisers, or one per bit.
Bug | What you see | What it follows |
|---|---|---|
Missing synchroniser | an FSM in an illegal state, a skipped state, a command dropped or done twice, X at gate level | the clock ratio and frequency; nothing in the stimulus |
Reconvergence (one signal, two synchronisers) | two blocks disagree for one cycle right after the signal changes: one corrupted word, one wrong decision | how often the source signal changes |
Incoherent multi-bit crossing | a value that was never sent, made of some bits of the old value and some of the new; a counter that goes backwards | how many bits change at once, and how often |
All three share the strongest clue: the failure needs a value or a state that no sequence of correct operations can produce, and it appears only at the moments the source side changes. On the debug loop this is Localise and Hypothesise: decide that the bug is a crossing before you look for it in the logic.
List the crossings on the failing path. For every signal that reaches the failing logic from another clock, write down how it crosses: synchroniser, Gray code, FIFO, handshake or nothing. A CDC tool does this structurally; reading a CDC report shows how to triage one.
Line the failure up with the source side. Was the source changing within a cycle or two of the failure? A corrupted word right after a mode change, or a torn value right after a configuration write, points at that crossing.
Run hostile simulation. Turn on a metastability model in the destination flops and rerun. A bug that fails with the model on and passes with it off is a crossing bug; a fix is proven only when it passes with the model on.
meta_ff is the model, in the style of the hostile simulation lesson: an ordinary flop, except that a bit of D that changed less than 0.5 ns before the edge resolves to its old or new value at random. Every destination flop in the bench is a meta_ff, the fixed designs included, so a fix cannot pass by avoiding the model. Clocks are 10 ns (A) and 7.3 ns (B). Each crossing exists twice, buggy (FIX = 0) and fixed.
`timescale 1ns/1ps
// meta_ff: a flop with a hostile-simulation model of metastability. When meta_en is 1 and a bit
// of D changed less than WIN ns before the clock edge, that bit resolves to its old or its new
// value at random. Otherwise it is an ordinary flop, so a design whose D pins never change near
// the edge (any correctly synchronised design) cannot tell the model is there.
module meta_ff #(parameter int W = 1, parameter int SALT = 0, parameter real WIN = 0.5) (
input logic clk, meta_en,
input logic [31:0] seed,
input logic [W-1:0] d,
output logic [W-1:0] q
);
logic [W-1:0] d_q = '0, old_v = '0, cap;
real t_chg [W];
logic [31:0] rng;
bit seeded = 1'b0;
initial begin
q = '0;
for (int i = 0; i < W; i++) t_chg[i] = -100.0;
end
always @(d) begin // when did each bit last change?
for (int i = 0; i < W; i++)
if (d[i] !== d_q[i]) begin t_chg[i] = $realtime; old_v[i] = d_q[i]; end
d_q = d;
end
always @(posedge clk) begin
cap = d;
for (int i = 0; i < W; i++)
if (meta_en && $realtime - t_chg[i] < WIN) begin
if (!seeded) begin rng = seed * 32'h9E37_79B9 + 32'(SALT) * 32'h85EB_CA6B + 32'h7F4A_7C16; seeded = 1'b1; end
rng ^= rng << 13; rng ^= rng >> 17; rng ^= rng << 5;
cap[i] = rng[0] ? d[i] : old_v[i]; // resolves either way
end
q <= cap;
end
endmodule
// sync2: the standard two-flop synchroniser, built from the model flop.
module sync2 #(parameter int SALT = 0) (input logic clk, meta_en, input logic [31:0] seed, input logic d, output logic q);
logic s1;
meta_ff #(.W(1), .SALT(SALT)) f1 (.clk, .meta_en, .seed, .d(d), .q(s1));
meta_ff #(.W(1), .SALT(SALT + 1)) f2 (.clk, .meta_en, .seed, .d(s1), .q(q));
endmodule
// Crossing 1: a start request into a two-bit FSM. IDLE=00 -> BUSY=11 (4 cycles) -> WAIT=10 -> IDLE.
// FIX = 0: go_a feeds the next-state logic directly. FIX = 1: through sync2.
module start_fsm #(parameter bit FIX = 0, parameter int SALT = 0) (
input logic clk_b, meta_en, input logic [31:0] seed, input logic go_a,
output logic [1:0] state
);
localparam logic [1:0] IDLE = 2'b00, BUSY = 2'b11, WAIT = 2'b10;
logic go;
logic [1:0] nxt, n = '0;
if (FIX) sync2 #(.SALT(SALT)) u_sync (.clk(clk_b), .meta_en, .seed, .d(go_a), .q(go));
else assign go = go_a; // bug: no synchroniser
always_comb
case (state)
IDLE: nxt = go ? BUSY : IDLE;
BUSY: nxt = (n == 2'd3) ? WAIT : BUSY;
WAIT: nxt = go ? WAIT : IDLE;
default: nxt = IDLE; // 01 is illegal
endcase
meta_ff #(.W(2), .SALT(SALT + 7)) u_state (.clk(clk_b), .meta_en, .seed, .d(nxt), .q(state));
always @(posedge clk_b) n <= (state == BUSY) ? n + 2'd1 : 2'd0;
endmodule
// Crossing 2: one mode bit, used by two pipeline stages that must agree (scramble, then unscramble).
// FIX = 0: each stage has its own synchroniser. FIX = 1: one synchroniser feeds both.
module mode_pipe #(parameter bit FIX = 0, parameter int SALT = 0) (
input logic clk_b, meta_en, input logic [31:0] seed, input logic mode_a,
output logic err // the word leaving stage 2 is wrong
);
localparam logic [7:0] KEY = 8'h5A;
logic m1, m2, m2_d = 1'b0;
logic [7:0] in = '0, s1 = '0, out = '0, in_d1 = '0, in_d2 = '0;
sync2 #(.SALT(SALT)) u_s1 (.clk(clk_b), .meta_en, .seed, .d(mode_a), .q(m1));
if (FIX) assign m2 = m1;
else sync2 #(.SALT(SALT + 2)) u_s2 (.clk(clk_b), .meta_en, .seed, .d(mode_a), .q(m2));
always @(posedge clk_b) begin
in <= in + 8'd1;
s1 <= in ^ (m1 ? KEY : 8'h00); // stage 1 scrambles when the mode is on
m2_d <= m2;
out <= s1 ^ (m2_d ? KEY : 8'h00); // stage 2 unscrambles a cycle later
in_d1 <= in; in_d2 <= in_d1;
end
assign err = (out !== in_d2);
endmodule
// Crossing 3: an 8-bit configuration word. FIX = 0: each bit through its own sync2.
// FIX = 1: the word is held stable and a toggle says when to capture it (qualified capture).
module cfg_cross #(parameter bit FIX = 0, parameter int SALT = 0) (
input logic clk_b, meta_en, input logic [31:0] seed, input logic [7:0] cfg_a, input logic upd_a,
output logic [7:0] cfg_b
);
if (FIX) begin : g_fix
logic t_s, t_q = 1'b0, cap;
logic [7:0] d;
sync2 #(.SALT(SALT)) u_t (.clk(clk_b), .meta_en, .seed, .d(upd_a), .q(t_s));
always @(posedge clk_b) t_q <= t_s;
assign cap = t_s ^ t_q; // the toggle arrived: the word is stable
assign d = cap ? cfg_a : cfg_b;
meta_ff #(.W(8), .SALT(SALT + 3)) u_cfg (.clk(clk_b), .meta_en, .seed, .d(d), .q(cfg_b));
end else begin : g_bad
logic [7:0] s1;
meta_ff #(.W(8), .SALT(SALT)) u_f1 (.clk(clk_b), .meta_en, .seed, .d(cfg_a), .q(s1));
meta_ff #(.W(8), .SALT(SALT + 9)) u_f2 (.clk(clk_b), .meta_en, .seed, .d(s1), .q(cfg_b));
end
endmodule
$ iverilog -g2012 -o c.vvp cdc_tb.sv
$ vvp -n c.vvp
model off: crossing 1 bad=0 fix=0 | crossing 2 bad=0 fix=0 | crossing 3 bad=0 fix=0 (errors in 20000 cycles)
model on : crossing 1 bad=24 fix=0 | crossing 2 bad=26 fix=0 | crossing 3 bad=14 fix=0 (errors in 20000 cycles)
PASS: zero-delay simulation passes all six; the model breaks the three bugs and none of the fixes
cdc_tb.sv:231: $finish called at 291997350 (1ps)The first line is your regression: three broken designs, all clean. With the model on, each buggy crossing fails 14 to 26 times in 20,000 cycles and each fix stays at zero, as it does for seeds 1 to 10. That control is what makes the model trustworthy: it breaks only structures that are physically wrong.
$ vvp -n c.vvp +BAD=1
FAIL time=9975.6 ns: crossing 1 (no synchroniser): FSM went from 00 to 10 (IDLE=00 BUSY=11 WAIT=10)
FATAL: cdc_tb.sv:173: illegal or skipped state
Time: 9975650 Scope: cdc_tb
$ vvp -n c.vvp +BAD=1 +SEED=2
FAIL time=9975.6 ns: crossing 1 (no synchroniser): FSM went from 00 to 01 (IDLE=00 BUSY=11 WAIT=10)
...
$ vvp -n c.vvp +BAD=2
FAIL time=3157.4 ns: crossing 2 (two synchronisers for one mode bit): a word left stage 2 scrambled
...
$ vvp -n c.vvp +BAD=3
FAIL time=8252.9 ns: crossing 3 (each bit synchronised): B uses f3; A wrote f9, then 23
...
$ vvp -n c.vvp +BAD=3 +SEED=2
FAIL time=8252.9 ns: crossing 3 (each bit synchronised): B uses b9; A wrote f9, then 23
...Read them against the signature table. Crossing 1: go_a rose just before a B edge and the two state flops resolved differently. With seed 1 only bit 1 saw it, so the FSM went from IDLE straight to WAIT and dropped the command: the lab's "acknowledged but never executed". With seed 2, at the same edge, only bit 0 saw it: state 01, the lab's impossible value. One cause, two symptoms.
Crossing 2: one synchroniser saw the mode change a cycle before the other, so stage 1 scrambled a word that stage 2 did not unscramble. Crossing 3: A changed the word from f9 to 23, five bits at once, and B used f3 (seed 1) or b9 (seed 2), mixes of old and new bits that A never wrote. Verilator 5.048 prints the same lines.
$ grep -n "async_fifo #\|sync2_rtl u_\|\.rclk\|\.wclk" rtl/periph_slave.sv
52: async_fifo #(.DW(32), .AW(AW)) u_fifo (
53: .wclk(clk), .wrst_n(rst_n), .winc(winc), .wdata(wdata), .wfull(wfull),
54: .rclk(pclk), .rrst_n(prst_n), .rinc(rinc), .rdata(fifo_q), .rempty(rempty)
56: sync2_rtl u_en_sync (.clk_dst(pclk), .rst_dst_n(prst_n), .async_level(pctrl_q[PCTRL_SINK_EN]),Check 1 on dbglab finds data crossing through an async FIFO with Gray pointers and the sink enable, a single level, through sync2_rtl: both fix patterns. pwr_on and iso_en go from the bus-clock register straight into the pclk domain; the source calls them static controls that software sequences, which is the waiver a CDC tool would need. None of dbglab's five bugs is a CDC bug.
Three bugs that zero-delay simulation cannot find now fail on demand with messages that match the lab, and each fix survives the model on ten seeds. That shows a mechanism, not what happened on the board. The next experiment is the one in when simulation cannot reproduce it: predict how the lab rate moves with the clock ratio, and check.
Missing synchroniser: one two-flop synchroniser per asynchronous control, and only its output is used. Its first flop may still resolve either way; that moves the event by a cycle, which the FSM tolerates.
Reconvergence: synchronise once and fan out the synchronised signal. If two controls really must cross, encode them so only one bit changes per step, or send one and qualify the other.
Multi-bit value: hold the word stable, synchronise one qualifier (a toggle here) and capture the word when it arrives; or Gray-code a counter; or use an async FIFO. Bus patterns and their costs are in multi-bit strategies.
A structural CDC check on every netlist, with each waiver naming a mechanism ("static: written only while the sink is stopped"), never "passes simulation".
A metastability model in the regression, switchable, on several clock ratios, reporting window hits per crossing: a crossing with none was not tested.
Assertions on the protocol: data stable while the qualifier is in flight, one bit per step on a Gray source, request held until acknowledged (assertion patterns).
Review question: "For every signal from another clock, which single flop samples it first?" Two answers, or none, is a bug.
Each bit is safe on its own, but they do not arrive together: 0010 to 0101 changes three bits, and each can resolve in a different cycle, so the receiver sees a mix such as 0110 for a cycle. Fixes: hold the command stable and send a single synchronised toggle that says when to capture it (as crossing 3's fix does), or use a handshake or an async FIFO. Gray code works only for values that step by one, like counters.
enable from domain A with their own two-flop synchroniser. Both are correct, so a reviewer approves it. What can go wrong, and what would hostile simulation show?Reconvergence: when enable changes near a B edge, one synchroniser can see it a cycle before the other, so for one cycle one block is enabled and the other is not. With the model on you would see a single wrong item at the moments enable changes, as crossing 2's scrambled word. Synchronise once and fan out the output.
cdc_tb.sv as shown, 2026-10-18: iverilog -g2012 -o c.vvp cdc_tb.sv and vvp -n c.vvp (Icarus Verilog 13.0, 0.97 s): model off, 0 errors in all six crossings in 20,000 B cycles; model on, 24, 26 and 14 errors in the buggy crossings and 0 in the fixes; PASS. +BAD=1, +BAD=2, +BAD=3 (seed 1) and +BAD=1, +BAD=3 with +SEED=2 give the lines shown; +FIX=1, +FIX=2, +FIX=3 pass. Seeds 1 to 10: buggy crossings 21 to 31, 21 to 34 and 14 to 19 errors, fixes 0 every time.
verilator --binary --timing --assert -Wno-fatal --top-module cdc_tb (Verilator 5.048) prints the same lines for the default run and for +BAD=1, +BAD=2, +BAD=3. dbglab at HEAD 49c5f04: grep -ln "pclk\|rclk" rtl/*.sv and grep -n on rtl/periph_slave.sv, output as shown.
Limits: the model's 0.5 ns window is far wider than a real flop's, and it resolves within the cycle; it shows mechanism, never a failure rate or an MTBF. Clocks are 10 ns and 7.3 ns with no jitter. No CDC tool is installed, so no structural report is shown.