The lab has the first boards. Once or twice a day, the slow peripheral's driver logs a word count that goes backwards: 19, then 17, then 20. The regression has run 10,000 seeds of the same traffic without a single error. The lab lead says it is a simulation gap; the RTL owner says it is a board problem. Both want you to decide. How do you debug a failure that your best instrument cannot show?
The first reflex is more seeds: 10,000 became 100,000, still clean. That cannot work if the bug needs something the simulator does not have. The second reflex is to declare it "a silicon issue" and wait for a respin that may contain the same bug. The third is to add debug logic to the FPGA build, which moves placement and timing, and the failure goes away while you watch.
Each move treats simulation as a lesser lab. It is a different instrument with known blind spots, and the method starts by naming them.
Where the bug lives | Why RTL simulation misses it | Where it shows |
|---|---|---|
Silicon timing (a slow path, a hold race) | zero-delay logic, every flop meets setup and hold | gate-level simulation with timing, STA, a voltage and frequency sweep |
Metastability and CDC | a flop captures the old or the new value cleanly, never in between | CDC checks, a metastability model, clock-ratio experiments |
Analogue and power (supply droop, noise, a PLL) | no voltage, no current, no noise | lab measurement, power analysis |
Power-up state | X in Icarus, 0 in Verilator, never the real random value | random initial values, such as dbglab's |
Environment (board, firmware, other chips) | not in the bench | the lab, then a bench that copies its stimulus |
Then, in order:
Match conditions first. Write down every difference between the lab run and the simulation: software, clock frequencies and their ratio, reset sequence, configuration, traffic. Close the cheap ones. Many "silicon only" bugs are a register setting the bench never used.
Use the lab as an instrument. Change one condition at a time, voltage, temperature, frequency, clock ratio, and measure the failure rate, as in making a flaky test fail. A rate that follows voltage and frequency suggests timing; one that follows the clock ratio suggests a crossing; one that follows power cycles suggests initial state.
Move the instruments, not the timing. Prefer what is already in the chip, counters, trace, status registers, scan dumps, over new logic that changes the design. The bring-up course covers the lab side.
Build a model of the suspected mechanism and add it to the simulation: a random-resolution flop, a delay on one path, random initial values. Keep it small and switchable.
Make the model earn trust. The symptom must appear with the model on and not with it off; a correct design must stay clean with the model on; and the model must predict a lab result you have not seen yet.
Here, the counter crosses from a fast to a slow clock domain, so the first suspect is the crossing.
The design sends a 16-bit counter from a 10 ns domain to a 13.7 ns domain through a two-flop synchroniser on every bit. That is a known mistake for a multi-bit value: each bit is safe on its own, but when several bits change at once, each can resolve differently, and the receiver sees a value that was never sent. RTL simulation cannot show it, because its flops never go metastable. So model it. sync2_model is an ordinary two-flop synchroniser, except that when its model is on and a bit changes inside a window around the capture edge, the first flop takes the old or the new value at random. The bench sends the same counter twice, once as plain binary and once as Gray code, where only one bit changes per step.
`timescale 1ns/1ps
// sync2_model: a two-flop synchroniser per bit, with a model of metastability.
// When meta_en is 1 and a bit changes less than SETUP ns before or HOLD ns after the capture
// edge, the first flop resolves that bit to the old or the new value at random. Plain RTL
// simulation (meta_en = 0) captures whatever value the bit has at the edge, cleanly.
module sync2_model #(parameter int W = 8, parameter real SETUP = 0.97, parameter real HOLD = 0.23) (
input logic clk, meta_en,
input logic [31:0] seed,
input logic [W-1:0] d,
output logic [W-1:0] q
);
logic [W-1:0] ff1 = '0, ff2 = '0, old_v = '0, d_q = '0, cap;
real t_chg [0:W-1];
real t_edge;
logic [31:0] rng;
bit seeded = 1'b0;
initial for (int i = 0; i < W; i++) t_chg[i] = -100.0;
always @(d) begin // remember when each bit last changed
for (int i = 0; i < W; i++)
if (d[i] !== d_q[i]) begin t_chg[i] = $realtime; old_v[i] = d_q[i]; end
d_q = d;
end
always @(posedge clk) begin
ff2 <= ff1;
cap = d; // what plain RTL simulation captures
t_edge = $realtime;
#(HOLD); // look just past the edge as well
for (int i = 0; i < W; i++)
if (meta_en && t_chg[i] > t_edge - SETUP && t_chg[i] <= t_edge + HOLD) begin
if (!seeded) begin rng = seed * 32'h9E37_79B9 + 32'h7F4A_7C16; seeded = 1'b1; end
rng ^= rng << 13; rng ^= rng >> 17; rng ^= rng << 5;
cap[i] = rng[0] ? d[i] : old_v[i]; // resolves either way
end
ff1 = cap;
end
assign q = ff2;
endmodule
$ iverilog -g2012 -o s.vvp sync_tb.sv
$ vvp -n s.vvp
model off: binary 0 errors, Gray 0 errors in 999 slow cycles
model on: binary 20 errors, Gray 0 errors in 1000 slow cycles
PASS: slow clock 13.7 ns, the Gray path never failsRead the two lines as an experiment with a control. With the model off, this is your regression: clean on both paths, as it was for 10,000 seeds. With the model on, the binary path breaks 20 times in 1,000 slow cycles, and the Gray path, which carries the same counts through the same synchronisers, stays clean. The symptom needs the mechanism, and the design that is immune to the mechanism is immune to the model. Both checks matter: a model that broke the Gray path too would be modelling something that does not exist.
$ vvp -n s.vvp +SHOW +SEED=1
FAIL time=609.8 ns: slow side received count 17 after 19; the sender is at 20
$ vvp -n s.vvp +SHOW +SEED=2
FAIL time=609.8 ns: slow side received count 23 after 19; the sender is at 20Same edge, different random resolution, different wrong value. 19 is 10011 and 20 is 10100: three bits change at once, so the receiver can see any mix of them. Seed 1 gives 10001, which is 17: the count goes backwards, exactly the lab's "19, then 17, then 20". Seed 2 gives 10111, 23, a count from the future. The lab's log now has a mechanism that produces it, in a simulation you can rerun on demand. Verilator 5.048 prints the same lines.
A model is a hypothesis, so ask it to predict something new. The mechanism needs a slow edge close to a change of the counter, so the failure should depend on the clock ratio. Run the model at other slow-clock periods:
$ for p in 13000 13700 14000 14500 15000 16000 20000; do echo "$p: $(vvp -n s.vvp +DPERIOD=$p | sed -n 2p)"; done
13000: model on: binary 22 errors, Gray 0 errors in 1000 slow cycles
13700: model on: binary 20 errors, Gray 0 errors in 1000 slow cycles
14000: model on: binary 33 errors, Gray 0 errors in 1000 slow cycles
14500: model on: binary 20 errors, Gray 0 errors in 1000 slow cycles
15000: model on: binary 0 errors, Gray 0 errors in 1000 slow cycles
16000: model on: binary 0 errors, Gray 0 errors in 1000 slow cycles
20000: model on: binary 0 errors, Gray 0 errors in 1000 slow cyclesAt 15, 16 and 20 ns the slow edges fall on a fixed set of phases relative to the counter changes, none within the window (at 16 ns, 15 phases, the closest 1 ns after a change), so the model predicts no failures there. At 13, 13.7, 14 and 14.5 ns some edges land on or just after a change, and the failures come back. This holds only when both clocks come from one reference, as when one PLL makes both, so their edges keep fixed phases. With independent oscillators the phase drifts, every ratio eventually lands in the window, and the model's prediction becomes a rate that rises with both clock frequencies, not a set of safe periods. That is a lab experiment for a board whose two clocks share a reference: if it can run the slow clock at 15 or 20 ns and the backwards counts disappear, while 13 or 14 ns keeps them, the crossing is very likely the cause. If the rate does not follow the ratio, the model is wrong, and you have learned that cheaply.
The model reproduces the lab's symptom only when the mechanism is on, never on the Gray path, and it predicts a clock-ratio dependence the lab can check. You still have not proven what happened on the board; the clock-ratio run is the step that does. If it confirms, the fix, sending the count as Gray code or through a handshake, can be proven in simulation with the model on, which plain simulation could never do. Reproduce is complete in the sense that matters: one command now fails on demand for the right reason.
It is strong evidence, because the model predicted exactly that before the run: no failures at 20 ns, failures at 13.7 ns. It is not proof on its own: a day at a once-a-day rate is a small sample, and 20 ns also changes timing margins. Run at least one other ratio the model says should fail (13 or 14.5 ns) and one it says should pass (15 or 16 ns). If the board follows the model in all of them, the case is closed. It also assumes the board's two clocks share a reference; with free-running oscillators, 20 ns would only lower the rate.
The model only makes sense at a clock-domain crossing, where an input really changes near the capture edge with no timing guarantee. Inside one domain, static timing analysis guarantees setup and hold, so randomising those flops models something that cannot happen in a design that meets timing. Apply the model only to the first flop of each synchroniser, and check that a known-good crossing stays clean.
sync_tb as shown, Icarus Verilog 13.0 (iverilog -g2012 -o s.vvp sync_tb.sv, vvp -n s.vvp) and Verilator 5.048 (verilator --binary --timing --assert -Wno-fatal --top-module sync_tb), 2026-10-17. No plusargs: model off, 0 errors on both paths in 999 slow cycles; model on, 20 binary errors and 0 Gray errors in 1000; PASS. +SHOW +SEED=1: FAIL at 609.8 ns (17 after 19, sender at 20); +SEED=2: 23 after 19.
+DPERIOD=<ps> with the model on, binary errors in 1000 slow cycles: 13000: 22, 13700: 20, 14000: 33, 14500: 20, 15000: 0, 16000: 0, 20000: 0. The Gray path had 0 in every run. Verilator printed the same numbers and lines in every case.
Limits: the capture window (0.97 ns before to 0.23 ns after the edge) is exaggerated, hundreds of times wider than a real flop's, so the rare event becomes common. The model shows that the mechanism can produce the symptom; only a lab result it predicts can show the mechanism is what happened on the board. No lab, FPGA or silicon was used.