Context, kept generic: a streaming datapath through an async FIFO between a fast producer and a slower consumer, with bursty traffic and back-pressure. Under sustained bursts the consumer occasionally reads a stale word twice and drops a fresh one, with no flag asserted. The FIFO pointers are Gray-coded and reviewed, the depth math checks out, and the corruption never appears when resets release together in simulation.
Corruption appears only under bursts long enough to fill the FIFO near full, and only on some boards. Full and empty both behave plausibly in isolation, which sends the first week of debug into the memory and the consumer logic. The telling detail arrives late: on corrupting runs, the first empty deassertion after reset comes one read cycle too early, handing the consumer a word from before the burst.
Hypothesis | Evidence sought | Verdict |
|---|---|---|
Depth too small for the burst plus synchroniser latency. | Recompute depth from measured burst length and both clock rates. | Killed: the depth covers the worst burst with margin to spare. |
Binary pointer sneaked into one crossing. | Inspect both pointer crossings against the Gray rule. | Killed: both pointers are registered Gray, one step at a time. |
Pointers start mismatched because the two sides leave reset at different times. | Stagger the releases in simulation and watch the first flags. | Survives: see evidence below. |
In the failing system the read clock is gated during early boot and starts several cycles after the write side has left reset. The write pointer advances through Gray values while the read side still sleeps; when the read side wakes, its synchronised copy of the write pointer starts from wherever the synchroniser settled, not from the reset value the empty logic assumes. The pointer-crossing waveform of this section shows how a sampled pointer takes a non-reset value; here the mismatch is present from the first cycle, so empty deasserts early and the consumer reads a stale entry as live data.
Root cause: release ordering across the FIFO reset domains was never specified, so the read side can wake into a stream already in flight with pointers the empty logic never agreed on. The fix has two parts: a documented release order that brings both sides out of reset before either clock or pointer moves, and reset-aware empty logic that holds empty until the first synchronised pointer has demonstrably arrived. The release-order lesson states the discipline; the reset-and-tool-checks lesson of the FIFO topic states what the tools expect around FIFO reset.
The reset-sequencing review owned this and never asked for the release order across the two FIFO clocks; the FIFO review checked Gray and depth but assumed simultaneous release, which the regression never violated because every testbench released all resets on one edge. The process change adds staggered and skewed releases to the reset regression for every multi-clock block, and reset coverage must now show each domain released last, first, and simultaneously against each other before the gate opens.
Because Gray and depth are the common causes and both check out on paper; the mismatch is in the initial condition the empty logic assumes, which only a staggered-release experiment exposes.
The early-empty error is present from reset, but the consumer only reads stale data when it actually reads: under light traffic it idles past the bad window, while a burst forces reads through it.