A credit-based link between two blocks has shipped in three designs. In the fourth, the receiving block runs at 250 MHz instead of 50, and the link hangs after a few dozen words. The number of words before the hang changes with the seed. The credit logic is byte-for-byte the same as in the chips that work, and so is its bench.
A hang that depends on the seed, in logic that has not changed. Where is the credit going?
The first move is the one from Part 5: look for a counter update that can be skipped. There is none; the sender adds every return it sees. The second is to add Part 5's invariant, credits + buffer level = depth, and check it on every sender clock. It fires on the very first word, on the good design too, because a word or a credit inside a synchroniser is in neither term. The check gets switched off as noisy. The third is to start the sender with five credits instead of four "for margin", which overruns the buffer the first time the receiver stalls.
Follow one token through its life. Where is it created (reset), spent (a word sent), stored (in the buffer), moved (on its way back) and returned (added to the count)? Each place is a term in the law.
Write the law: credits at the sender + words in the buffer + credits on their way back = depth, at all times.
In one clock domain, check the law every cycle. That is Part 5's check, and it is exact there.
Across domains, split the law. Each side checks its local part (the sender's count against what it sent and got back; the receiver's level against what arrived and left). The crossing gets a ledger: every credit the receiver releases must arrive at the sender within a deadline, the synchroniser's latency plus margin.
Sweep the rates the integration will use, and beyond. A crossing that is correct at one clock ratio can lose events at another.
Know what a crossing can carry. A toggle or pulse crosses one event per destination sample; two events between samples become none. A Gray-coded count crosses any number, because the receiver of the count only needs the difference.
Check | Where it runs | What it catches | Across clocks |
|---|---|---|---|
credits + level = depth | one clock edge | any loss or duplicate, at once | false alarms: words and credits in a synchroniser are in neither term |
local laws at each end | each domain | losses inside one side | exact, but blind to the crossing |
ledger with a deadline | both domains (bench) or both counts (design) | a credit that never arrives | exact, if the deadline exceeds the crossing latency |
watchdog | anywhere | the hang after the last credit | always late, says nothing about where |
On the debug loop this sits between Observe and Localise: the law says a credit is gone, the ledger says which one, and when.
The link below, built with the toggle return, and a directed stimulus: the sender sends four words, then the receiver takes two of them inside one sender cycle, and later the other two the same way. Sampled on the sender's clock:
Read taken against rtog_seen: every word the receiver takes flips rtog, and the sender counts flips of what it sees.
Slots 0 to 3: the sender spends its four credits and stops; credits reaches 0.
Slots 5 and 6: the receiver takes two words (taken goes 0, 1, 2). rtog flipped up and back down between two sender edges; it shows 1 in slot 5 only because the picture samples just after the edge.
rtog_seen, the sender's first synchroniser flop, never changes: both flips happened between its samples. Slots 9 and 10 repeat it.
Slot 12: four words taken, the buffer empty, credits still 0. The link is hung with the receiver idle, the shape of a chain that ends.
FIX=0 returns credits as a toggle, the classic pulse-to-toggle crossing, designed when the receiver ran at 50 MHz. FIX=1 sends the receiver's read pointer back in Gray code, exactly as an asynchronous FIFO does. +RXNS sets the receiver's half period. The bench has a scoreboard, a watchdog and the ledger.
`timescale 1ns/1ps
// ccdc_link: a credit-based link across two clocks. The sender (clk_t) writes words into a
// four-entry buffer that the receiver (clk_r) reads; the buffer has no full flag, because the
// sender sends only while it holds a credit. It starts with four, spends one per word, and gets
// one back for every word the receiver takes. The return path crosses from clk_r to clk_t:
// FIX=0: the receiver flips a toggle per word taken; the sender counts edges of the
// synchronised toggle (one credit per edge).
// FIX=1: the receiver's read pointer crosses as Gray code; the sender adds the difference
// between the pointer it sees now and the one it saw last cycle.
module ccdc_link #(parameter bit FIX = 1) (
input logic clk_t, rst_t_n, have_data,
input logic [7:0] din,
output logic send,
output logic [2:0] credits,
input logic clk_r, rst_r_n, take,
output logic pop,
output logic [7:0] dout,
output logic [2:0] level
);
function automatic logic [2:0] g2b(input logic [2:0] g);
return {g[2], g[2] ^ g[1], g[2] ^ g[1] ^ g[0]};
endfunction
logic [7:0] mem [4];
logic [2:0] rbin, rgray, wg1, wg2; // receiver: read pointer, its Gray copy, wgray synchronised
logic rtog;
// ---- sender (clk_t)
logic [2:0] wbin, wgray, rg1, rg2, seen, ret;
logic t1, t2, t3;
assign send = have_data && (credits != 3'd0);
assign ret = FIX ? g2b(rg2) - seen : {2'b00, t2 ^ t3}; // credits returned this cycle
always_ff @(posedge clk_t or negedge rst_t_n)
if (!rst_t_n) begin
wbin <= '0; wgray <= '0; credits <= 3'd4; seen <= '0; {rg1, rg2} <= '0; {t1, t2, t3} <= '0;
end else begin
if (send) begin
mem[wbin[1:0]] <= din;
wbin <= wbin + 3'd1;
wgray <= (wbin + 3'd1) ^ ((wbin + 3'd1) >> 1);
end
credits <= credits - 3'(send) + ret;
seen <= g2b(rg2);
{rg1, rg2} <= {rgray, rg1};
{t1, t2, t3} <= {rtog, t1, t2};
end
// ---- receiver (clk_r)
assign level = g2b(wg2) - rbin;
assign pop = take && (level != 3'd0);
assign dout = mem[rbin[1:0]];
always_ff @(posedge clk_r or negedge rst_r_n)
if (!rst_r_n) begin rbin <= '0; rgray <= '0; rtog <= 1'b0; {wg1, wg2} <= '0; end
else begin
if (pop) begin
rbin <= rbin + 3'd1;
rgray <= (rbin + 3'd1) ^ ((rbin + 3'd1) >> 1);
rtog <= ~rtog;
end
{wg1, wg2} <= {wgray, wg1};
end
endmodule
$ iverilog -g2012 -Pccdc_tb.FIX=0 -o cb.vvp ccdc_tb.sv
$ vvp -n cb.vvp
PASS: 400 words, credits back to 4 (FIX=0, receiver half period 10.0 ns)
$ vvp -n cb.vvp +RXNS=2 +NOLEDGER
FAIL t=2355 ns: WATCHDOG: no word moved for 200 cycles; sent=18 taken=18 credits=0 level=0
...At 50 MHz the toggle return is fine: the receiver can take at most one word per 20 ns, so the sender sees every flip. At 250 MHz it hangs after 18 words, with everything sent taken and nothing in the buffer. Watchdog line, wait-for table: the sender waits for a credit, the receiver waits for a word, and nobody owes either. Credits were lost; the watchdog cannot say where.
$ iverilog -g2012 -o c.vvp ccdc_tb.sv
$ vvp -n c.vvp +SUMCHK
FAIL t=35 ns: SUM: credits=3 + level=0 != 4
...This is the correct design. At 35 ns the sender has sent one word, so it holds three credits, but the receiver cannot see the word yet: its write pointer is still in the two synchroniser flops on the way across, so the receiver's level is 0. Later in the run the same thing happens to credits on their way back. The single-domain sum has no term for either, so it fails on the first word. This is the noisy check from the story, and it is noisy because it is wrong for a crossing, not because the design is.
$ vvp -n cb.vvp +RXNS=2
FAIL t=275 ns: CREDIT_LOST: the word taken at 206 ns has not returned its credit; taken 16, credits back 14, credits 2
...t=275 ns, not 2355. The ledger fires 2,080 ns earlier, while the link is still running on its two remaining credits.
The word taken at 206 ns is the first credit that missed its 60 ns deadline. The receiver took 16 words and the sender got 14 credits back: two flips cancelled each other.
credits 2: the sender is still working. A link with more credits would run on for much longer, slower and slower, which is how a leak becomes a once-an-hour hang.
The deadline of 60 ns is six sender cycles: the toggle needs three flops to become a return, so a credit that is not back after six cycles is not late, it is gone.
Ten seeds at each receiver half period, both return paths:
Receiver half period | Receiver clock | Toggle return (FIX=0) | Gray return (FIX=1) |
|---|---|---|---|
10 ns | 50 MHz | 10 of 10 pass | 10 of 10 pass |
7 ns | 71 MHz | 10 of 10 pass | not run |
5 ns | 100 MHz, same as the sender | 10 of 10 pass | 10 of 10 pass |
4 ns | 125 MHz | 0 of 10 | 10 of 10 pass |
3 ns and 2.5 ns | 167 and 200 MHz | 0 of 10 | not run |
2 ns | 250 MHz | 0 of 10 | 10 of 10 pass |
1.3 ns | 385 MHz | not run | 10 of 10 pass |
The edge is sharp and has a reason: once the receiver is faster than the sender, it can take two words between two sender edges, and two flips are no flip. The Gray return passes at every ratio tried, because the sender adds the difference of a count: three words taken between samples add three.
The loss is a property of the crossing at a clock ratio, not of any counter, which is why reading the counters found nothing and why the design worked for three chips. The law found it once written in a form that respects the crossing: the ledger was silent on the good design at every ratio and named the first lost credit on the bad one, 2,080 ns before the hang.
No. It is back inside the rate the toggle can carry: at most one word per destination sample. Any later change that lets the receiver take two words between sender edges, a faster clock, or a burst mode, brings the loss back. The ledger and a bench assertion on the event rate make the assumption visible; the Gray return removes it.
Two terms are missing, one per direction: words in flight forward (sent, not yet in the receiver's level) and credits in flight back (taken by the receiver, not yet added by the sender). A bench can compute both from the counters at the two ends, because it sees both domains. Even then the sum is exact only when sampled where both counts are settled, which is why the ledger, a deadline per credit, is the more robust form across clocks.
ccdc_tb.sv as shown, 2026-10-18, Icarus Verilog 13.0: iverilog -g2012 -o c.vvp ccdc_tb.sv and -Pccdc_tb.FIX=0 -o cb.vvp. vvp -n cb.vvp PASS; vvp -n cb.vvp +RXNS=2 +NOLEDGER WATCHDOG at 2355 ns; vvp -n cb.vvp +RXNS=2 CREDIT_LOST at 275 ns; vvp -n c.vvp +SUMCHK SUM at 35 ns. Sweep, seeds 1 to 10 each: the toggle return passes 10 of 10 at receiver half periods of 10, 7 and 5 ns and 0 of 10 at 4, 3, 2.5 and 2 ns (+NOLEDGER); the Gray return passes 10 of 10 at 10, 5, 4, 2 and 1.3 ns with the ledger on.
Verilator 5.048 (verilator --binary --timing --assert -Wno-fatal [-GFIX=0] --top-module ccdc_tb) prints the same PASS, CREDIT_LOST and WATCHDOG lines (with a ZERODLY warning for the variable clock delay).
Waveform: COURSE=dbg node scripts/tools/gen-waves.mjs hangs-and-deadlocks:credit-and-token-loss on wave-src/credit-and-token-loss/toggle-merge.sv (the same link, a directed stimulus); it matched its expectations and a second run changed nothing. Limits: a four-entry link, simulated without metastability; the Gray return also needs the usual synchroniser rules in silicon.