A chip-to-chip link uses credits: the sender may send a word only while it holds a credit, and the receiver gives one back for every word it hands on. It passed every regression. Then the lab report arrives (illustrative):
You have the RTL and its bench. One more fact matters: a recent change added a flush for link error recovery. When the link retrains, the receiver throws away every word it holds and returns their credits in one count. The flush tests pass.
Stop here if you want to try it yourself. The design and its bench are below, in "The hunt". What would you do first: open a waveform of the hang, or something else?
Two words in the report matter: the throughput falls before it stops. A deadlock stops at once. Something that slowly runs out is being lost a little at a time, and an hour of lab time is far too long to simulate. Which rare event can you make common?
A small model of the link, with the same structure as the real one: 16 credits, two link stages forward, a 16-entry receive buffer, a return register and three link stages back. The bench drives traffic at full rate, has a consumer that is ready three cycles in four, and a flush about once in FLUSH cycles. It has a scoreboard (flushed words removed from the expected list), a watchdog, and the credit law, which we will use later. FIX=1, the default, is the fixed design; FIX=0 is the one in the lab.
`timescale 1ns/1ps
// clink: a credit-based link. The sender holds CREDITS credits, spends one per word and sends
// it down a two-stage link into the receiver's buffer, where a consumer pops it. Each popped
// word gives its credit back: a return count, registered, then three link stages back.
// A flush (link error recovery) throws away every word the receiver holds and returns their
// credits in one count.
// FIX=0: the flush counts the buffered words but forgets a word arriving from the link in
// the same cycle. The word is thrown away and its credit is never returned.
// FIX=1: the flush also counts the arriving word.
module clink #(parameter bit FIX = 1, parameter int CREDITS = 16) (
input logic clk, rst_n,
input logic have_data,
input logic [15:0] din,
output logic send,
output logic [4:0] credits,
input logic take, flush,
output logic pop, arrive,
output logic [15:0] dout,
output logic [4:0] level,
output logic [4:0] ret_q
);
logic [1:0] fv; // link stages
logic [15:0] fd [2];
logic [4:0] rp [3]; // return link stages: credits on the way back
logic [15:0] mem [CREDITS];
logic [3:0] wp, rd;
assign send = have_data && (credits != 5'd0);
assign arrive = fv[1];
assign pop = take && !flush && (level != 5'd0);
assign dout = mem[rd];
// ---- sender
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) credits <= 5'(CREDITS);
else credits <= credits - 5'(send) + rp[2];
// ---- forward link
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) fv <= '0;
else begin fv <= {fv[0], send}; fd[0] <= din; fd[1] <= fd[0]; end
// ---- receiver
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) begin wp <= '0; rd <= '0; level <= '0; ret_q <= '0; end
else if (flush) begin
rd <= wp; level <= '0; // throw away the buffer and the arriving word
ret_q <= FIX ? level + 5'(arrive) : level;
end else begin
if (arrive) begin mem[wp] <= fd[1]; wp <= wp + 4'd1; end
if (pop) rd <= rd + 4'd1;
level <= level + 5'(arrive) - 5'(pop);
ret_q <= 5'(pop);
end
// ---- return link
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) begin rp[0] <= '0; rp[1] <= '0; rp[2] <= '0; end
else begin rp[0] <= ret_q; rp[1] <= rp[0]; rp[2] <= rp[1]; end
endmodule
First make the lab failure happen in simulation, with the law off, as the original bench had it. A flush once in about 100,000 cycles stands in for the lab's rare retrain. +RATE prints the delivered rate every 250,000 cycles:
$ iverilog -g2012 -Pleak_tb.FIX=0 -o cb.vvp leak_tb.sv
$ /usr/bin/time -p vvp -n cb.vvp +NOLAW +WORDS=10000000 +FLUSH=100000 +RATE=250000
RATE cycle=250000 delivered=187666 words/cycle=0.75
RATE cycle=500000 delivered=375128 words/cycle=0.74
RATE cycle=750000 delivered=562518 words/cycle=0.74
RATE cycle=1000000 delivered=750262 words/cycle=0.75
RATE cycle=1250000 delivered=935544 words/cycle=0.74
RATE cycle=1500000 delivered=1104364 words/cycle=0.67
RATE cycle=1750000 delivered=1262938 words/cycle=0.63
RATE cycle=2000000 delivered=1374466 words/cycle=0.44
RATE cycle=2250000 delivered=1443366 words/cycle=0.27
RATE cycle=2500000 delivered=1502122 words/cycle=0.23
RATE cycle=2750000 delivered=1532133 words/cycle=0.12
FAIL cycle=2928785: WATCHDOG: nothing delivered for 1000 cycles; delivered=1553468 credits=0 buffered=0 flushes=35 (16 with a word arriving)
...
real 15.91Cycles 0 to 1,250,000: 0.75 words per cycle, the consumer's rate. Credits are being lost already, but the link has more than the round trip needs, so nothing shows.
From 1,500,000: the sag. The rate falls window by window, the shape of the lab's throughput meter. Each lost credit now costs bandwidth.
Cycle 2,928,785: the hang, after 1.5 million words and 16 seconds of simulation. credits=0 buffered=0: the sender has no credit, the receiver holds no word, and nothing is in flight.
That last line is the wait-for chain of Part 5: the sender waits for a credit, the receiver waits for a word, and nobody owes either. A chain that ends in nobody means something conserved was lost. On the scale (illustrative): if the lab link retrains about once every 100 seconds, 35 flushes take about an hour. The model hangs after 35 flushes too.
Sixteen seconds per run is too slow to debug with, and in the lab it is an hour. The bench has two random knobs: the consumer's rate and the flush rate. The consumer runs all the time, so it cannot explain a rare failure. The flush can. Turn its rate up, as in making the rate go up, and see whether the hang follows:
$ for f in 20000 5000 500 50; do /usr/bin/time -p vvp -n cb.vvp +NOLAW +WORDS=10000000 +FLUSH=$f 2>&1 | grep -E "^FAIL|^real"; done
FAIL cycle=640730: WATCHDOG: nothing delivered for 1000 cycles; delivered=367777 credits=0 buffered=0 flushes=37 (16 with a word arriving)
real 3.37
FAIL cycle=147854: WATCHDOG: nothing delivered for 1000 cycles; delivered=78498 credits=0 buffered=0 flushes=44 (16 with a word arriving)
real 0.78
FAIL cycle=14291: WATCHDOG: nothing delivered for 1000 cycles; delivered=8500 credits=0 buffered=0 flushes=29 (16 with a word arriving)
real 0.08
FAIL cycle=1894: WATCHDOG: nothing delivered for 1000 cycles; delivered=490 credits=0 buffered=0 flushes=48 (16 with a word arriving)
real 0.01
$ /usr/bin/time -p vvp -n cb.vvp +NOLAW +WORDS=2000000 +FLUSH=2000000000 2>&1 | grep -E "^PASS|^FAIL|^real"
PASS: 2000000 words delivered, 0 flushed in 0 flushes, 2665564 cycles, credits back to 16 (FIX=0)
real 14.49Flush about once in | Hang at cycle | Flushes before the hang | Run time |
|---|---|---|---|
100,000 cycles | 2,928,785 | 35 | 15.9 s |
20,000 | 640,730 | 37 | 3.4 s |
5,000 | 147,854 | 44 | 0.8 s |
500 | 14,291 | 29 | 0.08 s |
50 | 1,894 | 48 | 0.01 s |
never | no hang in 2 million words | 0 | 14.5 s |
Read the columns, not the run times. The hang moves with the flush rate, by a factor of about 1,500 from the first row to the last, and without flushes there is no hang at all. The number of flushes it takes stays between 29 and 48. So the flush is in the cause, and the failure is now a hundredth of a second away. That is the step that wins this hunt: every experiment from here costs nothing.
One more number is the same in every row: 16 flushes with a word arriving. Sixteen is the number of credits. Hold that thought.
A watchdog says only that everything is gone. The question is when the first credit went. Credits are conserved, so write the law, exactly as in credit and token loss, and check it every cycle. Here the whole loop is in one clock domain, so the plain sum is exact: credits at the sender + words on the forward link + words buffered + credits on the way back = 16. That is the CREDIT_LAW check in the bench; run it without +NOLAW:
$ vvp -n cb.vvp
FAIL cycle=2655: CREDIT_LAW: credits=1 on link=2 buffered=0 coming back=12: total 15, not 16 (last cycle: flush=1 arrive=1)
...
$ vvp -n cb.vvp +WORDS=10000000 +FLUSH=100000
FAIL cycle=298373: CREDIT_LAW: credits=1 on link=2 buffered=0 coming back=12: total 15, not 16 (last cycle: flush=1 arrive=1)The first line is the default run (a flush about every 500 cycles). The law breaks at cycle 2655, the first time any credit goes missing, while the link is still running at full speed.
`coming back=12`: a flush has just returned a count of buffered words in one go. `total 15`: one credit is not in any of the four places.
`last cycle: flush=1 arrive=1`: the cycle the credit went had a flush and, at the same time, a word arriving from the link.
The second line is the long run's settings. The law fires at cycle 298373, ten times earlier than the watchdog's 2928785, at the first of the 16 leaks rather than after the last.
Now read the flush branch of the receiver with that cycle in mind:
// snippet: clink, the flush branch of the receiver (FIX=0 is the design in the lab)
else if (flush) begin
rd <= wp; level <= '0; // throw away the buffer and the arriving word
ret_q <= FIX ? level + 5'(arrive) : level;The flush throws away the words in the buffer and the word arriving this cycle (it is never written), but it gives back only level credits. The arriving word was already paid for by the sender, it is not on the link any more, and it never reaches the buffer. Its credit has nowhere to be.
Read credits and law_ok across the flush:
Slots 0 to 2: three words are sent; credits goes 16 (before slot 0), f, e, d. Words arrive from slot 1, and level counts the first two.
Slot 3: flush and arrive are both 1, with level at 2. This is the leaking cycle.
Slot 4: the buffer is empty and ret_q carries 2, not 3. law_ok falls to 0 and stays there.
Slot 8: the two credits arrive back; credits stops at f, 15. The sixteenth never comes.
Hypothesis: each flush that coincides with an arriving word loses exactly one credit, and nothing else leaks. A good hypothesis predicts something you have not looked at yet. This one predicts a number: the link hangs when its 16th credit is lost, so at every hang exactly 16 flushes had a word arriving, however many flushes there were in total. It also predicts that a flush on an idle link never leaks, which is why the existing flush tests, written with the link quiet, passed.
The bench counts both on its WATCHDOG line, and the rate table above already has the answer: 35, 37, 44, 29 and 48 flushes, and in every run, 16 with a word arriving. Five runs at five rates agree with a prediction that could easily have failed (a second leak path would make the count smaller). The hypothesis survives, and it explains the hour: a leak needs a rare event and an unlucky cycle at once.
// snippet: the fix (FIX=1), clink's flush branch
ret_q <= level + 5'(arrive);$ iverilog -g2012 -o c.vvp leak_tb.sv
$ vvp -n c.vvp
PASS: 19382 words delivered, 618 flushed in 57 flushes, 26049 cycles, credits back to 16 (FIX=1)
$ for s in 1 2 3 4 5 6 7 8 9 10; do vvp -n c.vvp +SEED=$s +FLUSH=50 | head -1; done
PASS: 16181 words delivered, 3819 flushed in 481 flushes, 22758 cycles, credits back to 16 (FIX=1)
PASS: 16489 words delivered, 3511 flushed in 429 flushes, 23134 cycles, credits back to 16 (FIX=1)
PASS: 16283 words delivered, 3717 flushed in 455 flushes, 22825 cycles, credits back to 16 (FIX=1)
PASS: 16303 words delivered, 3697 flushed in 472 flushes, 22962 cycles, credits back to 16 (FIX=1)
PASS: 16224 words delivered, 3776 flushed in 465 flushes, 22618 cycles, credits back to 16 (FIX=1)
PASS: 16438 words delivered, 3562 flushed in 441 flushes, 22826 cycles, credits back to 16 (FIX=1)
PASS: 16365 words delivered, 3635 flushed in 459 flushes, 22919 cycles, credits back to 16 (FIX=1)
PASS: 16244 words delivered, 3756 flushed in 461 flushes, 22759 cycles, credits back to 16 (FIX=1)
PASS: 16438 words delivered, 3562 flushed in 424 flushes, 22921 cycles, credits back to 16 (FIX=1)
PASS: 16085 words delivered, 3915 flushed in 485 flushes, 22702 cycles, credits back to 16 (FIX=1)
$ /usr/bin/time -p vvp -n c.vvp +WORDS=10000000 +FLUSH=100000 2>&1 | grep -E "^PASS|^FAIL|^real"
PASS: 9998489 words delivered, 1511 flushed in 133 flushes, 13327091 cycles, credits back to 16 (FIX=1)
real 77.99The default run passes with the law on every cycle. The sweep uses the most aggressive flush rate, where the bug failed every seed; the fix passes them all, and the law never fires. The long run, the one that hung after 2.9 million cycles, now carries all ten million words: 9,998,489 delivered and 1,511 thrown away by its 133 flushes, with all 16 credits back at the end. The proof is cheap because the repro is fast: the stress that found the bug is the same stress that proves the fix.
Two things would have caught this before the lab. The credit law was cheap to write and fires at the first leak: keep it on in every test, not only when a hang is being debugged (the assertion lesson shows how to prove such a check fires on the bug and stays silent on the fix). And a coverage point on the coincidence: flush in the same cycle as an arriving word. The flush tests passed because they flushed a quiet link, so that cross was never hit. A directed test that flushes under full traffic, like the waveform's stimulus, pins the bug for good.
Which step won this hunt? Reproduce. Turning the flush rate up turned an hour into a hundredth of a second, and the law then named the cycle. Test made it certain with a count that could have come out wrong.
level + 1 credits on every flush, to cover the arriving word. Is it right?No. It is right only when a word is arriving. A flush on a quiet link would return one credit too many, and the sender would then overrun the buffer. The return must count exactly what was discarded: level + arrive. The credit law catches the wrong fix at once: the total would read 17 after the first quiet flush.
The link had more credits than its round trip needs. A credit spends about seven cycles away (two link stages, the buffer, the return register, three return stages) plus however long the word waits for the consumer, so at 0.75 words per cycle only part of the 16 are needed. Lost credits came out of the spare ones first. Throughput is a late and blunt sign of a leak; the law sees the first one.
leak_tb.sv as shown, 2026-10-18, Icarus Verilog 13.0: iverilog -g2012 -o c.vvp leak_tb.sv and iverilog -g2012 -Pleak_tb.FIX=0 -o cb.vvp leak_tb.sv. Long run: vvp -n cb.vvp +NOLAW +WORDS=10000000 +FLUSH=100000 +RATE=250000, WATCHDOG at cycle 2928785 after 15.9 s (/usr/bin/time -p, this Mac). Same run with the law on: CREDIT_LAW at cycle 298373. +FLUSH= 20000, 5000, 500 and 50 (with +NOLAW): hangs at cycles 640730, 147854, 14291 and 1894, always with 16 flushes that had a word arriving. +FLUSH=2000000000 +WORDS=2000000: PASS after 2000000 words and 2665564 cycles (14.5 s).
Fix: vvp -n c.vvp PASS; seeds 1 to 10 at the default settings and at +FLUSH=50 all pass with the law on (the bug fails all ten at +FLUSH=50); +WORDS=10000000 +FLUSH=100000 PASS (9998489 words delivered, 1511 flushed in 133 flushes, 13327091 cycles, 78 s). Verilator 5.048 (verilator --binary --timing --assert -Wno-fatal [-GFIX=0] --top-module leak_tb) prints the same PASS, CREDIT_LAW and WATCHDOG lines for the default seed.
Waveform: COURSE=dbg node scripts/tools/gen-waves.mjs case-studies:the-credit-leak on wave-src/the-credit-leak/leak-cycle.sv (the same link with FIX=0 and a directed flush); it matched its expectations and a second run changed nothing. Limits: one small link, simulated; the hour in the lab is an illustrative scale, not a measurement.