A link's throughput is perfect after power-up. A few hours into a long test it has dropped by half. Nothing has failed, no error is logged, and a reset restores full speed. The next day it happens again. Engineers suspect thermal throttling, then a slow memory, then the test itself.

What kind of bug makes a link slowly lose throughput, and how do you catch it on the first cycle it happens?

The obvious answer, and why it breaks

The obvious place to look for a throughput problem is congestion: a busy destination, a slow arbiter, contention. Those show up as stalls that come and go with traffic. A link that gets permanently slower over time, until a reset, is not congested; it is losing something. In a credit-based link the thing it can lose is credits, and each lost credit is a buffer slot the sender will never use again.

The idea: credits are conserved, so check that they are

Every credit is always somewhere: at the sender, attached to a word in flight, as an occupied slot in the buffer, or on its way back. The total is fixed at the buffer size. Every credit bug breaks that conservation in one of four ways:

A leak is the most common and the most deceptive, because a credit loop degrades gracefully: losing one credit of twelve costs nothing if twelve was more than the round trip needed, and it shows only when enough are gone. That is why leaks are found hours or weeks into testing. The cure is to check conservation directly, every cycle, from the start.

Build it: a leak and the check that catches it

The listing's sender updates its counter with an if / else if: if it sends, decrement; else, if a credit arrived, increment. In a cycle where it does both, the arriving credit is silently dropped. The checker computes the conservation total every cycle and records the first cycle it is wrong, then compares throughput in the first round trip with a round trip much later.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

// A credit leak. The sender's counter is written as
//     if (send) credits--;  else if (crd_in) credits++;
// so in a cycle where it sends a word AND a credit comes back, the returned credit is
// dropped. Each such cycle loses one credit for ever, until the remaining credits are
// spaced so that a send and a return never coincide. Throughput falls and stays low,
// with nothing wrong anywhere else.
// The checker's conservation invariant catches the first lost credit.
module tb;
  localparam int L = 2, B = 4;
  logic clk = 0;
  always #5 clk = ~clk;
  logic rst_n = 0;
  initial begin
    $dumpfile("wave.vcd"); $dumpvars(0, tb);
    #12 rst_n = 1;
  end
  int cyc = 0;
  always @(posedge clk) if (rst_n) cyc <= cyc + 1;

  logic [3:0] credits = B;
  wire        send = (credits != 0);
  logic [L-1:0] fwd = 0, back = 0;
  wire        rx_vld = fwd[L-1], crd_in = back[L-1];
  logic [3:0] occ = 0;
  wire        pop = (occ != 0);

  always @(posedge clk) if (rst_n) begin
    if (send)        credits <= credits - 1;      // BUG: a credit arriving in this cycle is lost
    else if (crd_in) credits <= credits + 1;
    fwd  <= {fwd[L-2:0], send};
    back <= {back[L-2:0], pop};
    occ  <= occ + rx_vld - pop;
  end

  // Checker: conservation of credits.
  function automatic int ones(input logic [L-1:0] v);
    ones = 0; for (int i = 0; i < L; i++) ones += v[i];
  endfunction
  int first_leak = -1, early = 0, late = 0, left = 0;
  always @(posedge clk) if (rst_n) begin
    if (first_leak < 0 && credits + ones(fwd) + occ + ones(back) != B) first_leak = cyc;
    if (send && cyc < 6) early++;                  // the first round trip (6 cycles)
    if (send && cyc >= 40 && cyc < 46) late++;     // a round trip much later
    left = credits + ones(fwd) + occ + ones(back); // credits still in the system
  end
  initial begin
    #500;
    if (first_leak >= 0 && early == B && late == left && left < B)
      $display("PASS: conservation check flagged the first lost credit at cycle %0d; %0d of %0d credits leaked and throughput fell from %0d to %0d flits per 6-cycle round trip", first_leak, B - left, B, early, late);
    else $display("FAIL: first leak %0d, early %0d, late %0d, credits left %0d", first_leak, early, late, left);
    $finish;
  end
endmodule

The buggy two lines are the classic way to write it. The correct form updates once with both terms: credits <= credits - send + crd_in;, as in lesson 4.5's listing.

Watch it run

How to read it, slot by slot:

• Slots 0 to 3: the sender spends its four credits (credits 4, 3, 2, 1) on four words.

• Slot 5: the first credit returns (crd_in); the sender had none, so the counter goes to 1 and it sends again in slot 6.

• Slot 6: the sender sends and a credit arrives in the same cycle. The else if ignores the arrival: one credit is gone. The same happens in slot 8.

• From slot 12: only two credits are left in the loop, so the sender sends two words per six-cycle round trip instead of four (slots 12 and 14, then 18). The checker flagged the first loss in the cycle it happened; throughput alone would only have shown it as "a bit slower".

In this run the leak stops once the two remaining credits are spaced so that a send and a return never coincide again. Other traffic patterns keep leaking until the link stops altogether. Either way, nothing reports an error unless something checks conservation.

Checking conservation in practice

• In simulation: an assertion over the whole loop, as in the listing. It needs visibility of both ends, so it lives in a testbench or system-level checker.

• At each end: assertions that the sender's counter never exceeds the advertised limit, never goes negative, and that the receiver never returns more credits than words it has removed.

• In silicon: expose the sender's credit count and the receiver's occupancy for debug; a counter stuck below its expected idle value when the link is quiet means credits are missing.

The design space

How real protocols guard against it

What breaks

The opening story is the listing's bug in a large design (illustrative). The leak needed a send and a credit return to coincide on one specific internal path, which happened a few times an hour under heavy traffic. Each leaked credit was harmless until enough had gone to drop below the round trip. A conservation assertion added to the system testbench fired within the first thousand cycles of the next regression run, on the exact cycle and path.

Key takeaways

• Credits are conserved; leaks, double returns, counter overflow and one-sided resets all break conservation.

• Leaks are deceptive: throughput degrades gracefully and only drops once credits fall below the round trip.

• Write counters as one update (credits - send + return), assert conservation every cycle, and check idle counts in silicon.

Related topics

• Part 4 overview: Flow Control

• Lesson 4.5: The Credit Loop and Buffer Sizing