Two blocks sit far apart on a die, so the route between them has four pipeline stages in each direction. The link uses credit flow control and the receiver has a four-flit buffer. The receiver drains a flit every cycle it has one; it never holds anything back. Yet the measured throughput is 40% of one flit per cycle, and nobody can find a stall in the receiver.

Nothing is broken. The sender is waiting for credits that are still travelling. How big must the receiver's buffer be for a credit-controlled link to run at full speed?

The obvious answer, and why it breaks

The usual first answer is that the buffer only has to absorb bursts: if the receiver drains as fast as the sender sends, a buffer of a few flits is plenty, because nothing ever piles up in it.

That reasoning treats the buffer as a place where data waits. With credits, the buffer size is also the number of credits in circulation, and a credit is useless while it travels. After the sender spends a credit, that credit has to ride forward with its flit, wait for the receiver to free the slot, and ride back before it can be spent again. If the sender owns four credits and each one takes ten cycles to come home, it can send at most four flits every ten cycles, however fast the receiver drains. The buffer is not too small for the data; it is too small for the round trip.

The idea: credits circulate, and the loop has a length

Follow one credit around the loop. Count cycles from the moment it is spent to the moment it can be spent again; call that the credit round-trip time, RTT. Every credit is either at the sender, attached to a flit in flight, sitting in the receiver's buffer as an occupied slot, or travelling back. The total never changes: it is always B, the buffer size. That is the credit invariant, and it is the first thing to assert in any credit design.

Because each credit can be used at most once per RTT, the sender can send at most B flits per RTT:

For full throughput at one flit per cycle, the buffer must cover the round trip: B ≥ RTT. In units of bytes and seconds this is the bandwidth-delay product, a result as old as networking:

Count the round trip exactly

RTT is longer than the wires alone, and the extra cycles are where sizing mistakes hide. In the reference model used below, a credit spent in cycle t goes through these steps:

With L = 4 that is 10 cycles, so the opening example's four credits give 4 / 10 = 40%. In a real design, add every cycle of logic the credit passes through: arbitration for the return channel, clock-domain crossings, the receiver's own pipeline before a slot is truly free. Size from the worst-case round trip, not the typical one, because the loop runs at the speed of its slowest lap.

Build it: a model, then the circuit

First, a cycle model. It sends a flit whenever the sender holds a credit, moves flits and credits through L-stage pipelines, drains one flit per cycle at the receiver, asserts the credit invariant every cycle, and measures the steady-state rate for three link lengths and ten buffer sizes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

"""Cycle model of credit-based flow control over a pipelined link.

Each cycle the sender sends one flit if it holds a credit. A flit takes L cycles
to reach the receiver's buffer. The receiver removes one flit per drain slot and
returns one credit, which takes L cycles to reach the sender. The model checks
the credit invariant every cycle and measures steady-state throughput.
"""
from collections import deque


def simulate(L, B, cycles=6000, drain_every=1):
    credits = B                       # credits held by the sender
    fwd = deque([0] * L)              # forward pipeline: 1 = a flit in that stage
    back = deque([0] * L)             # return pipeline: 1 = a credit in that stage
    buf = 0                           # flits waiting in the receiver buffer
    sent_late = 0
    for t in range(cycles):
        arrive = fwd[-1]              # flit reaching the receiver this cycle
        crd_in = back[-1]             # credit reaching the sender this cycle
        send = 1 if credits > 0 else 0
        pop = 1 if (buf > 0 and t % drain_every == 0) else 0
        # rising edge: every register updates at once
        credits += crd_in - send
        buf += arrive - pop
        fwd.rotate(1); fwd[0] = send
        back.rotate(1); back[0] = pop
        # invariant: every credit is somewhere, and the buffer never overflows
        assert credits + sum(fwd) + buf + sum(back) == B
        assert 0 <= buf <= B
        if t >= cycles // 2:
            sent_late += send
    return sent_late / (cycles - cycles // 2)


BUFFERS = [1, 2, 4, 6, 8, 10, 12, 14, 16, 18]
print("throughput (flits per cycle) by buffer depth B; RTT = 2L + 2")
print("   B: " + " ".join(f"{b:>5}" for b in BUFFERS))
for L in (1, 4, 8):
    rtt = 2 * L + 2
    row = [simulate(L, b) for b in BUFFERS]
    for b, thr in zip(BUFFERS, row):
        assert abs(thr - min(1.0, b / rtt)) < 0.002, (L, b, thr)
    print(f"L={L:<2} " + " ".join(f"{x:5.3f}" for x in row) + f"   (RTT {rtt})")

# A receiver that drains only every other cycle caps throughput at 0.5.
half = [simulate(4, b, drain_every=2) for b in BUFFERS]
print("L=4, drain every 2nd cycle: " + " ".join(f"{x:5.3f}" for x in half))
print("PASS: measured throughput equals min(1, B / RTT) for every case")

It prints:

throughput (flits per cycle) by buffer depth B; RTT = 2L + 2
   B:     1     2     4     6     8    10    12    14    16    18
L=1  0.250 0.500 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000   (RTT 4)
L=4  0.100 0.200 0.400 0.600 0.800 1.000 1.000 1.000 1.000 1.000   (RTT 10)
L=8  0.056 0.111 0.223 0.334 0.445 0.555 0.666 0.777 0.889 1.000   (RTT 18)
L=4, drain every 2nd cycle: 0.100 0.200 0.400 0.500 0.500 0.500 0.500 0.500 0.500 0.500
PASS: measured throughput equals min(1, B / RTT) for every case

Every line is a straight ramp of slope 1/RTT that flattens at 100% exactly when B reaches the round trip. The last line of the output shows a receiver that drains only every other cycle: throughput is capped at 50%, and it reaches that cap at B = 6 (B = 4 gives only 40%). A receiver that drains at rate r needs only about r × RTT credits, here 0.5 × 10 = 5. Credits beyond that buy nothing.

Now the circuit. The testbench below builds the same loop in SystemVerilog with L = 2 and B = 3, so RTT = 6, and checks the invariant and the steady-state rate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

// Credit flow control over a pipelined link: L = 2 stages each way, B = 3 receiver
// slots. The credit round trip is RTT = 2L + 2 = 6 cycles, so the sender can send
// only B = 3 flits per 6 cycles: 50% throughput, even though the receiver drains
// a flit every cycle it has one. The checker confirms the credit invariant and the
// steady-state rate.
module tb;
  localparam int L = 2, B = 3, RTT = 2 * L + 2;
  reg clk = 0;
  always #5 clk = ~clk;
  reg rst_n = 0;
  initial begin
    $dumpfile("wave.vcd");
    $dumpvars(0, tb);
    #12 rst_n = 1;
  end

  // Sender: spends one credit per flit.
  reg  [3:0] credits = B;
  wire       send = (credits != 0);

  // Forward and return pipelines (one register per stage).
  reg  [L-1:0] fwd  = 0;
  reg  [L-1:0] back = 0;
  wire rx_vld = fwd[L-1];          // flit reaching the receiver this cycle
  wire crd_in = back[L-1];         // credit reaching the sender this cycle

  // Receiver: a B-slot buffer, drained one flit per cycle; each drained slot
  // sends one credit back.
  reg  [3:0] occ = 0;
  wire       pop = (occ != 0);

  always @(posedge clk) begin
    if (!rst_n) begin
      credits <= B; fwd <= 0; back <= 0; occ <= 0;
    end else begin
      credits <= credits + crd_in - send;
      fwd     <= {fwd[L-2:0], send};
      back    <= {back[L-2:0], pop};
      occ     <= occ + rx_vld - pop;
    end
  end

  // Checker
  integer t = 0, sends = 0, errors = 0;
  function automatic int ones(input [L-1:0] v);
    ones = 0;
    for (int i = 0; i < L; i++) ones += v[i];
  endfunction
  always @(posedge clk) if (rst_n) begin
    if (credits + ones(fwd) + occ + ones(back) != B) begin
      $display("ERROR: credit invariant broken at cycle %0d", t);
      errors++;
    end
    if (occ > B) begin
      $display("ERROR: receiver buffer overflow at cycle %0d", t);
      errors++;
    end
    if (t >= 2 * RTT && t < 6 * RTT) sends += send;  // four round trips in steady state
    t++;
  end

  initial begin
    #400;
    if (errors == 0 && sends == 4 * B)
      $display("PASS: %0d flits in %0d cycles (B = %0d per RTT of %0d cycles), no overflow", sends, 4 * RTT, B, RTT);
    else
      $display("FAIL: %0d errors, %0d flits in %0d cycles", errors, sends, 4 * RTT);
    $finish;
  end
endmodule

Watch it run

How to read it, slot by slot:

• Slots 0–2: the sender spends its three credits (credits 3, 2, 1; send high).

• Slots 3–5: credits is 0 and send is low. The receiver is not stalled: it drains a flit in each of these slots (pop high). The credits it frees are on the return path.

• Slot 5: the first credit arrives (crd_in). The counter updates on the next edge, and in slot 6 the sender sends again. The credit spent in slot 0 was reused in slot 6: one round trip of six cycles.

• From slot 6: the pattern repeats, three flits on and three off. Throughput is 3 / 6 = 50%, as the formula says. With B = 6 the sender would never run dry.

Try it

Change the pipeline stages, the credits, the batch size and the drain rate, then step through the loop. The plot underneath measures steady throughput for every buffer size from 1 to 32, using the same model as the listing above.

Worked numbers: from cycles to bytes

Real links are sized in bytes and nanoseconds. With illustrative numbers: a link carrying 16 GB/s whose credit round trip (wire, logic, return arbitration and update) is 500 ns needs

buffer >= bandwidth x RTT
       = 16 GB/s x 500 ns
       = 16e9 B/s x 500e-9 s
       = 8,000 bytes
In 16-byte data credits (the PCIe data-credit unit): 8,000 / 16 = 500 data credits

Halve the round trip and the buffer halves. Double the bandwidth and it doubles. This is why buffer area grows with both link speed and distance, and why every cycle shaved off the credit return path is worth real silicon.

The sizing rule is a case of Little's law: the average number of items in a system equals their throughput times the time each spends there. Credits in circulation are the items, the round trip is the time, so full rate needs B ≥ 1 × RTT. The same law limits any requester with a cap on outstanding requests (Part 6): to keep a memory with 40 cycles of latency busy at half a request per cycle, it needs 20 requests in flight.

The design space

How real protocols chose

What breaks

A late timing fix adds two pipeline stages in each direction to a long route between two subsystems (illustrative numbers). Before the change, L = 3 and the receiver buffer is B = 8: RTT = 2 × 3 + 2 = 8, exactly covered, so the link runs at 100%. After the change, L = 5, RTT = 12, and the same buffer gives 8 / 12 = 67%. Functional regressions all pass, because a slower link is still a correct link. The drop appears weeks later as a performance regression with no failing test.

The evidence is in the waveform: bursts of exactly B sends separated by gaps, while the receiver's buffer drains and sits empty. That signature, sender out of credits while the receiver is idle, always means the loop is longer than the buffer. The fix is a 12-entry buffer (B = RTT = 12). Shortening the loop helps only in proportion: returning credits a cycle earlier at each end brings RTT to 10, so 8 / 10 = 80%, better but not full rate. Without a bigger buffer, full rate needs RTT back to 8, which means taking all four added cycles back out of the loop, in either direction. The lasting fix is a sizing assertion: a check that compares the buffer parameter with the round trip the integration actually built, so the next timing fix cannot silently break it.

Key takeaways

• With credit flow control, throughput = min(1, B / RTT): each credit can be used at most once per round trip.

• Size the buffer from the worst-case credit round trip, counted cycle by cycle, times the rate you must sustain: bandwidth × RTT in bytes.

• Sender out of credits while the receiver sits idle means the loop is longer than the buffer; check the invariant and the sizing, not the receiver.

Related topics

• Part 4 overview: Flow Control

• Architecture course: Credit Flow Control and Buffers

• Lesson 3.3: The Valid/Ready Handshake