An engineer new to a project asks why the on-chip network's links have counters at the sending end instead of ready signals. The answer is one sentence: "The sender always knows how many slots the receiver has free, so it never sends a word that has nowhere to go." The rest of this lesson is why that one sentence is enough.

How can a sender know, without waiting, that the receiver has room?

The obvious answer, and why it breaks

Stop-and-go (lesson 4.2) tells the sender about a full buffer after it fills, and must reserve a round trip of room above the threshold for words already in flight. That room is idle most of the time, and its size depends on a latency that can change. Worse, the receiver's state is only known approximately at the sender: it knows "not yet told to stop", not "how much room is left".

The idea: permission in advance

With credits, the receiver tells the sender, in advance, exactly how much room it has, and the sender keeps count:

1. At start-up the sender's credit counter is set to the number of free slots at the receiver (lesson 4.4).

2. To send a word, the sender must hold a credit, and spends it: the counter goes down by one.

3. When the receiver removes a word from its buffer, freeing a slot, it sends one credit back.

4. When a credit arrives, the sender's counter goes up by one.

A sender with zero credits does not send. Every word in flight or in the buffer used a credit, and no credit is ever returned until its slot is free, so the receiver can never overflow, however long the path and however late the credits arrive. This is the credit invariant: credits at the sender + words in flight + words in the buffer + credits in flight back = the buffer size, always.

Build it: DCLink step 3

The listing is the simplest credit link: a 2-slot receiver buffer, so 2 credits, one pipeline stage each way, and a consumer that pauses for a few cycles. A checker asserts the credit invariant every cycle and checks that the buffer never overflows and every word arrives in order.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

// DCLink step 3: credit-based flow control in its simplest form. The receiver has a
// 2-slot buffer, so the sender starts with 2 credits. Sending a word spends a credit;
// when the receiver removes a word from its buffer it returns one credit. One pipeline
// stage each way. The receiver's consumer sometimes pauses, so the buffer fills, the
// credits run out, and the sender waits without ever overflowing the buffer.
module tb;
  localparam int B = 2;
  logic clk = 0;
  always #5 clk = ~clk;
  logic rst_n = 0;
  initial begin
    $dumpfile("wave.vcd"); $dumpvars(0, tb);
    #12 rst_n = 1;
  end
  int cyc = 0;
  always @(posedge clk) if (rst_n) cyc <= cyc + 1;

  // Sender
  logic [2:0] credits = B;
  logic [7:0] next_w = 0;
  wire        send = (credits != 0);
  // Channel: one register each way
  logic       fwd_vld = 0, crd_in = 0;
  logic [7:0] fwd_data = 0;
  // Receiver: B-slot FIFO, consumer pauses in cycles 3-6
  logic [7:0] fifo [0:B-1];
  logic [1:0] wp = 0, rp = 0;
  logic [2:0] occ = 0;
  wire        pause = (cyc >= 3 && cyc <= 6);
  wire        pop = (occ != 0) && !pause;
  wire [7:0]  rx_data = fifo[rp[0]];

  always @(posedge clk) if (rst_n) begin
    credits  <= credits - send + crd_in;
    if (send) next_w <= next_w + 1;
    fwd_vld  <= send; fwd_data <= next_w;
    crd_in   <= pop;                               // one credit back per word removed
    if (fwd_vld) begin fifo[wp[0]] <= fwd_data; wp <= wp + 1; end
    if (pop) rp <= rp + 1;
    occ <= occ + fwd_vld - pop;
  end

  // Checker
  logic [7:0] expect_w = 0; int errors = 0, got = 0;
  always @(posedge clk) if (rst_n) begin
    if (occ > B) errors++;
    if (credits + fwd_vld + occ + crd_in != B) errors++;   // every credit accounted for
    if (pop) begin if (rx_data != expect_w) errors++; expect_w++; got++; end
  end
  initial begin
    #200;
    if (errors == 0 && got >= 6) $display("PASS: %0d words delivered in order; the 2-slot buffer never overflowed", got);
    else $display("FAIL: errors %0d, delivered %0d", errors, got);
    $finish;
  end
endmodule

Watch it run

How to read it, slot by slot:

• Slots 0 and 1: the sender spends both credits (credits 2, 1) and sends two words (send). From slot 2 it has none, so it stops.

• Slot 2: the first word has arrived (occ = 1) and is removed immediately (pop). Its credit comes back in slot 3 (crd_in), and the counter shows it in slot 4, when the sender sends a third word.

• Slots 3 to 6: the consumer pauses (pause). Nothing is removed, the buffer fills to 2 (occ = 2 in slot 6), and no credits come back. The sender sits at 0 credits from slot 5 and waits. The buffer never exceeds its 2 slots.

• Slots 7 to 9: the consumer resumes, removes words (pop), credits return (crd_in in slots 8 and 9), and the sender restarts in slot 9.

The sender stopped because it had no credits; it never needed a message saying "stop". With only two credits and a four-cycle credit round trip, this link can never exceed half throughput, which is lesson 4.5's sizing rule.

The design space

How real protocols chose

What breaks

A designer builds a credit link but returns a credit when a word arrives at the receiver rather than when it leaves the buffer (illustrative). Under light traffic it works. When the consumer pauses, the receiver keeps returning credits for words it is still holding, the sender keeps sending, and the buffer overflows. A credit stands for a free slot, and a slot is free only when its word has left. The invariant check in the listing would flag this on the first pause.

Key takeaways

• With credits, the sender spends one per word and gets one back each time the receiver frees a slot; with no credits it waits.

• The receiver can never overflow, at any distance, because every word in flight or buffered used a credit.

• The credit invariant (credits + in flight + buffered + returning = buffer size) is the first thing to assert.

Related topics

• Part 4 overview: Flow Control

• Lesson 4.5: The Credit Loop and Buffer Sizing