A link between two chips runs at tens of gigabits per second. Every so often, a bit flips on the wire. The receiver's CRC check catches it: packet 3 is corrupt, so the receiver throws it away. By then the sender has already sent packets 4, 5, 6 and 7, and it has no reason to think anything went wrong.

The upper layers expect every packet, exactly once, in order. How does the link put packet 3 back where it belongs, without the layers above ever noticing?

The obvious answers, and why they break

Answer one: the receiver asks for packet 3 again. The sender sent it and forgot it; it has nothing to resend. And if the receiver keeps packets 4 to 7 while it waits, it has to hold them and reorder later, which costs buffering at the receiver and complicates every packet.

Answer two: correct the error at the receiver. That is forward error correction (lesson 10.6). It works for the errors its code can fix, costs bandwidth and latency on every packet, and still needs a fallback for errors beyond its strength.

Answer three: leave it to software. A missing packet becomes a timeout far up the stack, orders of magnitude slower than the link's own round trip, and every layer above has to handle it. The end-to-end argument says the ends must check correctness anyway, but a link that fails often makes everything above it slow. Recovering at the link is the performance enhancement the argument allows.

The idea: keep a copy until the receiver says it arrived

Link-level replay rests on four pieces, each simple on its own:

1. Sequence numbers. The sender numbers every packet. The receiver knows which number it expects next, so it can tell a missing, repeated or out-of-order packet from a good one.

2. A replay buffer at the sender. Every packet sent is also kept, until an acknowledgement covers it. Only then is its slot freed.

3. Cumulative ACKs. The receiver accepts only the packet it expects, intact. An ACK carrying number n means "everything up to and including n arrived", so one ACK can free many slots and a lost ACK is repaired by the next one.

4. A NAK and a replay. When the receiver sees a bad CRC or a gap in the numbers, it drops the packet and sends one NAK carrying the number of the last good packet. The sender frees everything up to that number and resends everything after it, oldest first. A replay timer covers the case where the NAK, or every ACK, is lost: if nothing comes back for too long, the sender replays anyway.

The receiver drops packets 4 to 7 even though they were intact. That is the price of the simplest scheme, go-back-N: the receiver never buffers out-of-order packets, so it needs no reordering logic, and the sender resends a little more than it strictly must. Lesson 10.12 compares it with selective repeat, which resends only the bad packet.

Build it: DCLink step 9

The listing below is the circuit that produced the waveform further down. To stay short, every packet is a single flit carrying a 4-bit sequence number, an 8-bit payload and a CRC-8; there are two pipeline stages each way; and the replay timer is left out (the NAK path is enough to show the mechanism). The channel corrupts the first transmission of packet 3. A checker confirms that all eight payloads arrive once and in order.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

// Link-level replay (DCLink step 9, single-flit packets). The sender stamps each
// packet with a 4-bit sequence number and a CRC-8, keeps a copy in an 8-entry
// replay buffer, and frees copies when a cumulative ACK covers them. The channel
// corrupts the first transmission of packet 3. The receiver drops it and every
// later packet until packet 3 arrives intact, and sends one NAK naming the last
// good packet. The sender then replays from packet 3 onwards (go-back-N).
// The checker confirms all eight payloads are delivered once, in order.
module tb;
  localparam int L = 2;            // pipeline stages each way
  localparam int N = 8;            // packets to deliver
  reg clk = 0;
  always #5 clk = ~clk;
  reg rst_n = 0;
  initial begin
    $dumpfile("wave.vcd");
    $dumpvars(0, tb);
    #12 rst_n = 1;
  end

  // CRC-8, polynomial x^8 + x^2 + x + 1, initial value 0, over the 12 bits {seq, payload}.
  function automatic [7:0] crc8(input [11:0] d);
    reg [7:0] c;
    c = 8'h00;
    for (int i = 11; i >= 0; i--)
      c = {c[6:0], 1'b0} ^ ((c[7] ^ d[i]) ? 8'h07 : 8'h00);
    return c;
  endfunction

  // ---------------- Sender ----------------
  reg  [7:0] rbuf [0:7];           // replay buffer: payload by seq[2:0]
  reg  [3:0] next_seq = 0;         // sequence number of the next new packet
  reg  [3:0] send_ptr = 0;         // sequence number of the next packet to put on the wire
  reg  [3:0] una      = 0;         // oldest unacknowledged sequence number
  reg  [3:0] made     = 0;         // new packets created so far
  wire       replaying = (send_ptr != next_seq);
  wire       room      = (4'(next_seq - una) < 4'd8);   // replay buffer not full
  wire       new_pkt   = !replaying && room && (made < N);
  wire       tx_vld    = replaying || new_pkt;
  wire [3:0] tx_seq    = send_ptr;
  wire [7:0] tx_pay    = replaying ? rbuf[send_ptr[2:0]] : 8'hA0 + made;
  wire [7:0] tx_crc    = crc8({tx_seq, tx_pay});

  // ACK/NAK arriving from the receiver (end of the return pipeline)
  wire       ack_in, nak_in;
  wire [3:0] ack_seq_in;

  always @(posedge clk) begin
    if (!rst_n) begin
      next_seq <= 0; send_ptr <= 0; una <= 0; made <= 0;
    end else begin
      if (new_pkt) begin
        rbuf[next_seq[2:0]] <= tx_pay;   // keep a copy until it is acknowledged
        next_seq <= next_seq + 1;
        made     <= made + 1;
      end
      if (tx_vld) send_ptr <= send_ptr + 1;
      if (ack_in || nak_in) una <= ack_seq_in + 1;     // cumulative: frees everything up to ack_seq
      if (nak_in) send_ptr <= ack_seq_in + 1;          // go back: resend from the first unacknowledged packet
    end
  end

  // ---------------- Channel: L stages forward, one injected error ----------------
  reg        corrupted_once = 0;
  wire       corrupt = tx_vld && tx_seq == 4'd3 && !corrupted_once;
  always @(posedge clk) if (corrupt) corrupted_once <= 1;
  reg [20:0] fwd [0:L-1];          // {vld, seq, payload, crc}
  always @(posedge clk) begin
    fwd[0] <= rst_n ? {tx_vld, tx_seq, tx_pay ^ (corrupt ? 8'h01 : 8'h00), tx_crc} : 21'd0;
    for (int i = 1; i < L; i++) fwd[i] <= rst_n ? fwd[i-1] : 21'd0;
  end
  wire       rx_vld = fwd[L-1][20];
  wire [3:0] rx_seq = fwd[L-1][19:16];
  wire [7:0] rx_pay = fwd[L-1][15:8];
  wire [7:0] rx_crc = fwd[L-1][7:0];

  // ---------------- Receiver ----------------
  reg  [3:0] expected = 0;
  reg        nak_sent = 0;
  wire       crc_ok    = (crc8({rx_seq, rx_pay}) == rx_crc);
  wire       rx_accept = rx_vld && crc_ok && rx_seq == expected;
  wire       send_nak  = rx_vld && !rx_accept && !nak_sent;
  reg  [7:0] delivered [0:N-1];
  integer    ndel = 0;
  always @(posedge clk) begin
    if (!rst_n) begin
      expected <= 0; nak_sent <= 0;
    end else if (rx_accept) begin
      expected <= expected + 1;
      nak_sent <= 0;
      delivered[ndel] <= rx_pay;
      ndel <= ndel + 1;
    end else if (send_nak) begin
      nak_sent <= 1;
    end
  end

  // Return path: L stages carrying {ack, nak, ack_seq}. ACK names the packet just
  // accepted; NAK names the last good packet (expected - 1); otherwise ack_seq is 0.
  reg [5:0] back [0:L-1];
  always @(posedge clk) begin
    back[0] <= rst_n ? {rx_accept, send_nak, rx_accept ? rx_seq : send_nak ? 4'(expected - 1) : 4'd0} : 6'd0;
    for (int i = 1; i < L; i++) back[i] <= rst_n ? back[i-1] : 6'd0;
  end
  assign ack_in     = back[L-1][5];
  assign nak_in     = back[L-1][4];
  assign ack_seq_in = back[L-1][3:0];

  // ---------------- Checker ----------------
  integer errors = 0, sends = 0;
  always @(posedge clk) if (rst_n && tx_vld) sends++;
  initial begin
    #400;
    for (int i = 0; i < N; i++)
      if (i >= ndel || delivered[i] !== 8'hA0 + i) errors++;
    if (errors == 0 && ndel == N && una == 4'(N))
      $display("PASS: %0d packets delivered once and in order, %0d transmissions (%0d replayed)", ndel, sends, sends - N);
    else
      $display("FAIL: %0d delivered, %0d errors, una = %0d", ndel, errors, una);
    $finish;
  end
endmodule

The sender's whole recovery logic is two lines: una <= ack_seq_in + 1 frees every slot an ACK or NAK covers, and send_ptr <= ack_seq_in + 1 on a NAK rewinds the transmit pointer to the first unacknowledged packet. While send_ptr is behind next_seq, the sender is replaying from its buffer instead of making new packets.

Watch it run

How to read it, slot by slot. Sequence-number lanes mean something only while their valid lane (tx_vld or rx_vld) is high.

• Slots 0–7: the sender transmits packets 0 to 7. In slot 3, corrupt is high: the channel flips one payload bit of packet 3.

• Slots 2–4: packets 0, 1 and 2 reach the receiver two cycles later and are accepted (rx_accept).

• Slot 5: corrupted packet 3 arrives. Its CRC fails, so rx_accept stays low and the receiver sends a NAK naming packet 2, its last good packet.

• Slots 6–9: packets 4 to 7 arrive intact but are dropped: the receiver still expects 3.

• Slots 4–6 and 7: ACKs for 0, 1 and 2 reach the sender (ack_in), and una, the oldest unacknowledged number, steps to 1, 2, 3. In slot 7 the NAK arrives (nak_in).

• Slots 8–12: the sender replays packets 3 to 7 from its buffer. They arrive in slots 10 to 14 and are accepted in order; their ACKs move una from 4 to 8 in slots 13 to 17, emptying the buffer.

The single bit error cost five extra transmissions: packet 3 itself and the four packets behind it. Those four were the packets sent during the time it took the NAK to come back. On a longer or faster link, more packets fit into that time, so each error costs more. That is the first reason replay buffers and error rates are designed together.

Size the replay buffer

A copy leaves the buffer only when its acknowledgement arrives. So the buffer must hold everything the sender can transmit during one acknowledgement round trip: the packet's own transmission, the trip forward, the receiver's CRC check, any time the receiver waits before sending an ACK (coalescing), the trip back, and the sender's processing. If the buffer is smaller, it fills and the sender stalls, exactly like a credit loop that is shorter than its round trip (lesson 4.5):

With illustrative numbers: a link moving 4 GB/s of packets, with an acknowledgement round trip of 600 ns and a largest packet of 280 bytes including its header:

replay buffer >= 4 GB/s x 600 ns + 280 B
              = 4e9 B/s x 600e-9 s + 280 B
              = 2,400 B + 280 B
              = 2,680 bytes

The sequence-number field has its own limit. With n-bit numbers, go-back-N may keep at most 2ⁿ − 1 packets unacknowledged; with 2ⁿ outstanding, a full window of new packets and a full window of replays would carry identical numbers and the receiver could not tell them apart. The listing uses 4-bit numbers (at most 15 outstanding) and an 8-entry buffer, so the buffer is the tighter limit. PCI Express gives every packet a 12-bit sequence number.

The design space

How real protocols chose

What breaks

A team reduces return-path traffic by sending one ACK per 16 packets instead of one per packet (illustrative numbers). With 64-byte packets at 4 GB/s, one packet takes 16 ns, so the receiver now holds an ACK for up to 16 × 16 = 256 ns. The acknowledgement round trip grows from 600 ns to 856 ns, and the replay buffer from the sizing above, 2,680 bytes, is now too small:

needed = 4 GB/s x 856 ns + 280 B = 3,424 B + 280 B = 3,704 bytes
usable = 2,680 B - 280 B = 2,400 bytes for the round trip
throughput while the buffer limits it = 2,400 / 3,424 = about 70%

Nothing fails functionally. The link simply stalls whenever the replay buffer fills, waiting for an ACK that the receiver is deliberately delaying. The signature on a waveform is the same as a short credit loop: the sender idle with its buffer full while the receiver has nothing to do. The fix is to size the buffer for the new round trip, or to ACK more often when the buffer is filling, which many designs do: they coalesce ACKs on a lightly loaded link and ACK promptly when the sender is running short.

Key takeaways

• Link-level replay: number every packet, keep a copy until a cumulative ACK covers it, and on a NAK or timeout resend everything after the last good packet.

• The replay buffer must cover bandwidth × the acknowledgement round trip, including ACK coalescing, plus one maximum packet, or the sender stalls with no errors at all.

• With n-bit sequence numbers, go-back-N allows at most 2ⁿ − 1 packets outstanding; watch the replay rate, because replay hides a failing link.

Related topics

• Part 10 overview: Reliability and Recovery

• Lesson 4.5: The Credit Loop and Buffer Sizing

• RTL course: Parity Datapath