A new graphics card supports a link speed its motherboard has never heard of; the motherboard's slot has 16 lanes, but the card in it today has 8. Neither was designed knowing about the other. Yet within milliseconds of power-up they are talking, at the fastest speed and widest width both support. Nobody configured that. The two sides negotiated it.

How do two sides with different abilities agree on how to talk?

The obvious answer, and why it breaks

The obvious answer is to configure both sides the same, by strapping pins or firmware. That only works when one team builds both sides. For anything interoperable (a card and a slot, a cable and two devices, chips from different vendors) neither side knows what it will meet. Fixing a configuration means choosing the lowest setting anyone might support, wasting the hardware, or risking a mismatch that never links.

The idea: advertise, take the minimum, confirm

• Advertise: each side sends what it supports (highest speed, number of lanes, optional features), repeatedly, in a known training format.

• Compute the common set: each side independently takes the minimum speed, the minimum width and the intersection of features. Because both apply the same rule to the same two advertisements, they reach the same answer without a further exchange.

• Confirm: before committing, make sure the advertisement was received correctly (here, seeing the same word twice in a row), so one corrupted word cannot cause a disagreement.

• Acknowledge: tell the partner its advertisement has arrived (an ack bit in the training word, the role PCIe's TS2 plays), and keep sending until the partner has acknowledged yours. A side that stops as soon as it is satisfied can strand a partner that still needs one more good word.

• Commit, then start: switch to the agreed settings; only then begin normal traffic.

Build it: DCLink step 10

Side A supports speed 3 and 4 lanes; side B supports speed 2 and 8 lanes and starts two cycles later. Each sends a 7-bit training word {ack, speed, lanes} until it is active. A side moves to CONFIRM on the first word it receives and to ACK when the next word carries the same abilities; a missing word sends it back to SEND. In ACK it sets the ack bit, and once it sees the partner's ack it sends three more ack words and becomes ACTIVE. To test the protection, one word on the way to B is corrupted.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

// DCLink step 10: capability negotiation. After reset, each side repeatedly sends
// a training word {ack, speed, lanes}: its highest supported speed (a code 1-3), its
// lane count, and an ack bit. With ack = 0 (TS1) it says "here is what I support".
// Once it has received the partner's abilities twice in a row (so a corrupted word is
// not believed) it agrees on the lower speed and the smaller width and sends ack = 1
// (TS2): "I have your abilities". It becomes ACTIVE, and stops sending, only after it
// has seen the partner's ack and sent LINGER more ack words, so the partner is never
// left waiting for words that stopped coming. A missing word restarts confirmation.
module dclink_train #(parameter logic [1:0] SPEED = 1, parameter logic [3:0] LANES = 1,
                      parameter int LINGER = 3) (
  input  logic       clk, en,                   // en: this side has come out of reset
  input  logic [6:0] rx,                        // partner's word: {ack, speed, lanes}, 0 = none
  output logic [6:0] tx,
  output logic [1:0] st, speed,
  output logic [3:0] lanes
);
  localparam logic [1:0] SEND = 0, CONFIRM = 1, ACK = 2, ACTIVE = 3;
  logic [5:0] last = 0;                         // partner's abilities seen last cycle
  logic       seen = 0;                         // partner's ack received
  int         left = LINGER;                    // ack words still to send before ACTIVE
  initial begin st = SEND; speed = 0; lanes = 0; end
  wire partner_ack = rx[6] && rx[5:0] == last;
  assign tx = (en && st != ACTIVE) ? {st == ACK, SPEED, LANES} : 7'h0;

  always @(posedge clk) if (en) case (st)
    SEND:    if (rx != 0) begin st <= CONFIRM; last <= rx[5:0]; end
    CONFIRM: if (rx == 0) st <= SEND;                          // missing word: start again
             else if (rx[5:0] != last) last <= rx[5:0];        // mismatch: confirm the new one
             else begin                                        // twice in a row: believe it
               st    <= ACK;
               speed <= (rx[5:4] < SPEED) ? rx[5:4] : SPEED;
               lanes <= (rx[3:0] < LANES) ? rx[3:0] : LANES;
             end
    ACK: if (seen || partner_ack) begin
           seen <= 1;
           left <= left - 1;
           if (left == 1) st <= ACTIVE;
         end
    default: ;
  endcase
endmodule

module tb;
  logic clk = 0;
  always #5 clk = ~clk;
  initial begin $dumpfile("wave.vcd"); $dumpvars(0, tb); end
  int cyc = 0;
  always @(posedge clk) cyc <= cyc + 1;

  // Side A: speed 3, 4 lanes. Side B: speed 2, 8 lanes, out of reset two cycles later.
  logic [6:0] tx_a, tx_b;
  logic [1:0] st_a, st_b, speed_a, speed_b;
  logic [3:0] lanes_a, lanes_b;
  wire [6:0] rx_a = tx_b;
  wire [6:0] rx_b = (cyc == 4) ? tx_a ^ 7'h01 : tx_a;   // one word reaches B corrupted
  dclink_train #(.SPEED(3), .LANES(4)) a (.clk, .en(1'b1),     .rx(rx_a), .tx(tx_a), .st(st_a), .speed(speed_a), .lanes(lanes_a));
  dclink_train #(.SPEED(2), .LANES(8)) b (.clk, .en(cyc >= 3), .rx(rx_b), .tx(tx_b), .st(st_b), .speed(speed_b), .lanes(lanes_b));

  initial begin
    #300;
    if (st_a == 3 && st_b == 3 && speed_a == 2 && speed_b == 2 && lanes_a == 4 && lanes_b == 4)
      $display("PASS: both sides active despite a corrupted word; agreed on speed %0d with %0d lanes", speed_a, lanes_a);
    else $display("FAIL: A %0d/%0d/%0d B %0d/%0d/%0d", st_a, speed_a, lanes_a, st_b, speed_b, lanes_b);
    $finish;
  end
endmodule

Neither side ever learns the other's "decision": each computes the same minimum from the same two words. That is why the rule must be deterministic and identical on both sides.

Watch it run

How to read it, slot by slot (states: 0 = SEND, 1 = CONFIRM, 2 = ACK, 3 = ACTIVE; the top bit of a word, 0x40, is the ack):

• Slots 0 to 1: only A is sending (tx_a = 0x34); B has not started (tx_b = 0).

• Slot 2: B starts sending 0x28.

• Slot 3: each side has received the other's word once: both st_a and st_b are 1 (CONFIRM). But the word B receives now is corrupted (rx_b = 0x35, 5 lanes).

• Slot 4: A sees 0x28 a second time, agrees on speed 2 and 4 lanes, and enters ACK: tx_a becomes 0x74 (ack set). B's second word (0x34) does not match the corrupted 0x35, so B stays in CONFIRM and starts again with 0x34.

• Slot 6: B has now seen A's abilities twice, agrees on the same speed 2 and 4 lanes, and enters ACK (tx_b = 0x68). Each side now sees the other's ack.

• Slot 9: after three more ack words each, both are ACTIVE (3) and stop sending.

Without the ACK phase, A would have gone active in slot 4 and stopped sending; B, one good word short, would sit in CONFIRM waiting for a word that never comes. The ack is what turns "I am satisfied" into "we are both satisfied".

The design space

How real protocols chose

What breaks

An early version of a negotiation (illustrative) accepted the first training word it saw. A single bit error made one side read the partner's speed as 1 instead of 2; the two sides agreed on different speeds and the link never came up. Requiring two consecutive matching words fixed that, and exposed the next bug: each side stopped sending the moment it had confirmed. In the lab, where both sides started together and no word was ever corrupted, that always worked. On a board, one corrupted word on one side meant that side needed one more word than its partner, and the partner had already gone quiet: one end ACTIVE, the other stuck in CONFIRM. Adding the acknowledgement phase, plus a timeout that restarts training from SEND, removed it. Test negotiation with corruption on one side only and with the two sides starting at different times; the symmetric, error-free case hides both bugs.

Key takeaways

• Interoperable sides must negotiate: advertise abilities, take the common set, confirm, then commit.

• A deterministic, identical rule lets both sides reach the same answer without exchanging decisions.

• Protect the exchange against corruption, keep advertising until the partner acknowledges, and define a fallback if the agreed setting fails.

Related topics

• Part 11 overview: Bring-Up and Link Management

• Recovery and Degradation