A graphics card from one vendor plugs into a motherboard from another, over a connector and traces that neither designed, at 32 gigatransfers per second per lane. It must find the other side, train the link, agree on speed and width, survive bit errors without losing a packet, keep producer-consumer ordering for software written decades ago, sleep when idle, and pass a compliance test. PCI Express is what a protocol looks like when every one of the fifteen questions matters.
How does PCI Express answer each question, and which constraint drives each answer?
Constraint | PCI Express |
|---|---|
C1 Distance and medium | Boards, connectors and cables: lossy channels at very high rates |
C2 Bandwidth and latency | Very high bandwidth per lane (fact P17), latency in hundreds of nanoseconds |
C3 Wires | Few: one differential pair each way per lane, 1 to 32 lanes (fact P13) |
C4 Topology | Point-to-point links forming a tree through switches |
C5 Who builds each side | Any vendor; must interoperate across generations; compliance tested (fact P37) |
Each answer links to the lesson that explains the idea behind it.
# | Question | Answer |
|---|---|---|
1 | When is the data valid? | Clock embedded in the data and recovered by CDR; SKP ordered sets compensate for clock differences (fact P11) (1.4) |
2 | Where does a message start and end? | 8b/10b (Gen1-2), 128b/130b blocks (Gen3-5), 256-byte flits (Gen6), with framing tokens (facts P8, P10, P12, P23) (2.6) |
3 | Is the receiver ready? | No ready signal: a sender may transmit only with credits for the receiver's buffers (4.3) |
4 | How much may I send before I hear back? | Six credit types (posted, non-posted, completion; header and data), initialised with InitFC and returned with UpdateFC (facts P1 to P4) (4.6) |
5 | What exactly is being asked? | Memory, I/O, configuration and message requests; posted writes and non-posted reads; atomics; no request crosses 4 KB (facts P9, P15, P19, P22) (5.7) |
6 | How many requests can be in flight? | Many outstanding non-posted requests matched by tag; completions may be split and may return out of order across tags (facts P24, P25) (6.8) |
7 | In what order do results become visible? | The producer-consumer ordering table, with relaxed ordering and ID-based ordering as opt-outs (facts P20, P21, P28) (7.6) |
8 | Who gets the wire? | Point-to-point links joined by switches; traffic classes mapped to virtual channels (fact P27) (8.8) |
9 | Why can it hang? | Posted writes must be able to pass non-posted requests and completions, which breaks the classic request/response deadlock (fact P28) (9.4) |
10 | What if bits go wrong? | Non-flit mode (Gen1 to Gen5): per-TLP LCRC, 12-bit sequence numbers, ACK/NAK and go-back-N replay. Flit mode (Gen6, 64 GT/s): 256-byte flits with FEC and a flit CRC; sequence numbers, ACK/NAK and replay work per flit and TLPs carry no LCRC. Both: optional end-to-end ECRC, AER reporting (facts P5 to P7, P10, P30, P40) (10.11) |
11 | How do the two sides agree at start-up? | LTSSM: Detect, Polling, Configuration to L0, with Recovery for retraining and speed changes; software enumeration through configuration space (facts P32, P33) (11.4) |
12 | How do we save power without losing data? | Device states D0 to D3cold, link states L0s, L1 (with substates), L2; ASPM; WAKE# or beacon (fact P34) (12.2) |
13 | What if there are several copies of one value? | Not a coherence protocol; DMA is normally snooped by the host, and No Snoop opts out (fact P21); CXL adds coherence on the same physical layer (13.7) |
14 | Who is allowed to ask? | Requester IDs, ACS, ATS and PASID with the host IOMMU, SR-IOV, and IDE link encryption (fact P35) (14.3) |
15 | How do we know it is correct? | PCI-SIG compliance workshops and the Integrators List (fact P37) (15.8) |
Layering: physical, data link and transaction layers each answer separate questions, so generations can change encoding without changing transactions. Gen1 to Gen5 changed only the physical layer (8b/10b, then 128b/130b). Gen6's flit mode went further and changed the data link layer too: reliability moved from per-TLP LCRC and sequence numbers to per-flit CRC, FEC and replay (fact P40). The transaction layer, and so software, stayed the same.
Reliable links, lossless fabric: per-link replay plus credits mean no TLP is dropped for errors or congestion.
Software compatibility: the ordering model and configuration space preserve the behaviour of the PCI bus that software was written for.
Interoperability machinery: training, negotiation, enumeration and compliance exist because any vendor's part must work with any other's.
An endpoint (illustrative) advertised more completion credits than its buffer could hold, assuming the host would never have that many reads outstanding. Under a new driver with deeper queues, completions arrived faster than it could drain, its buffer overflowed and data was corrupted silently. The fix was in the endpoint's read side, not its credits: endpoints must advertise infinite completion credits (fact P3), so an endpoint has to limit its outstanding reads to the completions its buffer can always sink.
Only the physical layer; transactions, credits and ordering are unchanged. (Gen6 did more than change the line code: its flit mode also reworked the data link layer's framing and replay, still without changing transactions.)
Otherwise a non-posted request waiting for a completion can block the posted writes the completer needs to make progress, a deadlock.
PCIe answers every question: embedded clocks, block codes, credits, posted and non-posted transactions, ordering table, replay, training, power, security, compliance.
Layering lets each generation change the lower layers (the physical layer, and in Gen6 flit mode the data link layer too) while keeping transactions and software unchanged.
Severe constraints (lossy channels, unknown partners, legacy software) explain its complexity.