An accelerator card wants to work on data in the host's memory without copying it, keeping its own cache of what it uses, coherent with the processors'. A memory expansion card wants to add terabytes of memory that the host can load and store directly. PCI Express can move the bytes but has no coherence and treats device memory as I/O. CXL keeps PCIe's wires and adds the missing questions' answers as two new protocols beside the old one.
How does CXL answer each question, and which answers are new compared with PCIe?
Constraint | CXL |
|---|---|
C1 Distance and medium | PCIe slots and cables: the PCIe physical layer (fact P38) |
C2 Bandwidth and latency | PCIe bandwidth, with low latency for coherent and memory accesses |
C3 Wires | PCIe lanes |
C4 Topology | Host to device, and from CXL 2.0 through switches |
C5 Who builds each side | Processor, accelerator and memory vendors; CXL Consortium specification with compliance testing (fact P38) |
Each answer links to the lesson that explains the idea behind it.
# | Question | Answer |
|---|---|---|
1 | When is the data valid? | The PCIe physical layer: embedded clock, recovered (fact P38) (1.4) |
2 | Where does a message start and end? | 68-byte flits (CXL 1.1, 2.0) and 256-byte flits (CXL 3.0) (fact P38) (2.10) |
3 | Is the receiver ready? | Credits, as PCIe (4.3) |
4 | How much may I send before I hear back? | Credits per protocol and message class (4.6) |
5 | What exactly is being asked? | Three protocols: CXL.io (PCIe transactions), CXL.cache (coherent requests from the device), CXL.mem (memory reads and writes from the host) (fact P38) (5.6) |
6 | How many requests can be in flight? | Many outstanding requests in each protocol, matched by tags (6.4) |
7 | In what order do results become visible? | CXL.io keeps PCIe's ordering; CXL.cache and CXL.mem define their own (fact P38) (7.6) |
8 | Who gets the wire? | The link layer arbitrates between the three protocols on one link (fact P38) (8.5) |
9 | Why can it hang? | Separate channels and credits per protocol and message class, so I/O traffic cannot block coherence or memory responses (9.4) |
10 | What if bits go wrong? | Flit CRC and link-layer retry; FEC with 256-byte flits at PCIe 6.0 rates (facts P10, P38) (10.11) |
11 | How do the two sides agree at start-up? | CXL mode is negotiated during PCIe link training; devices are discovered through CXL.io like PCIe devices (fact P38) (11.5) |
12 | How do we save power without losing data? | Built on PCIe's link power management |
13 | What if there are several copies of one value? | Coherence is new: CXL.cache lets Type 1 and Type 2 devices cache host memory coherently, and Type 2 and Type 3 devices expose memory the host can use (fact P38) (13.7) |
14 | Who is allowed to ask? | IDE for CXL from CXL 2.0 (fact P38) (14.4) |
15 | How do we know it is correct? | Compliance testing defined in the CXL specification (fact P38) (15.8) |
Reuse the link, add protocols: CXL.io is PCIe, so discovery, configuration and software support come for free.
Coherence and memory semantics on a serial link: CXL.cache and CXL.mem answer questions 5, 7 and 13 differently from PCIe.
Device types by need: caching only (Type 1), caching plus memory (Type 2), memory expansion (Type 3).
Latency matters: coherent and memory traffic need lower latency than I/O, so they travel in their own protocols rather than as PCIe transactions.
A memory expansion device (illustrative) was validated with bandwidth tests, which passed. Under a real database, tail latency was several times higher than expected, because the device's controller handled CXL.mem reads behind a queue it shared with background maintenance. Memory accessed by loads and stores is judged by latency, not bandwidth; separating the queues fixed it.
CXL.io for discovery and configuration and CXL.mem for the host's loads and stores; it does not cache host memory, so no CXL.cache.
They need different semantics (coherence, memory access), different ordering and lower latency than PCIe-style I/O transactions.
CXL multiplexes CXL.io, CXL.cache and CXL.mem over the PCIe physical layer.
CXL.io reuses PCIe; the other two add coherence and memory semantics with their own ordering.
Device types 1 to 3 combine the protocols; latency, not only bandwidth, decides success.