CXL Attach Modes: Worked Example

Worked example

Worked Example for CXL Attach Modes focuses on cache/memory latency, coherency traffic, attach-mode bandwidth. The goal is to connect the observable symptom to protocol mechanism, ownership, and regression risk.

A product workload shows cache/memory latency, coherency traffic, attach-mode bandwidth. The first review mistake is to blame the whole interface. A better review starts by pinning one transaction, proving where protocol progress stopped, and checking whether the observed behavior is legal for CXL Attach Modes.

Sequence under inspection

SEQUENCE — CXL Attach Modes

  initiator            interconnect/PHY            target
      |  request (id) ------->  |                     |
      |                         |  forward ----------> |
      |                         |                     | work
      |                         |  <---- response ---- |
      |  <----- complete ------ |                     |
      |
   metric captured here: cache/memory latency, coherency traffic, attach-mode bandwidth

CXL.io / CXL.cache / CXL.mem

CXL SUB-PROTOCOLS (share PCIe PHY)

CXL.io   : PCIe-like discovery, config, DMA            (Type 1/2/3)
CXL.cache: device caches host memory coherently        (Type 1/2)
CXL.mem  : host accesses device-attached memory         (Type 2/3)

Type 3 example: memory expander = mostly CXL.mem + CXL.io.

1. Capture the failing waveform and transaction log.

2. Tag the request ID, address, endpoint, or lane.

3. Find the first response, retry, stall, or missing completion.

4. Compare against CXL mode table, coherency trace, memory window map.

5. Choose one reversible fix and write the regression list before editing RTL or firmware.

Did the fix work?

BEFORE / AFTER — CXL Attach Modes

           failing        target
metric  |    ●              ┄┄┄┄┄┄┄
        |     \
        |      \___ ● bounded fix
        |           \
        |            ● validated
        +-------------------------------> change set
Prove the mechanism moved the metric; one good dot is not proof.

Protocol deep dive

PCIe is reliable packet delivery over unreliable links; debug flows PHY -> DLL -> TLP -> firmware.

Concept diagram

PCIe DEBUG TOP-DOWN

L0 link healthy?  -> credits OK?  -> TLP completes?  -> driver happy?

Skip a layer and you will mis-own the bug.

Metric graph

LINK DEGRADE EXAMPLE

target x4 Gen4  ---- ---- ---- ----
actual   x4 Gen4  ---- ---- ---- ----   (eval board)
actual   x1 Gen3  -                   (product board)

Package/SI often shows up as width downgrade, not hard fail.

Metrics and artifacts to collect

• link width/speed

• replay count

• completion timeout

• AER error log

• LTSSM history

Mini case study

Endpoint enumerated but DMA timed out: completion credits exhausted because a switch port was misconfigured in firmware, not because the endpoint was broken.

Debug branches

• If degrade at width/speed, PHY/SI before driver.

• If replay storm, link layer before transaction layer.

• If CXL coherency bug, separate .io vs .cache vs .mem traffic.

Senior review question

Ask: what is the first transaction that deviates, and which spec rule does it test?

Key takeaways

• Connect every protocol claim to a transaction identity and measurable metric.

• Store the artifact (waveform, log, counter) next to every signoff decision.

Common pitfalls

• Debugging timeouts without finding the first bad transaction.

• Quoting peak bus width without payload efficiency and retry overhead.

• Treating VIP compliance as a substitute for system integration replay.

Narrative walkthrough

A team sees cache/memory latency, coherency traffic, attach-mode bandwidth drop 40% after a seemingly small change near CXL Attach Modes.

They almost widen the interface. Instead they capture id=7 read burst and find W beats never matched AW len.