A 16-lane link in a server has one lane with a damaged connector pin. That lane produces errors constantly; replay keeps up at first, then the replay counter rolls over, and the link drops. Without recovery, the server loses the device until a technician visits. With recovery, the link retrains with 8 lanes, at half the bandwidth, and keeps running until the next maintenance window.
What can a link do when errors are too frequent for replay to hide?
The obvious response to a link that keeps failing is to declare it dead and report a fatal error. That is safe, but it turns one failing wire among dozens into a total loss of service. Most link failures are partial: one lane, one marginal speed, a temporary loss of lock. A link that can step down gracefully keeps the system alive at reduced performance.
Step | Fixes | Costs |
|---|---|---|
Retrain (re-run alignment and equalisation) | Lost lock, drifted equalisation, a transient glitch | Microseconds to milliseconds without traffic |
Drop lanes (narrower link) | A broken or noisy lane | Bandwidth in proportion to lanes lost |
Lower the speed | A marginal channel at the top rate | Bandwidth; sometimes latency |
Declare the link down | Nothing: it contains the failure | The device is lost until reset or repair |
Detection with thresholds: a replay counter or error-rate monitor that decides when replay is no longer enough (PCIe uses a replay-number counter: when it rolls over, the link retrains).
Traffic that survives: transactions in flight during a retrain must be held and replayed, or completed with errors, never silently lost.
Reporting: every step down is logged and reported, because the system now runs slower than designed and someone should know (lesson 10.15).
A way back up: after repair or a quiet period, retrain at full width and speed.
Protocol | Recovery |
|---|---|
PCI Express | The LTSSM's Recovery state retrains the link; a replay counter that rolls over triggers retraining and is logged as a correctable error; links can be retrained to a lower speed or fewer lanes to keep working. |
High-speed Ethernet | Loss of lock or alignment triggers link re-establishment; FEC statistics expose degrading channels. |
A system (illustrative) allowed its link to drop lanes automatically but never reported it. A server ran for months at half its storage bandwidth after a lane failed, and performance investigations blamed the software. A degraded-mode report to the operating system, and a log entry, would have pointed straight at the connector.
It halves (at the same speed per lane).
The system now runs below its designed performance; without a report, the cause of the slowdown is invisible and the failing part is not replaced.
When replay is not enough, a link can retrain, drop lanes, or lower its speed before giving up.
Each step trades bandwidth for stability; only the last one declares the link dead.
Preserve in-flight traffic, report every degraded state, and provide a way back to full capability.