A request crosses a link, reaches a device, and the device pops a byte from its receive FIFO to answer it. The response is corrupted on the way back. The requester times out and sends the request again. The device pops another byte. The requester now has the second byte; the first is gone for ever. The replay that was meant to recover from an error caused one.
When is it safe to perform a request again, and how do protocols make replay safe?
The obvious recovery for a lost or damaged message is to send it again. For reading memory, that is harmless: reading twice gives the same value. But many operations change something: popping a FIFO, clearing a status bit, incrementing a counter, starting a DMA transfer. Doing them twice has a different effect from doing them once. Whether a replay is safe depends on the operation, and on whether the replay can reach the target a second time.
An operation is idempotent if performing it twice has the same effect as performing it once.
Operation | Idempotent? | Why |
|---|---|---|
Read of normal memory | Yes | Reading does not change anything |
Write of a value to normal memory | Yes | Writing 5 twice leaves 5 (if nothing wrote in between) |
Read of a FIFO data register | No | Each read pops an entry (lesson 5.4) |
Read-to-clear status | No | The second read sees nothing |
Increment, fetch-and-add, other atomics | No | Each performs the change again |
Write to a "start" or "push" register | No | Starts the job or pushes the item twice |
Replay only below the target. A link-level replay (lesson 10.11) resends a packet only until the receiver at the far end of that link has it, and that receiver discards duplicates by sequence number. The target behind it sees each request once, whatever the operation. This is why link replay is always safe.
Suppress duplicates end to end. When a request is retried end to end (after a timeout, through a fabric), the target must recognise a repeat and answer it from a saved result instead of performing it again: a tag or sequence number per request, and a record of recent completions.
What is never safe is an end-to-end retry of a non-idempotent request with no duplicate suppression: the opening FIFO story.
Where | Replay safety |
|---|---|
PCI Express data link layer | Replays packets on one link only; the receiver discards duplicates by sequence number, so the transaction layer sees each request once. |
Requests after a completion timeout | Software decides whether to retry, because the hardware cannot know whether the target performed the request. |
Network protocols | Duplicate suppression with request identifiers for non-idempotent operations. |
The opening system (illustrative) added an automatic end-to-end retry in an interconnect after a timeout, to hide occasional lost responses. Memory accesses were fine. Device register reads that popped FIFOs or cleared status bits occasionally lost data when the response, not the request, had been the casualty. The retry was removed for device regions, where the timeout now returns an error to software, which knows whether it may try again.
The add is performed twice; the counter is wrong by one increment.
The replay happens on one link, and the link's receiver discards the duplicate by sequence number before it reaches the transaction layer and the device.
Replay is safe only for idempotent operations: twice must have the same effect as once.
Link-level replay is always safe, because the far receiver discards duplicates before the target.
End-to-end retry needs idempotent targets or duplicate suppression; device accesses should surface errors to software.