A driver writes "power down" to a device's control register, receives the write response, and switches off the device's clock. The device never powers down cleanly. The write response had come from a buffer in the interconnect, not from the device, and the write was still in that buffer when the clock stopped. The driver read the response as "done". It meant "accepted".
What does a write response actually promise, and how does a requester get the promise it needs?
The obvious meaning of a write response is that the write has happened at its target. For many writes, waiting for that is wasteful: memory writes can be accepted by a nearby buffer and finished later, saving a long round trip (lesson 6.13). So many fabrics answer early, from the first buffer that takes responsibility for the write. Both meanings are useful; trouble starts when the requester assumes the stronger one and gets the weaker.
Early acknowledgement | Final (end-to-end) acknowledgement | |
|---|---|---|
Sent by | A buffer on the way (interconnect, bridge, write buffer) | The target, or the point of serialisation (lesson 7.3) |
Promises | The write is accepted and will be done; later writes the protocol orders with it (its ordering domain) will not overtake it | The write has reached its destination and is visible to later accesses |
Latency | Short | A full round trip |
Errors | Reported later, elsewhere, or not at all | Reported in the response |
Safe for | Normal memory, where nobody depends on exactly when it lands | Device registers, power and clock control, anything followed by an action outside the path |
Mark the write so buffers may not answer early: a non-bufferable attribute (lesson 5.12) makes the response come from the target.
Follow up with a read from the same target, on a path where reads cannot overtake earlier writes; the read's data cannot return until the write has landed (lesson 5.7).
Use a fence or barrier that waits for earlier writes to complete at their destinations (lesson 7.5).
An early acknowledgement does not remove the protocol's ordering rules, and it does not add any. A buffer that answers early must keep in order exactly the writes the specification says are ordered; writes the specification leaves unordered may still overtake each other inside or after it. So "same requester, same path" is not enough to make data-then-flag safe. The specification has to say which writes form an ordering domain. In AXI, that depends on the ID, the addresses and the memory type: writes with the same ID to overlapping addresses, or to Device memory in the same peripheral region, are observed in order. Two Normal-memory writes to different addresses are not ordered by sharing an ID, and writes with different IDs are not ordered at all.
Safe: the producer writes the data, waits for that write's response, then writes the flag. The response may come early from a buffer, because the buffer may give it only from a point where any later access to that address will see the data.
Not safe: the producer issues data and flag back to back, to different Normal-memory addresses, relying on the shared path. Both get early responses. The flag may become visible first, and the consumer reads stale data.
Before relying on any data-then-flag sequence, find three things in the specification: which writes share an ordering domain (ID, address, memory type, channel), what each buffer or memory attribute allows, and what a response promises about visibility. If any of these is missing, use one of the three methods above.
Protocol | What a write response means |
|---|---|
AXI | For a bufferable write, the response may come from an intermediate point; for a non-bufferable write, from the final destination. The AxCACHE attribute decides. |
PCI Express | Memory writes are posted, so there is no response at all; a read from the same device, which may not pass the earlier posted writes, confirms they arrived. |
DCLink | Every WR's CPL comes from R after the write is performed (step 4). |
The opening story: the device's control register was mapped as bufferable, so the power-down write was acknowledged early, and the clock was stopped with the write still in a buffer. The fix was to map device registers as non-bufferable, so the response came from the device itself; a read-back of the register before stopping the clock would also have worked.
Yes: early acknowledgements order writes only along their own path; different paths can reorder. Use the same path, a read-back, or a fence.
That the write will eventually be performed, and that later writes in the same ordering domain, as the specification defines it (in AXI: ID, address overlap and memory type), will not overtake it. It promises nothing about writes outside that domain, even on the same path.
A write response may mean "accepted" (early) or "arrived" (final); attributes decide which.
Early acknowledgement is safe for memory; device registers and writes followed by outside actions need final acknowledgement.
Get the stronger promise with a non-bufferable write, a read-back from the same target, or a fence.