Two processors write different values to the same address at almost the same time. Which write wins? Each processor issued its write first, from its own point of view. A trace on processor A's port shows A's write first; a trace on B's port shows B's first. The answer is decided somewhere else: at the one place in the fabric where the two writes first meet in a single line.
Where in a system is the order of requests from different sources actually decided?
The obvious answer is "in the order they were issued". But there is no global clock that says which of two requests from different sources came first, and each travels its own path at its own speed. Until two requests reach a point where they must go one after the other, their order is undefined. After that point, it is fixed.
The point of serialisation for a set of requests is the first place where they all pass through a single, in-order structure: an arbiter feeding a queue, a target's input buffer, or, in a coherent system, the home node responsible for an address. Requests are ordered from that point on, and every agent must agree with the order chosen there.
Which write wins to a shared address: the one serialised last.
What a read returns: the latest write serialised before it.
When a write may be acknowledged as globally visible: once it has been serialised, every later request to that address will see it. An acknowledgement sent earlier (from a buffer on the requester's side of that point) promises less (lesson 7.4).
System | Point of serialisation |
|---|---|
Shared bus | The bus itself: one transaction at a time, in arbitration order |
Interconnect with independent targets | Each target's input queue, separately: there is no single order across targets |
Coherent fabric (Part 13) | The home node (or directory) for each address: all requests for that address are ordered there |
Single in-order path | The first in-order queue the requests share |
The second row matters most: in a fabric with many targets there are many points of serialisation, one per target, and no global order. Two observers can see writes to two different targets in different orders unless something else (a fence, a protocol rule) makes them agree.
An interconnect (illustrative) acknowledged writes from a buffer on the requester side, before the arbiter in front of a shared memory. Requester A got its acknowledgement and signalled requester B, which read the location and saw an older value: A's write was still waiting at the arbiter behind other traffic. The acknowledgement had been sent from the wrong side of the point of serialisation. Moving it to after the arbiter fixed the bug, at the cost of a little latency.
Not necessarily: each target serialises its own requests, and nothing orders writes at one target against writes at another.
The write has not yet been ordered against other requests to the same location, so a later reader may still be serialised ahead of it and see the old value.
Requests from different sources have no order until they meet in one in-order structure: the point of serialisation.
That point decides which write wins and what reads return; every agent must agree with it.
Acknowledgements that promise visibility must come from at or after it; many targets mean many separate orders.