A design review asks: "Does this interconnect keep transactions in order?" One engineer says yes, another says no. Both are right, because the question is incomplete. The interconnect keeps writes with the same ID in order, keeps accesses to the same address in order, and keeps nothing else in order. "In order" always means "in order within some scope".
Which transactions does a protocol keep in order, and why not all of them?
The obvious wish is total order: every transaction becomes visible to everyone in the order it was issued. It is easy to reason about and very expensive: every transaction must wait for every earlier one, across all paths and all targets, which removes the concurrency Part 6 worked to create. So protocols promise order only within a scope, as narrow as correctness allows, and leave everything outside it free.
Scope | Kept in order | Free | Cost |
|---|---|---|---|
None | Nothing | Everything | None; requesters must enforce any order themselves |
Same ID (or class) | Transactions sharing an ID, on one channel | Different IDs | Same-ID traffic cannot use parallel paths or overtake |
Same address | Accesses to one location (or one cache line) | Different addresses | A check per access; usually free at the target |
Same source to same destination | Traffic from one requester to one target, along one path | Different targets, different sources | In-order queues along that path |
Global | Everything, as seen by every observer | Nothing | Serialises the system |
Requesters choose the scope they need by how they issue traffic: put transactions that must stay ordered in the same scope (one ID, one path), and give independent traffic different ones. Anything that needs ordering across scopes, such as a write and a later read on different IDs, must wait for a response or use a fence.
Almost every system keeps accesses to the same address from one requester in order: a read after a write to the same location returns the written value. Without that, even single-threaded code would break. It is also cheap, because the accesses meet at the same target, which can process them in arrival order. Lesson 7.8 shows where even this can fail: when the two accesses take different paths and meet only at the target.
Protocol | Scope |
|---|---|
AXI | Same ID: in order, per channel. Different IDs: no order. No order between the read and write channels, even for the same ID: a requester that needs a read to follow a write waits for the write response. |
PCI Express | Ordering rules between posted, non-posted and completion traffic along a path from one source to one destination (lesson 7.6); no ordering between different traffic classes; Relaxed Ordering and ID-Based Ordering narrow it further (lesson 7.7). |
DCLink | None between different tags (step 5); a FENCE orders everything before it against everything after (step 6). |
A requester (illustrative) wrote a value with one AXI ID and immediately read it back with another, expecting the new value. Most of the time the read returned it; under load, the read overtook the write and returned the old value. The write and read were on different channels and different IDs: no scope covered both. The fix was to wait for the write response before issuing the read.
Yes: transactions with the same ID on the same channel complete in order.
Global order makes every transaction wait for every earlier one across all paths and targets, which removes the concurrency that out-of-order completion provides.
"In order" is always within a scope: same ID, same address, same path, or global.
Protocols keep the narrowest scope correctness allows and leave the rest free for performance.
Anything that needs order across scopes must wait for a response or use a fence.