A network card streams packet data into memory at full rate. Somewhere on the path, one write stalls briefly because its memory bank is busy. Every write behind it waits, because posted writes may not pass posted writes, although none of those writes depend on each other. Only the final "packet ready" entry depends on all of them. The ordering rule built for that one write is slowing down the thousands before it.
How can a requester give up ordering it does not need, without breaking the ordering it does?
The obvious fix is to drop the ordering rule for bulk traffic. But the fabric cannot tell bulk writes from flag writes: they look the same. The requester knows. So the rules are relaxed per transaction, by the requester marking the transactions that need no ordering, while the ones that do (the flag, the completion entry) stay strictly ordered behind them.
Mechanism | What it relaxes | Safe when |
|---|---|---|
Relaxed ordering (per transaction) | A marked posted write may pass earlier posted writes; a marked completion may pass earlier posted writes | The marked transaction is not the "flag" of a producer/consumer pair: bulk data, not the write that announces it |
ID-based ordering | Ordering is kept only among transactions from the same source; traffic from different sources may pass each other | Different sources' traffic is independent, as it usually is |
Per-ID ordering in AXI | Different IDs are never ordered (lesson 7.2) | The requester chooses IDs so that dependent transactions share one |
The key point is asymmetry: the bulk writes may pass each other, but the flag is not marked relaxed, so it may not pass them. The flag still arrives after all the data; the data writes no longer wait for each other.
The requester, per transaction: only it knows which writes are data and which announce data.
System software must allow it, per device: relaxing order is only safe if the device's driver and the platform agree, so there is a control bit to enable it, and a platform whose memory system cannot handle relaxed traffic correctly keeps it disabled.
Nobody downstream may add it: a bridge or switch that marks traffic relaxed on its own initiative breaks producer/consumer patterns it knows nothing about. Components may ignore the bits (and keep strict order) but must not invent them.
Protocol | Relaxation |
|---|---|
PCI Express | A Relaxed Ordering attribute bit in the packet header, set by the requester and enabled per device by system software; ID-Based Ordering (from PCIe 3.0) keeps order only within one requester's traffic. |
AXI | No relaxation bits: ordering is per ID, so requesters relax order by using different IDs. |
Relaxing the flag (illustrative): a device marked every write relaxed, including its completion-queue entries. The entries occasionally arrived before the data they described. The flag must be strictly ordered.
Enabling it on a platform that mishandles it: some root ports have had problems with relaxed traffic, and Linux keeps a list of them on which it is not used. That is exactly why system software must opt in per device.
A bridge that invents relaxation to speed up a benchmark breaks every driver that relied on strict order.
The data writes. The completion entry must not be relaxed, so it cannot pass the data.
Ignoring it only keeps stricter order, which is always correct; setting it removes order the requester may depend on.
Relaxed ordering lets marked transactions pass earlier ones; ID-based ordering keeps order only within one source.
Only independent traffic may be relaxed; the "flag" stays strictly ordered behind it.
Requesters set the bits, system software enables them, and nothing downstream may add them.