Core concepts

Why outstanding exists

Memory and IO latency is hundreds of cycles. If a master waited for each transaction to complete before issuing the next, bandwidth would be data_width / latency — abysmal for DRAM. Outstanding transactions let the master pipeline requests: issue N reads, then collect responses as they arrive.

The hardware cost: reorder buffers, ID trackers, and response matching logic. The verification cost: scoreboards must handle any legal completion order. The software cost: programmers must know which orders are guaranteed.

LATENCY HIDING WITH OUTSTANDING

without OSTD:  |--RD--|........wait........|--RD--|
with OSTD=4:   |--RD--|--RD--|--RD--|--RD--|
               .......|data0|data2|data1|data3|....  (may reorder)

ID-based ordering rules (AXI model)

For ordinary AXI4 traffic, same-ID read responses preserve read-request order, and same-ID write responses preserve write-request order. Different IDs permit additional reordering; R and B are separate streams even when their numeric IDs match. Required read-after-write visibility depends on the system's completion and memory-access contract. Base AXI4 has no universal barrier transaction that automatically orders every read and write.

This is not universal. AXI-Lite has no IDs and implies in-order. APB is strictly sequential. PCIe has relaxed ordering bits per TLP. Know your protocol's rules before writing a scoreboard.

• AXI: same ARID ordered on R. Same AWID ordered on B. AR vs AW independent.

• ACE: adds barrier transactions that drain outstanding before proceeding.

• PCIe: Relaxed Ordering, ID-based Ordering, No Snoop attributes per TLP.

Barriers, fences, and happens-before

A driver publishing work must satisfy both its language/OS synchronisation rules and the platform's device-access contract. Compiler ordering, CPU memory barriers, MMIO accessor semantics and hardware transaction ordering are different layers. Do not pick a particular instruction merely from the symptom 'DMA used stale data'.

For a descriptor followed by a doorbell, establish which operation publishes the descriptor, what ownership the device observes and whether the buffer is coherent or streaming-mapped. Use the platform's DMA synchronisation and MMIO interfaces. Hardware coherence does not by itself order the notification, and a CPU barrier is not a universal cache flush or proof of device completion.

ORDERING QUESTIONS (ordinary AXI4 / platform software)

Same observed port + ARID? -> preserve read-response order
Same observed port + AWID? -> preserve write-response order
ARID equals AWID?          -> not a shared R/B ordering stream
Different agents?         -> inspect ownership + memory model
Descriptor then doorbell? -> DMA publication + MMIO contract

Do not infer system-wide visibility from equal numeric IDs.

Bridge ordering traps

Bridges are ordering contract rewrites. An ID remapper that collapses 16 upstream IDs into 4 downstream IDs can cause responses to arrive in an order illegal for upstream masters. A width splitter that issues sub-bursts must preserve write response order per original AWID.

Merge bridges (multiple masters → one slave port) interleave transactions. Unless the merge preserves per-port ordering domains, software on one core can observe reordering caused by another core's traffic.

• Document: ID map, ordering guarantees, response matching rules.

• Width converter: W beat splitting must not reorder B responses.

• Merge arbiter: per-port ordering domain or global strict order?

Common misconceptions

AXI guarantees that writes complete before reads to the same address.

AXI read and write channels are independent even when ARID equals AWID; same-ID ordering applies separately within R and B streams. Required read-after-write visibility needs the system's completion/visibility and software synchronization contract (or applicable ACE barrier), not equal IDs alone.

More outstanding always improves performance.

Beyond the knee, more outstanding increases reorder buffer area, verification complexity, and head-of-line blocking risk. Diminishing returns after DRAM page policy saturates.

Single-ID tests are sufficient for DMA validation.

Validate the supported matrix: a one-ID master can have multiple ordered outstanding contexts and needs same-ID queue/beat/credit tests. A multi-ID design additionally needs cross-ID reordering stress; one-ID tests cannot cover that behavior.

Key takeaways

• AXI: same ARID ordered on R. Same AWID ordered on B. AR vs AW independent.

• ACE: adds barrier transactions that drain outstanding before proceeding.

• PCIe: Relaxed Ordering, ID-based Ordering, No Snoop attributes per TLP.