A DMA engine writes a block of data into memory, then writes 1 to a "ready" flag. Software polls the flag; when it reads 1, it reads the data. Most of the time the data is correct. Occasionally software reads the old contents of the buffer, even though it saw the flag set. Each write was performed correctly. They were performed in the wrong order, as seen by the reader.

Why does the order in which writes become visible matter, and what makes a fabric get it wrong?

The obvious answer, and why it breaks

The obvious assumption is that writes land in the order they were issued: data first, flag second. On a single bus with one path to memory, that is true, because the flag queues behind the data. But high-performance fabrics have several paths, buffers that drain at different speeds, and targets that answer out of order (Part 6). The flag can take a short path while the data takes a long one, and arrive first. The consumer sees a promise ("ready") before the thing promised.

The idea: producer, consumer, and an order both must see

Almost every hand-off between agents is this pattern: write the payload, then write something that says the payload is there (a flag, an index, a doorbell, an interrupt). It works only if every observer that sees the second write also sees the first. That is an ordering requirement between two writes to different addresses, and nothing in the addresses themselves enforces it.

Build it: how often does the flag win?

The model sends a data write and then a flag write with random delays, 100,000 times, in three fabrics: one path (the flag queues behind the data), two independent paths, and two paths with a fence: the producer waits until the data write is acknowledged as having reached memory before it sends the flag.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

"""Data then flag through a fabric that may reorder.

A producer writes a buffer (data) and then a flag. A consumer polls the flag and,
when it sees it set, reads the data. Writes reach memory after a random delay:
  in order:        one path; the flag cannot arrive before the data
  two paths:       data and flag take different paths with independent delays
  two paths+fence: the producer waits until the data write is acknowledged as
                   having reached memory before it sends the flag
The model counts how often the consumer reads stale data.
"""
import random


def trial(mode, rng):
    t_data = rng.randint(5, 40)                     # data write lands at this time
    if mode == "in order":
        t_flag = max(t_data, rng.randint(5, 40)) + 1  # same path: queues behind the data
    elif mode == "two paths":
        t_flag = 1 + rng.randint(5, 40)             # sent one cycle later, own path
    else:                                           # fence: wait for the data's acknowledgement
        t_flag = t_data + 1 + rng.randint(5, 40)
    t_seen = t_flag                                 # consumer polls continuously
    return t_seen < t_data                          # flag seen before data landed: stale


rng = random.Random(42)
TRIALS = 100000
results = {}
for mode in ("in order", "two paths", "two paths + fence"):
    stale = sum(trial(mode, rng) for _ in range(TRIALS))
    results[mode] = stale
    print(f"{mode:18s} stale reads: {stale:6d} of {TRIALS} ({stale / TRIALS * 100:.1f}%)")
assert results["in order"] == 0 and results["two paths + fence"] == 0 and results["two paths"] > 0
print("PASS: the flag overtakes the data only when the paths can reorder and nothing forces order")

It prints:

in order           stale reads:      0 of 100000 (0.0%)
two paths          stale reads:  45884 of 100000 (45.9%)
two paths + fence  stale reads:      0 of 100000 (0.0%)
PASS: the flag overtakes the data only when the paths can reorder and nothing forces order

With one path, never. With two paths and nothing forcing order, nearly half the time: the bug in the opening is not rare at all on such a fabric, only hidden by timing in most tests. With a fence, never again, at the price of waiting for the data write's acknowledgement before sending the flag.

The ways to restore order

What breaks

The opening DMA engine (illustrative) issued its data writes on one AXI ID, for throughput, and its flag write on another, out of habit. Different IDs carry no ordering (lesson 7.2), and the interconnect sent the flag down a less busy path. The fix was to issue the flag write only after the last data write's response arrived. Putting the flag on the same ID would also have worked, if every component on the path kept same-ID order end to end.

Key takeaways

• Producer/consumer hand-offs need every observer that sees the flag to also see the data.

• Fabrics with several paths, buffers or out-of-order targets can let the flag overtake the data.

• Restore order with one ordered path, a shared ID, a final acknowledgement, a fence, or protocol rules.

Related topics

• Part 7 overview: Ordering

• Architecture course: Memory Ordering Models