A storage device finishes reading a block, writes it into memory by DMA, and signals completion. With one kind of interrupt, the driver always finds the block complete. With another, it occasionally finds part of the old contents, and the bug disappears whenever someone adds a debug print to the handler. The device is the same; only the interrupt mechanism differs.
Why does the kind of interrupt decide whether the data is there when the handler runs?
The obvious assumption is that a device signals completion only after it has finished writing, so the data must be in memory. But "finished writing" from the device's side means the writes have left the device, not that they have arrived. They may still be queued in a switch or a write buffer. Whether the interrupt can overtake them depends entirely on whether it travels on the same ordered path.
DMA data followed by an interrupt is data-then-flag (lesson 7.1), with the interrupt as the flag. It is safe only if the interrupt cannot become visible before the data.
Mechanism | Path | Ordered behind the data? |
|---|---|---|
Message-signalled interrupt | A posted memory write on the same path as the DMA writes | Yes: a posted write may not pass earlier posted writes from the same device (lesson 7.6) |
Wired interrupt | A separate wire to the interrupt controller | No: it is not on the data path at all |
Emulated wired interrupt (message) | A message on the link, but not required to follow the ordering of memory writes | Not guaranteed |
Read a device register first. In the handler, read a status register from the device before touching the data. The read is non-posted and may not pass the device's earlier posted writes on the same path, so by the time its completion returns, the DMA data has landed (lesson 5.7). It costs a round trip per interrupt.
Check a sequence number in the data. The device writes an incrementing number or valid bit as the last part of each completion entry; the driver checks it and retries if it is stale.
The opening's "disappearing" bug fits exactly: a debug print delayed the handler long enough for the buffered writes to land.
System | Interrupt ordering |
|---|---|
PCI Express | An MSI or MSI-X is a posted memory write and may not pass posted writes sent earlier by the same function, so DMA data reaches memory first. Legacy INTx messages carry no such ordering; drivers using them read a device register to flush earlier writes. |
On-chip devices with wired interrupts | The interrupt wire and the data path are separate; drivers read back a device register, or the interconnect provides a barrier, before using DMA data. |
The opening device (illustrative) used a legacy wired interrupt behind a PCIe switch. Under load, DMA writes queued in the switch while the interrupt went straight through. The fix was to switch the device to MSI-X, which also gave each queue its own vector; on systems that could not use MSI-X, the driver read the device's status register at the start of the handler.
The read may not pass the device's earlier posted writes on that path; its completion cannot return until those DMA writes have landed.
The MSI may pass the DMA writes before it, so the handler can run before the data has landed.
DMA data followed by an interrupt is data-then-flag, with the interrupt as the flag.
An MSI is a posted write in the same ordered stream, so it lands after the data; a wired interrupt can overtake it.
With wired interrupts, read a device register (or check a sequence number) before using the data.