A device finishes a DMA transfer into memory and raises its interrupt wire. The interrupt reaches the processor in a few nanoseconds, by a short direct path. The last DMA writes are still queued in a buffer on the way to memory. The driver reads the buffer and finds old data. The wire was faster than the data it was announcing.
Should a signal travel on its own wire or as a message on the main path, and what does each choice cost?
The obvious way to send a signal (an interrupt, a reset request, a power-state change, an error flag) is a wire: one bit, instant, simple. Inside one small block that is right. Across a chip or between chips, wires cost pins and routing, do not cross links that carry packets, and, as the opening shows, travel by a different path from the data, so they arrive in an order nobody chose.
Sideband (a dedicated wire) | In-band (a message on the data path) | |
|---|---|---|
Cost | A wire or pin per signal, per destination | No extra wires: shares the existing link |
Reach | As far as you can run a wire; does not cross a serial link or a switch | Anywhere the data path reaches, through bridges and switches |
Latency | Very low and fixed | Like any transaction: queued behind other traffic |
Ordering with data | None: may overtake the data it refers to | Can be ordered behind earlier data on the same path |
Scaling | One wire per signal; many signals means many wires | A new signal is a new message code |
Works when the data path is down | Yes | No: reset and link-training signals often stay sideband for this reason |
The ordering row is the subtle one. A message that travels behind the data on the same path cannot arrive before the data, if the path keeps order. That turns the opening race into a non-issue (lesson 7.9).
In-band: interrupts (as message writes), error reports, power-management requests, vendor messages: anything that should be ordered with data or must cross links and switches.
Sideband: reset, clocks, wake signals and link-training controls, which must work before or without the data path; and signals inside one block where a wire is free.
Both: some systems keep a wire for urgency and send a message for detail.
Protocol | Choice |
|---|---|
PCI Express | Interrupts are in-band: message-signalled interrupts are posted memory writes, and legacy wired interrupts (INTx) are emulated with messages. Error reporting and power management also use messages. Reset and some wake signals remain sideband. |
AXI and APB | No messages; interrupts are separate wires from each peripheral to an interrupt controller. |
The opening race is the classic one. A device's interrupt wire outran its DMA writes, which were still in a write buffer. Drivers worked around it by reading a device register before touching the data, which forces earlier writes on that path to complete first (lesson 5.7). Message-signalled interrupts remove the need: the interrupt is a write on the same path, behind the data, so it cannot overtake it.
Reset may be needed precisely when the data path is broken or not yet trained, so it needs a path that does not depend on it.
Ordering: the interrupt message travels behind the earlier writes on the same path, and the path does not let it pass them.
Sideband wires are fast and simple but cost pins, do not cross links, and race the data they announce.
In-band messages share the data path, cross switches, and can be ordered behind the data.
Signals that must work without the data path, such as reset, stay sideband.