A network card receives packets at full rate and raises an interrupt for each one. The processor spends all its time entering and leaving the interrupt handler, and packets start being dropped, not because the processor is slow at handling packets, but because it is too busy being interrupted to handle any. Interrupting more did not deliver more.
How should a device tell software that work is ready, and how does the answer change with rate?
The obvious answer is an interrupt per event: the device signals, software runs a handler, done. At low event rates it is ideal: no wasted work while idle, prompt response. But each interrupt has a fixed cost (saving state, running the handler, returning), and at high event rates that fixed cost, multiplied by the rate, can exceed the time available. Notification must adapt to the rate.
Mechanism | How it works | Strength | Weakness |
|---|---|---|---|
Level-triggered wire | The wire stays asserted while the device needs service; software clears the cause | Never lost: still asserted if missed; easy to share | Shared lines need every handler polled; races data (lesson 5.14) |
Edge-triggered wire | A transition signals an event | No need to clear a level | An edge missed while masked is lost |
Message-signalled | The device writes a value to a programmed address; each value can mean a different event | Many distinct vectors, no wires, ordered behind data | Needs a data path and an interrupt controller that decodes writes |
Polling | Software checks a status (or a ring in memory) regularly | No interrupt overhead; best at high rates | Wastes processor time when idle; adds latency up to the poll interval |
Coalescing | The device interrupts once per N events or per T microseconds | Bounds interrupt rate | Adds up to T of latency to the first event |
Minimum-size Ethernet frames, 64 bytes, each with 20 bytes of fixed overhead on the wire (preamble, start delimiter and inter-frame gap).
Bytes per frame on the wire: 64 + 20 = 84 bytes = 672 bits
Frames per second: 100e9 / 672 = 148.8 million
Time per frame: 1 / 148.8e6 = 6.7 ns
An interrupt costing 1 us (illustrative) per frame needs 148.8 seconds of processor time per second: impossible.
Coalescing 64 frames per interrupt: 2.3 million interrupts/s = 2.3 s of processor time per second at 1 us each: still too many for one core
Polling the receive ring while busy: no interrupts at all; interrupt again only when the ring goes idleThat is why fast devices combine mechanisms: interrupt when idle (prompt response, no wasted polling), switch to polling while busy (no per-event cost), and coalesce in between. Each event is written into a ring in memory (lesson 5.16), so notification only says "look at the ring", not "here is one event".
System | Choice |
|---|---|
PCI Express | Message-signalled interrupts: MSI with up to 32 vectors per function, MSI-X with up to 2048; each is a posted memory write. Legacy INTx is emulated with messages. |
Linux network drivers | NAPI: take an interrupt, then disable it and poll the device's ring until it is empty, then re-enable the interrupt. |
On-chip peripherals on AXI or APB | Wired interrupts, usually level-triggered, into an interrupt controller. |
Interrupt livelock: the opening network card. The processor is saturated by interrupt overhead and does no useful work.
Lost edge: an edge-triggered interrupt that fires while masked (illustrative) is never seen; the device waits for service that never comes. Level-triggered interrupts or a status register that records events avoid it.
Stale data: a wired interrupt that overtakes DMA writes (lesson 5.14). A message-signalled interrupt behind the data avoids it.
2e6 × 2 µs = 4 seconds per second: four cores' worth, so it cannot work. Coalescing 32 events per interrupt brings it to 12.5% of one core.
The level stays asserted until software clears the cause, so if it is missed while masked, it is still there when unmasked; an edge is gone once it has passed.
Level, edge and message-signalled interrupts trade reliability, sharing and ordering; MSI is a posted write that follows the data.
Every interrupt has a fixed cost; at high event rates coalesce or poll.
Fast devices put events in a memory ring and switch between interrupts when idle and polling when busy.