A driver wants a network card to send 1,000 packets. It could write each packet's address and length into the card's registers, one register write per field, waiting for the card to finish each before giving it the next. Every register write crosses the bus, and the card idles while the driver prepares the next packet. The card can send far faster than the driver can feed it one packet at a time.
How does software hand a device a stream of work without feeding it item by item?
The obvious interface is registers: write the job's parameters into the device and tell it to go. For an occasional job that is fine. For a stream of jobs it couples the two sides tightly: the device waits for software to write each job, software waits for the device to finish before writing the next, and each job costs several register writes across the bus. The two sides need a place to leave work for each other, so each can run at its own pace.
A descriptor describes one job: a buffer address, a length, flags. Software writes descriptors into a ring, a circular array in ordinary memory.
Software (the producer) adds descriptors at the tail; the device (the consumer) takes them from the head.
After adding a batch, software writes the new tail index to the device's doorbell register: one write that says "there is work up to here".
The device fetches descriptors itself, by DMA, at its own pace, and reports its progress (a new head, or completion entries in a second ring) back in memory, with an interrupt if needed.
With indices that wrap at the ring size, head = tail could mean empty or full. Two standard fixes: keep one slot always empty (full when tail + 1 = head), or let the indices count up freely and take them modulo the ring size only to pick a slot. Then empty is tail = head and full is tail − head = size. It is the same problem, and the same fix, as a FIFO's wrap bit.
The model runs an 8-slot ring. In each step the driver has some descriptors ready and posts as many as fit, writing one doorbell per batch; the device then processes up to its capacity. The check: every descriptor processed once, in order, and the ring never overflows.
"""A descriptor ring between a driver (producer) and a device (consumer).
The ring has 8 slots. The driver writes descriptors at its tail and then writes
the tail to a doorbell; the device reads descriptors from its head up to the last
doorbell value and reports its new head back. Indices count up without limit and
are taken modulo 8 for the slot, so full (tail - head == 8) and empty
(tail == head) are never confused.
"""
SLOTS = 8
ring = [None] * SLOTS
tail = head = doorbell = 0 # driver tail, device head, last doorbell
posted, done, doorbells, backlog, waited = [], [], 0, 0, 0
# Each step: (descriptors the driver wants to post, descriptors the device can process)
script = [(3, 0), (4, 2), (5, 1), (2, 6), (0, 4), (6, 3), (0, 9)] # 20 descriptors in all
next_id = 1
for want, work in script:
backlog += want # descriptors the driver has ready
while backlog and tail - head < SLOTS:
backlog -= 1
ring[tail % SLOTS] = next_id
posted.append(next_id)
next_id += 1
tail += 1
if tail != doorbell: # one doorbell write per batch, not per descriptor
doorbell = tail
doorbells += 1
for _ in range(work):
if head == doorbell: # nothing the device has been told about
break
done.append(ring[head % SLOTS])
head += 1
waited += 1 if backlog else 0 # steps where the ring was too full for everything ready
print(f"ready {want}, process {work}: tail {tail} head {head}, {tail - head} in the ring, {backlog} waiting")
assert done == posted and head == tail and backlog == 0 and len(posted) == 20
print(f"{len(posted)} descriptors posted with {doorbells} doorbell writes; in {waited} of {len(script)} steps the ring was full and the driver had to wait")
print("PASS: the device processed every descriptor once, in order, and the ring never overflowed")It prints:
ready 3, process 0: tail 3 head 0, 3 in the ring, 0 waiting
ready 4, process 2: tail 7 head 2, 5 in the ring, 0 waiting
ready 5, process 1: tail 10 head 3, 7 in the ring, 2 waiting
ready 2, process 6: tail 11 head 9, 2 in the ring, 3 waiting
ready 0, process 4: tail 14 head 13, 1 in the ring, 0 waiting
ready 6, process 3: tail 20 head 16, 4 in the ring, 0 waiting
ready 0, process 9: tail 20 head 20, 0 in the ring, 0 waiting
20 descriptors posted with 6 doorbell writes; in 2 of 7 steps the ring was full and the driver had to wait
PASS: the device processed every descriptor once, in order, and the ring never overflowedTwenty descriptors needed only six doorbell writes, one per batch, not one per descriptor. In steps 3 and 4 the ring filled (8 in flight), so the driver held descriptors back until the device made room: the ring is a buffer with flow control, and "full" is its back-pressure.
Property | Why it matters |
|---|---|
Work lives in memory, not in device registers | Descriptors cost ordinary memory writes; only the doorbell crosses to the device |
One doorbell per batch | Register writes to a device are slow; batching amortises them |
The device fetches at its own pace | Neither side waits for the other until the ring is full or empty |
Completions in a ring too | Notification says "look at the ring", so interrupts can be coalesced (lesson 5.15) |
Ordering | Descriptors must be visible in memory before the doorbell: the doorbell must not overtake the descriptor writes (Part 7) |
System | Rings |
|---|---|
NVMe storage | Submission and completion queues in host memory. The host writes commands to a submission queue and then the new tail to that queue's doorbell; it consumes completions and writes the new head to the completion queue's doorbell. |
Network cards | Transmit and receive descriptor rings per queue, with doorbells and per-queue interrupts. |
A driver (illustrative) wrote descriptors and then the doorbell, on a system where the descriptor writes went through a write buffer and the doorbell, a device register write, took a faster path. The device fetched a descriptor slot before the descriptor had reached memory and sent a stale buffer. The fix was a write barrier between the descriptor writes and the doorbell, which forces the descriptors to be visible first (lesson 7.5).
tail − head = 16 = the ring size: full.
The doorbell tells the device to fetch those slots; if it overtakes the descriptor writes, the device reads stale descriptors.
Software hands work to devices through descriptor rings in memory, with head and tail indices and a doorbell per batch.
The ring decouples the two sides: each runs at its own pace until the ring is full or empty.
Distinguish full from empty with free-running indices (or a spare slot), and order descriptors before the doorbell.