A link carries two kinds of traffic: bulk DMA writes and short, urgent interrupt messages. Both use one pool of receiver credits. One day the DMA's destination memory gets busy, the DMA writes back up into the link's buffer, and an interrupt message waits more than a hundred cycles for a credit. Nothing is broken, and the interrupt is still late.

When several kinds of traffic share a link, how should credits be divided between them?

The obvious answer, and why it breaks

The obvious answer is one shared pool: any traffic may use any free slot, which uses the buffer most efficiently. It breaks when one class's consumer slows down. That class's words sit in the buffer holding credits, its sender keeps spending every credit that frees up, and soon the pool is full of words that cannot move. Other classes, whose consumers are perfectly fast, cannot get a single credit. The slow class has blocked everyone through the shared resource.

The idea: separate what must not block each other

Give each class of traffic its own credits for its own buffer space, so one class can only ever fill its own share. The choice of how to divide is a spectrum:

A second kind of split is by what is stored. Packets have a fixed-size header and a variable-size payload. Counting both in one unit wastes buffer: a header-only packet would consume a full packet's worth of space. Separate header credits and data credits let a receiver accept many small packets or a few large ones within the same memory.

Build it: starve, split, reserve

The model shares an 8-slot buffer between bulk class A, whose consumer is slow and blocked half the time, and urgent class B, which has one word every 10 cycles and a fast consumer. It measures the worst wait for a B word under each policy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

"""Two traffic classes sharing a receiver buffer of 8 slots.

Class A is bulk traffic that always has data, and its consumer is often blocked: it
drains one word every 4 cycles, but only during the first 100 of every 200 cycles. Class B is urgent and occasional (one word every 10 cycles), and
its consumer is fast. Credits are managed three ways:
  shared:   one pool of 8 credits for both classes
  split:    4 credits reserved for each class
  reserved: a shared pool of 6 plus 2 credits only class B may use
Measure the worst wait for a class B word to be sent.
"""


def run(policy, cycles=4000):
    buf = {"A": 0, "B": 0}
    b_waiting_since = None
    worst_b_wait, b_sent = 0, 0
    for t in range(cycles):
        if t % 10 == 0 and b_waiting_since is None:
            b_waiting_since = t                     # a B word becomes ready to send
        used = buf["A"] + buf["B"]
        def can_send(c):
            if policy == "shared":
                return used < 8
            if policy == "split":
                return buf[c] < 4
            if c == "A":                            # reserved: A may use only the shared 6
                return buf["A"] < 6 and used < 8
            return used < 8                         # B may use any free slot, including its 2
        if b_waiting_since is not None and can_send("B"):
            buf["B"] += 1; b_sent += 1
            worst_b_wait = max(worst_b_wait, t - b_waiting_since)
            b_waiting_since = None
        elif can_send("A"):
            buf["A"] += 1
        if t % 200 < 100 and t % 4 == 0 and buf["A"]:
            buf["A"] -= 1                           # A's consumer: slow, and blocked half the time
        if buf["B"]:
            buf["B"] -= 1                           # fast consumer of B
    return worst_b_wait, b_sent


print("policy    worst wait for a class B word (cycles)")
waits = {}
for policy in ("shared", "split", "reserved"):
    w, n = run(policy)
    waits[policy] = w
    print(f"{policy:8}  {w:5d}   ({n} B words sent)")
assert waits["shared"] > waits["reserved"] and waits["reserved"] <= 1
print("PASS: in a shared pool bulk traffic holds every credit and urgent traffic waits; a reservation fixes it")

It prints:

policy    worst wait for a class B word (cycles)
shared      101   (200 B words sent)
split         0   (400 B words sent)
reserved      0   (400 B words sent)
PASS: in a shared pool bulk traffic holds every credit and urgent traffic waits; a reservation fixes it

With a shared pool, class A fills every slot during its consumer's blocked periods, and class B waits over 100 cycles, sending only half its words in the run. Splitting the pool removes the wait entirely, at the cost of four slots that class B rarely uses. Reserving just two slots for B gives the same zero wait while class A keeps six slots: the best of both.

The design space

How real protocols chose

PCIe separates its three traffic types for more than urgency: its ordering and deadlock-avoidance rules depend on some types being able to pass others (lessons 7.6 and 9.4), which is impossible if they share one pool.

What breaks

The opening story, with the debug steps (illustrative):

• Symptom: interrupt latency spikes only when a particular DMA is busy.

• Evidence: at the spike, the receiver's shared buffer is full of DMA writes and the interrupt message's credit request is waiting.

• Root cause: one shared pool; the DMA's slow destination held every credit.

• Fix: a small reservation for the interrupt class (or a separate channel). The lasting lesson: classes that must not wait for each other must not share a pool without a guaranteed minimum.

Key takeaways

• A shared credit pool lets one class with a slow consumer take every slot and starve the others.

• Separate pools isolate classes completely; a shared pool with a reserved minimum per class isolates them almost as well with less waste.

• Separate header and data credits fit fixed headers and variable payloads into one buffer efficiently.

Related topics

• Part 4 overview: Flow Control

• Architecture course: Credit Flow Control and Buffers