A system's firewall and IOMMU are configured perfectly: every requester can reach only the memory it owns. One of them, a misbehaving accelerator, simply issues reads as fast as it can. Every one of them is allowed. Within milliseconds, the processor's memory latency has risen tenfold and a real-time audio stream starts to drop samples. No access rule was broken. The attack was volume.
How does a fabric stop one requester's volume from denying service to the others?
The obvious defence is access control: block requests the requester is not allowed to make. A flood consists entirely of allowed requests. What it exhausts is shared capacity: queue slots, buffer entries, credits, and the target's bandwidth. In a single shared first-come-first-served queue, whoever keeps the most requests in it decides everyone's waiting time.
A target serves one request per cycle. Three well-behaved requesters together use 30% of it. A greedy one keeps a fixed number of requests outstanding at all times. The model compares one shared FIFO queue with per-requester queues served round-robin.
"""Fabric denial of service: one greedy requester against three well-behaved ones.
A shared target serves one request per cycle. Three victims each issue a request with
probability 0.1 per cycle (30% of the target in total). The aggressor keeps LIMIT
requests outstanding at all times, re-issuing as soon as one completes.
shared FIFO: one queue in arrival order; the aggressor's requests fill it
round-robin: one queue per requester; the target serves non-empty queues in turn
Measured: the victims' mean and 99th-percentile latency (arrival to completion, cycles)
and the share of the target the aggressor gets.
"""
import random
from collections import deque
CYCLES, VICTIMS, P = 50000, 3, 0.1
def run(policy, limit, seed=1):
rng = random.Random(seed)
queues = [deque() for _ in range(VICTIMS + 1)] # index VICTIMS = aggressor
fifo = deque()
outstanding = 0
lat, aggr_done, rr = [], 0, 0
for t in range(CYCLES):
for v in range(VICTIMS):
if rng.random() < P:
(fifo.append((v, t)) if policy == "fifo" else queues[v].append(t))
while outstanding < limit: # aggressor tops up
(fifo.append((VICTIMS, t)) if policy == "fifo" else queues[VICTIMS].append(t))
outstanding += 1
# serve one request
if policy == "fifo":
who, t0 = fifo.popleft()
else:
for k in range(VICTIMS + 1):
q = (rr + k) % (VICTIMS + 1)
if queues[q]:
who, t0 = q, queues[q].popleft(); rr = q + 1; break
if who == VICTIMS:
outstanding -= 1; aggr_done += 1
else:
lat.append(t - t0 + 1)
lat.sort()
return sum(lat) / len(lat), lat[int(0.99 * len(lat))], aggr_done / CYCLES
limits = [1, 2, 4, 8, 16, 32]
fifo = [run("fifo", l) for l in limits]
rr = [run("rr", l) for l in limits]
print("aggressor outstanding: " + " ".join(f"{l:6d}" for l in limits))
print("FIFO victim mean (cyc): " + " ".join(f"{m:6.1f}" for m, _, _ in fifo))
print("FIFO victim p99 (cyc): " + " ".join(f"{p:6d}" for _, p, _ in fifo))
print("RR victim mean (cyc): " + " ".join(f"{m:6.1f}" for m, _, _ in rr))
print("RR victim p99 (cyc): " + " ".join(f"{p:6d}" for _, p, _ in rr))
print("aggressor share, FIFO: " + " ".join(f"{s:6.2f}" for _, _, s in fifo))
print("aggressor share, RR: " + " ".join(f"{s:6.2f}" for _, _, s in rr))
assert fifo[-1][0] > 10 * rr[-1][0] and max(p for _, p, _ in rr) <= 2 * (VICTIMS + 1)
print("PASS: in one shared queue the greedy requester sets everyone's latency; per-requester queues with round-robin keep the victims' latency bounded however greedy it is")It prints:
aggressor outstanding: 1 2 4 8 16 32
FIFO victim mean (cyc): 1.6 3.0 5.9 11.5 22.9 45.7
FIFO victim p99 (cyc): 4 6 10 17 30 55
RR victim mean (cyc): 1.5 1.5 1.5 1.5 1.5 1.5
RR victim p99 (cyc): 5 5 5 5 5 5
aggressor share, FIFO: 0.70 0.70 0.70 0.70 0.70 0.70
aggressor share, RR: 0.70 0.70 0.70 0.70 0.70 0.70
PASS: in one shared queue the greedy requester sets everyone's latency; per-requester queues with round-robin keep the victims' latency bounded however greedy it isTwo points stand out. In the shared queue, the victims' latency is set almost entirely by how many requests the greedy agent keeps queued: 32 outstanding requests mean about 46 cycles of waiting. And fairness costs nothing in throughput: under round-robin the greedy requester still gets the 70% the others leave unused. It simply cannot take their share or make them wait behind it.
Mechanism | What it limits |
|---|---|
Per-requester queues and fair arbitration | Waiting time behind other requesters (lessons 8.3 and 4.14) |
Outstanding-request limits per requester | How many shared slots one agent can hold |
Credits or buffers partitioned per requester or class | Buffer exhaustion by one agent (lesson 4.6) |
Rate limiting or bandwidth regulation | Sustained bandwidth per requester (lesson 4.10) |
QoS priorities with guarantees | Latency for real-time traffic, with limits so priority cannot itself become a weapon (lesson 8.7) |
Timeouts and isolation of a faulty agent | A requester that hangs or floods can be cut off (lesson 9.9) |
System | Defence against volume |
|---|---|
AXI interconnects | QoS signals (AxQOS) and arbitration, with regulators limiting outstanding transactions and bandwidth per port |
Arm MPAM, Intel RDT | Memory bandwidth and cache capacity partitioned between software groups (fact A28) |
PCI Express | Per-virtual-channel credits and arbitration in switches |
The opening system (illustrative) had a single shared request queue in front of its memory controller. Giving each major requester its own queue, a limit on outstanding reads, and a bandwidth regulator for the accelerator brought the processor's memory latency back to normal under the same flood, at a cost of under 1% in the accelerator's throughput when the system was otherwise idle.
Each requester has its own queue and gets a turn every few cycles, so a victim never waits behind the greedy requester's backlog.
A high-priority requester that floods can starve every lower-priority one; priority needs limits or guarantees.
Denial of service needs only volume: allowed requests that exhaust shared capacity.
In one shared queue, the greediest requester sets everyone's latency.
Bound each requester with separate queues, fair arbitration, limits and regulators; fairness costs no throughput.