After a second DMA engine is added to a subsystem, firmware reports that register reads "sometimes never come back". The bus monitor shows the bus busy every cycle. No watchdog fires, because transfers keep happening. Separately, a test of two agents sharing a mailbox times out after 20,000 cycles, with the mailbox's request lines toggling the whole time.
Neither is a hang. What are they, and how do you catch them early?
Both get treated as performance problems: raise the timeout, add a bigger buffer, run longer. Or they get treated as hangs, and the wait-for table looks normal, because every agent is waiting for something that does happen, just never for it. The watchdog stays quiet because it watches activity, and there is plenty. What is missing is progress for a particular agent, and nothing in the bench measures that.
Starvation, in counters: total throughput is fine; one agent's completion count stays at zero (or far below its share) while its oldest request keeps getting older. In a waveform, its request is high for hundreds of cycles while grants go to others.
Livelock, in counters: attempts, retries or NACKs climb steadily; completions are flat for every agent. In a waveform, a short repeating pattern: request, collision, back-off, request, with the same period for ever.
In a regression: the activity watchdog never fires; the global timeout does, or the test passes with a starved agent nobody checked.
Count completions per agent, and attempts per agent, over a window. Zero completions for one agent with others normal is starvation; attempts rising with zero completions for all is livelock.
For starvation, read the arbiter's policy and the other agents' load. Fixed priority starves the low end whenever the high end can fill the bus. Ask: can the higher-priority agents keep their requests up every cycle? A new master or a faster one often makes the answer yes.
For livelock, find the symmetry. Two agents that collide, back off by the same amount and retry together will collide again. Look for identical back-off constants, identical reset states and a resolver that rejects everybody instead of choosing one.
Two DMA masters each request three cycles in four; between them they keep a three-way arbiter busy every cycle. The CPU, master 2, asks every 50 cycles and holds its request until it is served. The bench checks the age of every waiting request. FIX=1 (the default) is round-robin; FIX=0 is the fixed priority the design shipped with.
// arb3: grants one of three requesters per cycle. FIX=0 is fixed priority (0 beats 1 beats 2);
// FIX=1 is round-robin, starting the search after the last winner.
module arb3 #(parameter bit FIX = 1) (
input logic clk, rst_n,
input logic [2:0] req,
output logic [2:0] grant
);
logic [1:0] last;
always_comb begin
grant = 3'b000;
if (!FIX) begin
grant = req & (~req + 3'd1); // lowest index wins, every time
end else begin
for (int k = 1; k <= 3; k++) begin
int i;
i = (int'(last) + k) % 3;
if (req[i] && grant == 3'b000) grant[i] = 1'b1;
end
end
end
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) last <= 2'd2;
else if (grant[0]) last <= 2'd0;
else if (grant[1]) last <= 2'd1;
else if (grant[2]) last <= 2'd2;
endmodule
$ iverilog -g2012 -o s.vvp starve_tb.sv && vvp -n s.vvp
PASS: 500 cycles, grants m0=245 m1=245 m2=10 (FIX=1)
...
$ iverilog -g2012 -Pstarve_tb.FIX=0 -o sb.vvp starve_tb.sv
$ vvp -n sb.vvp
FAIL cycle=43: STARVE: master 2 has waited 33 cycles; grants m0=33 m1=10 m2=0
...
$ vvp -n sb.vvp +AGE=100000
PASS: 500 cycles, grants m0=375 m1=125 m2=0 (FIX=0)
...Round-robin: the CPU asks 10 times and is served 10 times; the two DMAs share the rest evenly.
Fixed priority with the age check: the CPU's first request, made at cycle 10, is still waiting at cycle 43. The message carries the evidence: master 0 has had 33 grants, master 2 none.
Fixed priority without it: the test passes. The bus did 500 transfers in 500 cycles, a perfect throughput number, and the CPU got none of them. Master 1 is short-changed too: it asks three cycles in four but wins only the one cycle in four that master 0 leaves free (125 grants).
Two agents share a slot that serves one request per cycle. If both ask in the same cycle, both are refused and each backs off before asking again. With the same back-off for both, they ask together for ever. The bench has three guards: an activity watchdog, a progress check (no ack for 64 cycles while requests continue) and a global timeout.
// retry_agent: needs JOBS turns at a shared slot. It asks for one cycle; on a nack it waits
// and asks again. FIX=0 is the bug: a fixed back-off, the same for every agent.
// FIX=1 draws the back-off (1 to 8 cycles) from a small LFSR seeded per agent.
module retry_agent #(parameter bit FIX = 1, parameter logic [7:0] ID = 8'd1, parameter int JOBS = 20) (
input logic clk, rst_n,
output logic req,
input logic ack, nack,
output logic finished
);
logic [7:0] lfsr;
logic [3:0] wait_q;
int unsigned done_q;
assign req = rst_n && !finished && (wait_q == 4'd0);
assign finished = (done_q == JOBS);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) begin lfsr <= ID * 8'd37 + 8'd1; wait_q <= 4'd0; done_q <= 0; end
else begin
lfsr <= {lfsr[6:0], lfsr[7] ^ lfsr[5] ^ lfsr[4] ^ lfsr[3]};
if (ack) begin done_q <= done_q + 1; wait_q <= 4'd2; end // short pause between jobs
else if (nack) wait_q <= FIX ? 4'(1 + lfsr[2:0]) : 4'd3; // back off, then retry
else if (wait_q != 4'd0) wait_q <= wait_q - 4'd1;
end
endmodule
// slot: one turn per cycle. A lone request is acked; two at once collide and both are nacked.
module slot (input logic [1:0] req, output logic [1:0] ack, nack);
assign ack = (req == 2'b01 || req == 2'b10) ? req : 2'b00;
assign nack = (req == 2'b11) ? 2'b11 : 2'b00;
endmodule
$ iverilog -g2012 -o l.vvp livelock_tb.sv && vvp -n l.vvp
PASS: 40 jobs in 79 cycles, 46 tries (FIX=1)
...
$ iverilog -g2012 -Plivelock_tb.FIX=0 -o lb.vvp livelock_tb.sv
$ vvp -n lb.vvp
FAIL cycle=65: LIVELOCK: requests every few cycles, no ack for 65 cycles; 34 tries, 0 acks
...
$ vvp -n lb.vvp +NOPROG
FAIL cycle=20000: TIMEOUT: 10000 tries, 0 acks
...Random back-off: 40 jobs in 79 cycles with 46 tries, so six tries were refused: three collisions. Collisions still happen; they just do not repeat.
Fixed back-off with the progress check: it fires at cycle 65 with the whole diagnosis in one line: 34 tries, 0 acks. Activity without progress.
Without it: the activity watchdog never fires, because a request appears every few cycles. The global timeout ends the run at cycle 20000 after 10,000 tries. Every one of them collided.
Starvation: make the policy fair where fairness is required: round-robin, or fixed priority with an age or credit limit that promotes a request that has waited too long. Bound the worst-case wait and write the bound down.
Livelock: break the symmetry. Back-off that differs per agent or is random (and grows on repeated collisions), or a resolver that picks one winner instead of refusing everyone. A NACK that refuses all parties is the root of most retry livelocks.
The two checks on this page are the prevention, and both belong in every bench with shared resources: an age check per requester (no request waits more than N cycles) and a progress check per agent (no completion for N cycles while it is trying). As assertions they are liveness properties with a bound, such as req |-> ##[0:N] grant (formal tools can prove the unbounded form; see deadlock and liveness). In review, ask of every arbiter: what is the worst-case wait for the lowest-priority requester, under full load from the others?, and of every retry loop: what stops two agents retrying in lockstep?
Round-robin only rotates among masters that are requesting when the arbiter chooses. If the master drops its request between choices (for example it re-requests one cycle after each refusal) or the grant is held for long transfers or locks by others, it can still wait a long time. Check the age counter alongside the request line: a request that is not continuously held, or long ownership intervals, explain it.
Two identical LFSRs give identical back-offs, so the agents are still in lockstep. Seed each one differently (from its ID, as on this page) or add a resolver that grants one requester on a collision. Check it with a directed test that starts both agents in the same cycle and a progress check per agent.
Starvation (starve_tb.sv, on this page), 2026-10-18, Icarus Verilog 13.0: iverilog -g2012 -o s.vvp starve_tb.sv && vvp -n s.vvp prints PASS with grants 245/245/10. iverilog -g2012 -Pstarve_tb.FIX=0 -o sb.vvp starve_tb.sv, vvp -n sb.vvp fails STARVE at cycle 43; vvp -n sb.vvp +AGE=100000 prints PASS with master 2 granted 0 times.
Livelock (livelock_tb.sv, on this page): iverilog -g2012 -o l.vvp livelock_tb.sv && vvp -n l.vvp prints PASS in 79 cycles. With -Plivelock_tb.FIX=0: vvp -n lb.vvp fails LIVELOCK at cycle 65; vvp -n lb.vvp +NOPROG fails TIMEOUT at cycle 20000 with 10000 tries and 0 acks, and the activity watchdog never fires. Verilator 5.048 (-GFIX=0 for the bugs) prints the same lines for both benches.
Limits: two small designs with fixed traffic patterns, simulation only. dbglab's two-master arbiter is round-robin and has no starvation bug, so it is not used here.