The report

Tuesday's regression of dbglab has three new red lines. Nobody changed the test; nobody changed the peripheral. This is the whole report:

$ tb/regress.sh icarus 1 2 3
...
dma_burst_backpressure   seed 1      FAIL
dma_burst_backpressure   seed 2      FAIL
dma_burst_backpressure   seed 3      FAIL
...
$ grep FAIL build/icarus-dma_burst_backpressure-1.log
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008

What the test does: it stops the peripheral's sink, then starts a DMA copy of 48 words into the peripheral's FIFO with paced writes, so the DMA writes a word only when the FIFO has space (dreq). Each time the CPU finds the FIFO full, it turns the sink on for a while by writing CTRL at 0x3008, then off again. The FIFO fills and drains, again and again. That is back-pressure, and the test passed last week.

Stop here if you want to try it yourself. You have the design (dbglab, the debug loop lesson shows how to probe it) and the line above. Where would you look first?

The hunt

Reproduce: it fails the same way every time

$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
$ tb/run.sh dma_burst_backpressure 1 verilator
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008

Same cycle in both simulators, and every seed from 1 to 50 fails. This is not a race or a rare ordering; something always gets stuck. Good news: a deterministic hang is the easy kind.

Observe: there is no first wrong value

Your usual move, walking back from the first wrong value, has nothing to start from. No read returned bad data; the scoreboard never complained. The watchdog fired because nothing happened for 2,000 cycles. The last progress was at cycle 164, and the 2,000 cycles after it are identical. A waveform of the hang shows flat lines.

So change the question. In a hang, every agent is waiting for something. Take a snapshot of each one at the hang and write down what it is waiting for. A probe compiled next to the design prints that snapshot:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// hang_probe: a second top-level module compiled next to dbglab. It reads the design through
// hierarchical names and changes nothing. +AT=<c> prints a snapshot at bus cycle c;
// +FROM=<a> +TO=<b> prints one line per cycle from a to b.
`timescale 1ns/1ps
module hang_probe;
  int unsigned at_c, from_c, to_c;
  initial begin
    if (!$value$plusargs("AT=%d", at_c))     at_c   = 0;
    if (!$value$plusargs("FROM=%d", from_c)) from_c = 1;
    if (!$value$plusargs("TO=%d", to_c))     to_c   = 0;
    if (to_c >= from_c) $display("cycle  cpu v r addr     grant lock  dma     dreq full sink_en");
  end
  always @(posedge dbglab_tb.clk) begin
    #0.5;
    if (dbglab_tb.cycle >= from_c && dbglab_tb.cycle <= to_c)
      $display("%5d      %b %b %h  %b    %b    %-7s %b    %b    %b",
        dbglab_tb.cycle, dbglab_tb.dut.m_valid[0], dbglab_tb.dut.m_ready[0], dbglab_tb.dut.m_addr[0],
        dbglab_tb.dut.u_fab.grant, dbglab_tb.dut.m_lock, dbglab_tb.dut.u_dma.state.name(),
        dbglab_tb.dut.dreq, dbglab_tb.dut.u_per.wfull, dbglab_tb.dut.u_per.pctrl_q[0]);
    if (dbglab_tb.cycle == at_c) begin
      $display("--- cycle %0d ---", at_c);
      $display("cpu   valid=%b ready=%b write=%b addr=%h wdata=%h", dbglab_tb.dut.m_valid[0], dbglab_tb.dut.m_ready[0],
               dbglab_tb.dut.m_write[0], dbglab_tb.dut.m_addr[0], dbglab_tb.dut.m_wdata[0]);
      $display("fab   grant=%b busy=%b lock=%b", dbglab_tb.dut.u_fab.grant, dbglab_tb.dut.u_fab.busy, dbglab_tb.dut.m_lock);
      $display("dma   state=%s paced=%b dreq=%b", dbglab_tb.dut.u_dma.state.name(), dbglab_tb.dut.u_dma.paced_q, dbglab_tb.dut.dreq);
      $display("fifo  full=%b empty=%b", dbglab_tb.dut.u_per.wfull, dbglab_tb.dut.u_per.rempty);
      $display("sink  sink_en(reg)=%b sink_en(pclk)=%b", dbglab_tb.dut.u_per.pctrl_q[0], dbglab_tb.dut.u_per.sink_en_p);
    end
  end
endmodule

$ ../probes/run-probe.sh ../probes/hang_probe.sv dma_burst_backpressure 1 +AT=2165
dbglab: test=dma_burst_backpressure seed=1
--- cycle 2165 ---
cpu   valid=1 ready=0 write=1 addr=00003008 wdata=00000005
fab   grant=10 busy=0 lock=10
dma   state=WAIT paced=1 dreq=0
fifo  full=1 empty=0
sink  sink_en(reg)=0 sink_en(pclk)=0
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008

Now turn each line into a row: who is waiting, for what, and the evidence. Bit 1 of grant and lock is the DMA, bit 0 the CPU.

Read down the second column and you come back to where you started: the CPU waits for the fabric, the fabric for the DMA, the DMA for the FIFO, the FIFO for the sink, the sink for the CPU. That is a cycle of waits, which is what a deadlock is. Nothing outside the cycle can break it, so the watchdog is the only thing that ever happens again.

The picture folds the sink into the FIFO box, so the five rows become four arrows. Three of the four are normal, designed behaviour: a paced DMA waits for space, a stopped sink does not drain, a CPU waits for the bus. Only one arrow is a choice: the fabric keeping its grant for an owner that is not using it. That is where to look.

Localise: when did the cycle close?

A hang has no first wrong value, but it has a first cycle in which the cycle of waits is complete. Trace the window before the last progress:

$ ../probes/run-probe.sh ../probes/hang_probe.sv dma_burst_backpressure 1 +FROM=150 +TO=169
dbglab: test=dma_burst_backpressure seed=1
cycle  cpu v r addr     grant lock  dma     dreq full sink_en
  150      1 0 00003004  10    10    WAIT    1    0    0
  151      1 0 00003004  10    10    WR_REQ  1    0    0
  152      1 0 00003004  10    00    WR_RSP  0    1    0
  153      1 0 00003004  00    00    CHUNK   0    1    0
  154      1 1 00003004  01    10    RD_REQ  0    1    0
  155      1 0 00003004  01    10    RD_REQ  0    1    0
  156      0 0 00003004  00    10    RD_REQ  0    1    0
  157      1 0 00003008  10    10    RD_REQ  0    1    0
  158      1 0 00003008  10    10    RD_RSP  0    1    0
...
  164      1 0 00003008  10    10    RD_RSP  0    1    0
  165      1 0 00003008  10    10    POP     0    1    0
  166      1 0 00003008  10    10    LOAD    0    1    0
  167      1 0 00003008  10    10    WAIT    0    1    0
  168      1 0 00003008  10    10    WAIT    0    1    0
  169      1 0 00003008  10    10    WAIT    0    1    0
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008

• Cycle 152: the DMA's last write of a chunk fills the FIFO (full goes to 1), and the chunk's last response drops lock.

• Cycles 153 to 155: the grant is released; the CPU wins it and reads the peripheral STATUS (0x3004). It sees the FIFO full, so its next move is to turn the sink on.

• Cycle 157: the CPU asks to write CTRL (0x3008), one cycle too late: the DMA, already waiting, won the grant at this edge and raised lock for its next chunk.

• Cycles 158 to 166: the DMA reads four words while the CPU waits. Before last week, the CPU would have won the bus after the first of these responses.

• Cycle 167, the cycle closes: the DMA enters WAIT with dreq 0, still holding lock and the grant. From here every line is the same.

Who changed the rules? The test passed last week, so bisect it, exactly as in bisection in history:

$ git bisect start HEAD v0.1
...
$ git bisect run tb/run.sh dma_burst_backpressure 1
...
PASS test=dma_burst_backpressure seed=1
...
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
...
c4a71b585e287d527b9c2a827a3969bca0334445 is the first bad commit
commit c4a71b585e287d527b9c2a827a3969bca0334445
Author: dbglab <dbglab@example.com>
Date:   Thu Mar 12 10:00:00 2026 +0000

    dma: keep the bus for a whole chunk

    The arbiter re-arbitrates after every response, which costs a cycle
    per beat. The DMA now raises m_lock from the first read of a chunk to
    its last write, and the fabric keeps the grant while the owner holds
    lock. dma_copy runs about 20% faster.
...

Four runs, and the guilty commit is a performance change, not a bug fix, which is exactly why nobody suspected it. Its two key lines:

1
2
3
4
5

// snippet: commit 11, rtl/bus_fabric.sv line 35 and rtl/dma_engine.sv lines 74-75
  assign accept  = rsp_fire && !m_lock[owner];      // a locked owner keeps the bus for its next beat

  assign m_lock = ((state == RD_REQ) || (state == RD_RSP) || (state == POP) || (state == LOAD)
                || (state == WAIT) || (state == WR_REQ) || (state == WR_RSP)) && !chunk_end;

The waveform below shows the same mechanism in a reduced copy: dbglab's arbiter unchanged, these two rules copied, a two-word chunk and a two-entry FIFO that starts full with the sink stopped. Grant values: 0 nobody, 1 CPU, 2 DMA.

Read it as two masters and one bus: find the slot where the CPU should have won and did not.

• Slots 0 to 1: the DMA requests and is granted (grant = 2); dma_lock is high for the whole chunk.

• Slot 2: the CPU raises cpu_valid for the write that would turn the sink on.

• Slot 3: the DMA's first read has had its response, so the old fabric would hand the bus to the waiting CPU here. With lock high it keeps it, and the DMA makes its second read.

• Slots 5 to 11: the DMA has stopped requesting (dma_valid = 0) and waits for dreq; grant stays 2, cpu_ready stays 0. This never ends.

Hypothesise and test: the fix that looks right

Hypothesis: the DMA holds the bus through WAIT, and WAIT can only end after a bus write by someone else. The obvious fix follows: stop asserting lock in WAIT. Before you run it, predict. The fabric only looks at lock at a response: accept is rsp_fire && !m_lock[owner]. The last response before WAIT is the fourth read's, at cycle 164, when lock was still high, so the grant was kept then. In WAIT no response ever comes, so accept never fires, whatever lock says. Prediction: this fix still hangs, at the same cycle.

$ sed -n 74,75p rtl/dma_engine.sv        # a copy of HEAD with WAIT removed from m_lock
  assign m_lock = ((state == RD_REQ) || (state == RD_RSP) || (state == POP) || (state == LOAD)
                || (state == WR_REQ) || (state == WR_RSP)) && !chunk_end;
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008

The prediction held: the plausible fix fails. This experiment is worth more than a pass, because it shows exactly which arrow in the cycle matters. It is the fabric's decision to keep the grant at a response, made before the DMA knew it would have to wait. A fix has to change that decision.

The fix and the lesson

Fix: never keep a grant for an owner that may wait

The safe fix puts back the rule from before commit 11: the fabric re-arbitrates after every response.

// snippet: the fix, rtl/bus_fabric.sv line 35
  assign accept  = rsp_fire;

Prove: the repro, a sweep, and nothing new

$ sed -n 35p rtl/bus_fabric.sv
  assign accept  = rsp_fire;
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
PASS test=dma_burst_backpressure seed=1 cycles=772 time=7716 ns
$ for s in $(seq 1 50); do tb/run.sh dma_burst_backpressure $s > /dev/null || echo "fail $s"; done
$ tb/regress.sh icarus 1 2 3 | grep -E "FAIL|failing"
irq_w1c                  seed 1      FAIL
irq_w1c                  seed 2      FAIL
irq_w1c                  seed 3      FAIL
periph_power             seed 1      FAIL
periph_power             seed 2      FAIL
periph_power             seed 3      FAIL
6 failing run(s)

The repro passes, the 50-seed sweep prints nothing, and the regression's only failures are two other known bugs with their own signatures. The random test, which mixes CPU and DMA traffic for 60 seeds, shows no WATCHDOG either. One cost is real: dma_copy seed 1 goes back from 11816 ns to 15366 ns. The fix gives up last week's speed-up, and the lens below shows how to keep it.

Prevent: a check that fires 1,984 cycles earlier

The watchdog caught the hang 2,000 cycles late and could only say nothing moves. A check on the rule itself fires as soon as it is broken: an owner that holds the grant without using it while the other master waits.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// grant_idle_check: fails when a bus owner holds the grant without using it while another master waits.
`timescale 1ns/1ps
module grant_idle_check #(parameter int LIMIT = 16);
  logic [1:0] grant, valid;
  logic       busy;
  int unsigned idle = 0;
  assign grant = dbglab_tb.dut.u_fab.grant;
  assign valid = dbglab_tb.dut.m_valid;
  assign busy  = dbglab_tb.dut.u_fab.busy;
  always @(posedge dbglab_tb.clk) begin
    if (|(grant & ~valid) && !busy && |(valid & ~grant)) idle++;
    else idle = 0;
    if (idle > LIMIT) begin
      $display("FAIL cycle=%0d: GRANT_IDLE: owner grant=%b has not used the bus for %0d cycles while valid=%b waits",
               dbglab_tb.cycle, grant, idle, valid);
      $fatal(1, "grant held while idle");
    end
  end
endmodule

$ ../probes/run-probe.sh ../probes/grant_idle_check.sv dma_burst_backpressure 1      # at HEAD
dbglab: test=dma_burst_backpressure seed=1
FAIL cycle=181: GRANT_IDLE: owner grant=10 has not used the bus for 17 cycles while valid=01 waits

It fires at cycle 181 instead of 2165, and its message names the rule, the owner and the waiter. With the fix it stays silent on every test it was run on. The limit of 16 cycles is a design choice: long enough for a chunk's internal steps, short enough to catch an owner that is waiting on the outside world. In a real bench this belongs in the bus checker as an assertion, next to the valid/ready rules.

Which step won this hunt? Observe. The wait-for table turned "nothing happens" into a cycle with one suspicious arrow. Test saved it from a wrong fix that looked like the obvious one.

Related topics

• Part 10 · Case Studies

• Part 7: hangs and deadlocks

• Part 5: bug classes

• Bisection in history

• RTL: the valid/ready handshake

• RTL: bus arbitration

• RTL: stalling without deadlock