dbglab's dma_burst_backpressure test hangs: the watchdog fires at cycle 2165 with the CPU waiting for the bus. An engineer looks at the hang and sees the peripheral FIFO full, so they make it four times deeper. A colleague suggests the DMA should not hold the bus lock while it waits for FIFO space, so that goes in too. One patch, two changes:
$ git diff
...
- periph_slave #(.AW(4)) u_per (
+ periph_slave #(.AW(6)) u_per (
...
- || (state == WAIT) || (state == WR_REQ) || (state == WR_RSP)) && !chunk_end;
+ || (state == WR_REQ) || (state == WR_RSP)) && !chunk_end;
...
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=787: the FIFO filled only 0 times: no back-pressure was testedNo more watchdog. The test now fails for a new reason that sounds like the bench being fussy. Is the deadlock fixed?
They change several things at once, because each run seems expensive and every change seems sensible. When the failure goes away they keep the whole patch. Here, the new message looks like a complaint about the test, so the obvious next step is to relax it. On a copy with that check disabled:
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
PASS test=dma_burst_backpressure seed=1 cycles=787 time=7866 nsGreen. The patch goes to review with the message fixed the DMA deadlock by not locking in WAIT; deeper FIFO for headroom. Three changes now (two RTL, one bench), and the run that "proves" them answers one question, does this combination pass this test?, when the question that matters is which change removed the cause?
Keep the failing command fixed. The same test, seed, simulator and options as the repro. That command is your baseline.
Change one thing per run, then go back to the baseline. Not cumulatively: each change starts from the same tree.
Count everything as a change. RTL, bench, seed, test length, simulator, a tool option, a rebuilt library. git diff --stat before believing a result.
For a patch with n changes, run each alone first. Two changes: neither, A, B, both. Add combinations only if the single runs do not explain the result.
Read a changed message as a changed measurement. A different failure after a change means the test is now seeing something else. Find out what before you edit the check.
Keep the change that removes the cause, and only that, with the test that failed before and passes after.
Read the grid along each direction. Whatever the lock does, the result in a column is identical, so the lock change does nothing to this failure. Whatever the lock does, 64 entries change the message, so the depth is the only change that does anything. What it does is the next question.
$ tb/run.sh dma_burst_backpressure 1 # HEAD with only WAIT removed from m_lock
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008The same message at the same cycle as HEAD. This change has no effect on the hang. The case study explains why: the fabric decides whether to keep the grant at the response before WAIT, so what the lock does during WAIT is never looked at.
$ tb/run.sh dma_burst_backpressure 1 # HEAD with only the FIFO deepened
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=787: the FIFO filled only 0 times: no back-pressure was testedThis is the change that made the watchdog go away, and the message says how. The test copies 48 words into the peripheral and exists to fill its FIFO so the DMA has to wait. With 64 entries the FIFO never fills, the DMA never waits, and the deadlock, which needs a waiting DMA, never gets a chance. The bench's own guard noticed that its stimulus no longer does its job. Disabling that guard is what turned "the test can no longer see the bug" into PASS.
A prediction: if the deeper FIFO only hides the deadlock, a copy longer than 64 words will fill it and hang again, with both changes applied. On a copy of the bench with len = 120:
$ tb/run.sh dma_burst_backpressure 1 # both changes, 120-word copy
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2549: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008The same hang, the same waiting CPU write to 0x3008, only later. The patch under review would have shipped the deadlock, a FIFO four times bigger, and a test that can no longer find it.
Start from the repro and one hypothesis at a time. The case study's hypothesis is that the fabric keeps the grant for a locked owner at every response, so a DMA that locks and then waits for an event that needs the bus holds everyone off. Its fix is one line, applied alone:
$ tb/run.sh dma_burst_backpressure 1 # HEAD with only accept = rsp_fire in bus_fabric.sv
dbglab: test=dma_burst_backpressure seed=1
PASS test=dma_burst_backpressure seed=1 cycles=772 time=7716 ns
$ tb/run.sh dma_burst_backpressure 1 # the same fix, 120-word copy
dbglab: test=dma_burst_backpressure seed=1
PASS test=dma_burst_backpressure seed=1 cycles=1690 time=16896 nsIt passes with the original 16-entry FIFO, with the back-pressure check intact (seeds 2 and 3 pass too), and with the longer copy that hangs HEAD at cycle 2165. One change, one run, and the result means something.
Run | Changes from HEAD | Result | What it tells you |
|---|---|---|---|
HEAD | none | WATCHDOG 2165 | the baseline |
A | WAIT dropped | WATCHDOG 2165 | A has no effect |
B | 64 entries | FIFO filled 0 times | B removes the stimulus, not the bug |
A + B, 120 words | both, longer copy | WATCHDOG 2549 | the bug survives the patch |
fabric fix | accept = rsp_fire | PASS (48 and 120 words) | the cause |
Split into single changes, the patch explained itself in four runs: one change was inert, one moved the test away from the bug, and the real cause was elsewhere. Together they produced a convincing green run. This is the Test step done badly and then well, and the Fix step depends on it: a fix is only the change that removed the cause, proven alone.
Three runs, each change alone against the failing command, plus the baseline you already have. If exactly one of them passes, and the test still exercises the failing condition (check its guards or coverage, and push the stimulus further), that is your fix, and the combined run adds nothing. You need combinations only if no single change passes, which means the effect needs two changes together (an interaction); then try the pairs.
The check passed before the change and fails after it, so the change altered what the test does: here the FIFO no longer fills, so the condition the test exists for never happens. Deleting the check would hide that. First find what the change did to the stimulus; then either keep the change out of the fix, or adjust the test so it still reaches the condition (a longer copy), in a separate commit.
dbglab at HEAD 49c5f04 (rebuilt with dbglab/build-history.sh), Icarus Verilog 13.0, 2026-10-17, test dma_burst_backpressure. Each change on its own copy of HEAD: periph_slave #(.AW(6)) in dbglab_top.sv (64 entries); WAIT removed from m_lock in dma_engine.sv; both; assign accept = rsp_fire; in bus_fabric.sv. tb/run.sh dma_burst_backpressure 1 gives WATCHDOG at 2165 (HEAD, WAIT only), "filled only 0 times" at 787 (depth only, both), PASS in 772 cycles (fabric fix; seeds 2 and 3: 809 and 772).
Bench edits on copies: with the bursts < 2 check disabled, both changes PASS (seeds 1 and 2, 787 cycles). With len = 120: both changes WATCHDOG at 2549, HEAD WATCHDOG at 2165, fabric fix PASS in 1690 cycles.
Limits: one bug, one test, Icarus only (the case study shows the hang is identical in Verilator). The scene's patch is illustrative; every run in it is real.