First silicon is on the bench. The DMA-to-peripheral test starts, and the software's poll of the DMA status never returns. The debug probe can halt the processor, but a read of the CSR block through the system bus times out as well. Someone suggests a reset and a run with more prints.
You have one chip in a state nobody has seen before, and almost no view into it. What do you do first, and how do you get this failure into a simulator?
In this lesson dbglab plays the chip. Its dma_burst_backpressure test is the software, and we allow ourselves only what a lab would get. The lab instruments themselves (JTAG, trace capture, scan control) belong to the bring-up course; this page is the method.
The reset comes first, because it is easy. It destroys the evidence, and a rare failure may not come back for days. Prints in the software come next; they change the timing, and on a hung bus they never appear. Then registers are poked through the debug port at random, which hangs too if the port shares the hung bus, or changes the state it was meant to read.
Each move treats the chip as the place to experiment. On silicon every experiment is slow and many are one-shot. The method is the reverse: take as much state as you can, as gently as you can, and do the experiments in simulation.
Confirm it is the chip. Repeat it, then change one thing at a time: another unit, another board, a known-good software build. A failure that follows the board or the build is not yet a design bug (failure isolation flow).
Preserve the state. Do not reset. Stop the software if you can do so without touching the failing logic.
Observe from least to most intrusive. Pass or fail and counters first; then registers, over a path that avoids the failing logic; then a trace buffer of the last N events; then a scan dump, which stops the clocks and shifts out every flip-flop. The dump sees the most and is one-shot.
Read the dump with a question. For a hang the question is who is waiting for whom: the state of each block that can wait, and what it waits for.
Dump twice. Two dumps some time apart: if nothing but free-running bits changed, it is a deadlock; if counters or pointers moved, it is a livelock or a very slow chip.
Load the dump into simulation and run it. If the simulation, started from the dump with the same pins, stays stuck, you have a repro from that point with full visibility. Edit one dumped bit to test each hypothesis before you try anything on the chip.
On the debug loop this is Reproduce and Observe: the dump is the observation, and the simulation that starts from it is the reproduction.
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008That is the lab's view, and it is generous: a real chip would give a software timeout. The processor is stuck on a write to 0x3008, the peripheral's control register. Nothing says why.
A scan dump shifts out a long string of bits; the DFT tool's chain map turns each bit position into a flip-flop name. Here a probe compiled next to dbglab writes the named list directly, which is what the map gives you. The list of flip-flops lives in one place, gen_scan.py, which generates both the probe and the loader below, so the two cannot disagree. The RAM goes to a separate image.
$ python3 gen_scan.py
79 flops, 1244 bits
$ ./dump.sh dma_burst_backpressure 1 1000 && cp mem.txt mem1000.txt
dbglab: test=dma_burst_backpressure seed=1
scan_dump: 79 flops and 4 pins written to scan1000.txt
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
$ cat scan1000.txt
pin.cpu_valid 1 1
pin.cpu_write 1 1
pin.cpu_addr 32 00003008
pin.cpu_wdata 32 00000005
u_fab.busy 1 0
u_fab.sel_q 4 0
u_fab.u_arb.grant_q 2 2
...
u_dma.state 4 6
u_dma.src_q 32 000011e0
u_dma.dst_q 32 00003000
...
u_dma.left_q 8 20
...
u_dma.paced_q 1 1
...
u_dma.fill_val_q 32 xxxxxxxx
...
u_csr.len_q 10 030
...
u_per.pctrl_q 8 04
u_per.u_fifo.wbin 5 10
u_per.u_fifo.wgray 5 18
u_per.u_fifo.wfull 1 1
u_per.u_fifo.rbin 5 00
...
u_per.u_en_sync.stage1 1 0
u_per.u_en_sync.stage2 1 0
...Who is waiting for whom? Each line answers part of it:
Dump line | Means | Waits for |
|---|---|---|
| the CPU is writing 05 (sink on) to the peripheral's control register | the bus to accept it |
| the DMA (master 1) holds the grant, with no transfer in flight | the DMA's next response |
| the DMA is in WAIT (state 6 of its enum), paced, with 32 words to go | space in the peripheral FIFO |
| the FIFO is full and the sink has read nothing | the sink to run |
| the sink is disabled | the CPU's write of 05 |
The last column closes on the first row: a cycle of five waits, the deadlock that the case study finds from waveforms. Here it came from 1,244 bits of a stopped chip. Two lines are not about the hang: fill_val_q and its CSR source show xxxxxxxx, because FILL has no reset in this design. A real dump would show some definite value there instead of X; note it, do not chase it now.
$ ./dump.sh dma_burst_backpressure 1 2000
...
scan_dump: 79 flops and 4 pins written to scan2000.txt
...
$ diff scan1000.txt scan2000.txt
$ cmp mem1000.txt mem.txt && echo same RAM
same RAMA thousand cycles later not one flip-flop has changed: a deadlock, not a livelock.
replay_tb instantiates dbglab_top with no bench and no reset. At time 0 it loads every flip-flop and the CPU pins from the dump, holds the pins, starts the clocks and reports the first sign of progress. This is the generated loader, trimmed:
// snippet: from replay_tb.sv, generated by gen_scan.py (79 flops; the enum needs a loop: Icarus 13 has no $cast)
while ($fscanf(f, "%s %d %h", name, width, v) == 3) begin
n_flops++;
if (0) ;
else if (name == "pin.cpu_valid") cpu_valid = v[0:0];
...
else if (name == "u_fab.u_arb.grant_q") dut.u_fab.u_arb.grant_q = v[1:0];
else if (name == "u_dma.state") begin dut.u_dma.state = dut.u_dma.state.first(); repeat (v) dut.u_dma.state = dut.u_dma.state.next(); end
...
else $fatal(1, "unknown scan entry %s", name);
end$ ./replay.sh scan1000.txt +REDUMP
replay: loaded 83 entries from scan1000.txt; pins: cpu_valid=1 cpu_write=1 cpu_addr=00003008 cpu_wdata=00000005
replay: HUNG: no bus progress and no stream word in 3000 cycles from the loaded state
replay_tb.sv:228: $finish called at 30002000 (1ps)
$ diff scan1000.txt redump.txt
80c80
< u_per.u_pd.u_sink.phase 1 0
---
> u_per.u_pd.u_sink.phase 1 1The simulation, started from the chip's state, hangs too, and after 3,000 cycles the only changed bit is the sink's free-running phase. The dump is complete enough to reproduce the failure, and the hang is a property of the design in that state, not of the board, the clock or the software. Now every signal is visible, and experiments cost a second each.
The hypothesis is the cycle in the table. It predicts that breaking any one edge ends the hang. Two edits, each in its own copy of the dump:
$ sed 's/^u_per.pctrl_q 8 04$/u_per.pctrl_q 8 05/' scan1000.txt > edit_sink.txt
$ sed 's/^u_fab.u_arb.grant_q 2 2$/u_fab.u_arb.grant_q 2 0/' scan1000.txt > edit_grant.txt
$ ./replay.sh edit_sink.txt
replay: loaded 83 entries from edit_sink.txt; pins: cpu_valid=1 cpu_write=1 cpu_addr=00003008 cpu_wdata=00000005
replay: bus progress at cycle 18: m_valid=11 m_ready=10 m_rvalid=00
replay: cycle 43: the CPU's write of 00000005 to 00003008 is accepted
replay: RUNNING: 48 stream words in 3000 cycles; DMA state=0 left=0
...
$ ./replay.sh edit_grant.txt
replay: loaded 83 entries from edit_grant.txt; pins: cpu_valid=1 cpu_write=1 cpu_addr=00003008 cpu_wdata=00000005
replay: bus progress at cycle 2: m_valid=01 m_ready=01 m_rvalid=00
replay: cycle 2: the CPU's write of 00000005 to 00003008 is accepted
replay: RUNNING: 48 stream words in 3000 cycles; DMA state=0 left=0Sink on (as if the CPU's write had landed): the sink drains, the DMA gets space and moves at cycle 18 (m_ready=10 is the DMA's), and the CPU's write gets in at cycle 43, between the DMA's chunks. All 48 words of the run reach the stream.
Grant released (as if the fabric had re-arbitrated): the CPU's write is accepted at once, the sink starts, and the DMA finishes.
Both predictions held, so the cycle is the cause of the hang. The edit to the grant also says which rule to question: the fabric keeps a locked master's grant while that master waits for something else.
A lab can test the same prediction if the chip has a path to the peripheral's control register that does not use the system bus: writing sink-enable through it should end the hang. That is a design-for-debug question worth asking before tape-out.
From a chip that said only "timeout", one dump gave the wait-for cycle, a second dump showed it was a true deadlock, a replay proved the dump reproduces it, and two one-bit edits confirmed the mechanism. None of it needed the chip again. The next step is to reproduce it from reset, at the cheapest stage that shows it, and that is the subject of moving the failure to the cheapest stage. Here the bench's own test already does it.
It is tempting to load the dump into the fixed RTL and watch it recover. The fix for this bug releases the grant at every response. Loaded with this state it stays hung:
$ D=../fixed ./replay.sh scan1000.txt
replay: loaded 83 entries from scan1000.txt; pins: cpu_valid=1 cpu_write=1 cpu_addr=00003008 cpu_wdata=00000005
replay: HUNG: no bus progress and no stream word in 3000 cycles from the loaded state
...
$ ../fixedtree/tb/run.sh dma_burst_backpressure 1 # dbglab with the same fix, from reset
dbglab: test=dma_burst_backpressure seed=1
PASS test=dma_burst_backpressure seed=1 cycles=772 time=7716 nsThe fix prevents the state; it cannot leave it, because the grant was kept at a response that has already happened. The full test from reset passes. Prove a fix by running from reset, never by loading the failing state into the fixed design.
A dump is only as good as its chain map and the netlist version behind it. An engineering change made after the map was generated, or a simulation of a newer RTL, shifts names and values silently. Check the dump against things you know: the ID register, a register you just wrote, a counter that must be odd. Here u_csr.len_q 030 (48, the length the test programmed) is such a check.
It is not a deadlock: the DMA is retrying, so this is a livelock (or a very slow retry loop). Look at what it retries and why each attempt fails: the response it gets, the slave it addresses, the condition it waits for. Load one dump into simulation and run it long enough to see two retries with full visibility; the failing response or condition is the next suspect.
The dump does not match the simulated design (another netlist version, a wrong chain map); something that matters is not in the dump (a memory, a register off the chain, an input pin); or the hang needs timing the replay lacks, such as the phase between two clocks. Check the dump against known values and list what it leaves out before suspecting the silicon.
dbglab at HEAD 49c5f04 plays the chip; no silicon, lab or debug port was used. Icarus Verilog 13.0, Python 3, 2026-10-18. tb/run.sh dma_burst_backpressure 1: FAIL ... cycle=2165: WATCHDOG. gen_scan.py (a list of all 79 flip-flops of dbglab_top, 1,244 bits) generated the probe scan_dump.sv and the harness replay_tb.sv; dump.sh dma_burst_backpressure 1 1000 and ... 2000 wrote scan1000.txt and scan2000.txt (the RAM image of the first kept as mem1000.txt); diff of the two dumps: empty; cmp of the RAM images: identical.
replay.sh scan1000.txt +REDUMP (dbglab RTL, no bench, no reset, pins held): HUNG for 3,000 cycles; the end state differs from the dump only in u_sink.phase. With u_per.pctrl_q edited 04 to 05: the CPU write accepted at cycle 43, 48 stream words, DMA idle. With u_fab.u_arb.grant_q edited 2 to 0: accepted at cycle 2, 48 words. With the fabric fixed (assign accept = rsp_fire;) and the unedited dump: still HUNG; the full test with the fix (fixedtree/tb/run.sh dma_burst_backpressure 1, dbglab with that one line changed): PASS ... cycles=772 time=7716 ns.
Limits: the dump is read from simulation by name, so the chain map is exact by construction; a real dump needs the DFT tool's map and the same netlist version. The RAM is loaded from a separate image, which on silicon needs a read-out path that does not use the hung bus. The replay starts both clocks fresh, so their phase differs from the original run; that cannot matter for a state in which nothing moves, and could for one that does.