Shrinking took dbglab's failing seed 10 down to one operation: a DMA copy of 308 words that leaves the end of the destination wrong. But that one operation still runs on the whole system: the CPU model, the arbiter and fabric, the CSR block, a stream peripheral in a second clock domain behind a power gate. A Verilator build takes eight seconds, and a waveform shows signals from fifteen modules. How much of that does the bug need?
Most engineers keep debugging in the full system because that is where the test lives. Every experiment then pays for the whole design, and every waveform shows everything. On a real chip that can mean an hour per run and a dump nobody can read.
The other habit is worse: copy the suspect block into a fresh bench and edit it until it compiles, renaming a package here and deleting a port there. Now you are debugging a copy. If the copy passes, you do not know whether the bug is outside the block or whether your edits removed it.
Choose what to keep from evidence, not from a hunch. Here the one-operation test says a DMA copy ends short, and the debug loop probe saw dma_engine write 52 words of 308. Keep dma_engine, the sync_fifo it instantiates, and a real slave to copy into.
Compile the kept RTL from where it lives. Same files, same package, no edits. The reduced bench is a new top level, not a new design.
Stub each cut neighbour to its interface contract. The fabric's contract for the DMA is valid/ready, one request in flight, one response pulse. The stub does that and nothing more.
Drive configuration directly. Instead of a CPU writing registers over the bus, set the DMA's configuration inputs from the bench.
Check the signature. The reduced bench must fail the same way: a 308-word copy that writes 52 words. If it does not, stop and find out why before going on.
Use the speed. A reduced bench is cheap enough to sweep every length, dump every signal, or hand to a formal tool.
The reduced bench instantiates dbglab's dma_engine and ram_slave unchanged. The stub is three lines: ready while nothing is in flight, and in flight from the handshake until the RAM's response.
// not-run: compiles against dbglab's rtl/ files (dbglab_pkg, sync_fifo, ram_slave, dma_engine); command in the Evidence box
// dma_alone_tb: dbglab's dma_engine (with its sync_fifo) and ram_slave, nothing else.
// The bus fabric, arbiter, CSR block, CPU and peripheral are cut; a three-line stub stands in
// for the fabric, and the bench drives the DMA's configuration pins directly.
// +LEN=<n> one copy of n words (default 308) +SWEEP every length from 1 to 511
// +PACED paced writes: dreq (the peripheral's "space") is low for 40 of every 64 cycles
`timescale 1ns/1ps
module dma_alone_tb;
import dbglab_pkg::*;
logic clk = 1'b0, rst_n = 1'b0;
always #5 clk = ~clk;
logic start = 1'b0, busy, done, err;
logic [31:0] cfg_src, cfg_dst, err_addr;
logic [9:0] cfg_len;
logic m_valid, m_ready, m_write, m_lock, rv;
logic [31:0] m_addr, m_wdata, rdata;
logic [1:0] resp;
logic [7:0] ctrl = 8'h00;
logic dreq = 1'b1;
dma_engine u_dma (.clk, .rst_n, .start, .cfg_src, .cfg_dst, .cfg_len, .cfg_ctrl(ctrl),
.cfg_fill(32'h0), .dreq, .busy, .done, .err, .err_addr,
.m_valid, .m_ready, .m_addr, .m_wdata, .m_write, .m_lock,
.m_rvalid(rv), .m_rdata(rdata), .m_resp(resp));
// Stub for the fabric: one master, one slave, one request in flight.
logic inflight = 1'b0;
assign m_ready = m_valid && !inflight;
always_ff @(posedge clk) if (m_valid && m_ready) inflight <= 1'b1; else if (rv) inflight <= 1'b0;
ram_slave u_ram (.clk, .rst_n, .req(m_valid && m_ready), .write(m_write),
.addr(m_addr), .wdata(m_wdata), .rv, .rdata, .resp);
int unsigned cycle = 0, reads, writes;
always @(posedge clk) cycle <= cycle + 1;
always @(posedge clk) if (ctrl[CTRL_PACED]) dreq <= (cycle % 64) >= 40;
always @(posedge clk) if (m_valid && m_ready) begin if (m_write) writes++; else reads++; end
// Copy len words (at most 511) from word 0 to word 512; wrong = the first wrong word, or -1.
task automatic copy(input int len, output int wrote, output int wrong);
int d;
d = 512;
for (int i = 0; i < 1024; i++) u_ram.mem[i] = 32'hA000_0000 + i;
reads = 0; writes = 0;
@(posedge clk) #1;
cfg_src = 32'h1000; cfg_dst = 32'h1000 + 4 * d; cfg_len = len; start = 1'b1;
@(posedge clk) #1 start = 1'b0;
while (!done) @(posedge clk) #1;
wrote = writes;
wrong = -1;
for (int i = len - 1; i >= 0; i--) if (u_ram.mem[d + i] !== 32'hA000_0000 + i) wrong = i;
endtask
initial begin
int len, wrote, wrong, bad, first_bad;
bit sweep;
sweep = $test$plusargs("SWEEP");
if ($test$plusargs("PACED")) ctrl = 8'h1 << CTRL_PACED;
if (!$value$plusargs("LEN=%d", len)) len = 308;
repeat (2) @(posedge clk);
rst_n = 1'b1;
if (!sweep) begin
copy(len, wrote, wrong);
$display("LEN=%0d: read %0d, wrote %0d words, done at cycle %0d", len, reads, wrote, cycle);
if (wrong >= 0) $fatal(1, "FAIL: LEN=%0d, destination word %0d is wrong", len, wrong);
end else begin
bad = 0; first_bad = 0;
for (len = 1; len <= 511; len++) begin
copy(len, wrote, wrong);
if (wrong >= 0) begin bad++; if (first_bad == 0) first_bad = len; end
if (wrong >= 0 && (len < 260 || len > 509))
$display("LEN=%0d: read %0d, wrote %0d, first wrong word %0d", len, reads, wrote, wrong);
end
$display("sweep: %0d of 511 lengths fail, the first is %0d; %0d cycles", bad, first_bad, cycle);
if (bad != 0) $fatal(1, "FAIL: %0d lengths leave a wrong word", bad);
end
$display("PASS: dma_alone");
$finish;
end
endmodule$ R=../dbglab/rtl
$ iverilog -g2012 -Wno-timescale -o alone.vvp $R/dbglab_pkg.sv $R/sync_fifo.sv $R/ram_slave.sv $R/dma_engine.sv dma_alone_tb.sv
../dbglab/rtl/sync_fifo.sv:75: vvp.tgt sorry: Case unique/unique0 qualities are ignored.
$ vvp -n alone.vvp
LEN=308: read 52, wrote 52 words, done at cycle 381
FATAL: dma_alone_tb.sv:66: FAIL: LEN=308, destination word 52 is wrong
Time: 3806000 Scope: dma_alone_tb.$unm_blk_21
$ vvp -n alone.vvp +LEN=255
LEN=255: read 255, wrote 255 words, done at cycle 1853
PASS: dma_alone
...The Icarus note about unique is about dbglab's FIFO and is harmless. Then the line that matters: 308 words asked, 52 read and written, the first wrong word is number 52. That is the full system's signature, the same 52 words the probe saw inside dbglab. The reduction holds the bug. A copy of 255 words, one under the line, is correct.
Full dbglab, | Reduced: DMA, FIFO, RAM, stub | |
|---|---|---|
RTL and bench compiled | 15 RTL files and the bench, 1,465 lines | 4 RTL files and an 81-line bench, 374 lines |
Cycles until the failure | 4,895 | 381 |
Icarus run | 0.06 s | under 0.01 s |
Verilator build and run | 8.35 s | 3.45 s |
Exploring copy lengths | 60 random seeds: 10.6 s, 10 of them happen to make a long run | every length from 1 to 511: 1.44 s |
Signals in a waveform | 15 modules, two clocks, a power domain | 3 modules, one clock |
dbglab is small, so the wall-clock gains are fractions of a second. The real gains are the cycle count (about 13 times fewer), the signals a waveform shows, and above all the last row: the reduced bench can ask questions the full system can only answer by luck.
$ /usr/bin/time -p vvp -n alone.vvp +SWEEP
LEN=256: read 256, wrote 256, first wrong word 4
LEN=257: read 1, wrote 1, first wrong word 1
LEN=258: read 2, wrote 2, first wrong word 2
LEN=259: read 3, wrote 3, first wrong word 3
LEN=510: read 254, wrote 254, first wrong word 254
LEN=511: read 255, wrote 255, first wrong word 255
sweep: 256 of 511 lengths fail, the first is 256; 476351 cycles
FATAL: dma_alone_tb.sv:76: FAIL: 256 lengths leave a wrong word
Time: 4763506000 Scope: dma_alone_tb.$unm_blk_21
real 1.44
...Every length from 256 to 511 fails, and none below. Not "long runs sometimes fail": exactly the lengths that need a ninth bit.
257 to 511 copy the length minus 256. 257 copies 1 word, 511 copies 255. That is a counter losing its top bits.
256 is different: 256 words are read and written, but from word 4 on they are wrong. A two-line probe beside the bench shows why: the counter loads 0, the 3-bit chunk counter wraps, and the DMA makes 8 reads before its first write into a FIFO with 4 entries. Same cause, a second symptom, and one the random test had never produced.
No amount of reading seed 10's waveform would have shown the 256 case: the random test draws long runs of 257 to 320 words, so it never makes that copy. A sweep that costs one and a half seconds did.
The bug needs nothing but dma_engine and its FIFO: no CPU, no arbiter, no CSR block, no second clock. That rules out the whole bus and register path in one experiment, and it makes the next steps of the loop cheap: the hypothesis ("the beat counter is too narrow for a 10-bit length") is now one line of RTL in a 139-line file, and a fix can be tested over all 511 lengths in under two seconds before going back to the full regression.
dbglab has a second DMA-related bug, the deadlock under back-pressure. Reduce it the same way, with the DMA in paced mode and dreq low most of the time to imitate a full peripheral FIFO:
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
$ vvp -n alone.vvp +PACED +LEN=48
LEN=48: read 48, wrote 48 words, done at cycle 763
PASS: dma_alone
...The reduced bench passes. That is not a contradiction; it is information. The deadlock is a cycle of waits through four agents: the DMA holds the bus, waits for FIFO space; the FIFO needs the sink; the sink needs a CPU write; the CPU needs the bus. The stub has no second master and ignores m_lock, so the cycle cannot close. When a reduction passes, the cause lives in what you cut, often an interaction. Put blocks back one at a time, the fabric first here, and the failure returns when the missing piece does. That is bisection in the design hierarchy, the subject of Part 3.
Do not debug the new failure yet. A different signature means the reduction changed the behaviour: most likely the stub breaks the interface contract, or the bench drives the block in a way the real system never does. Check the stub against the protocol rules and the configuration against what the full system programmed. Only when the reduced bench fails with the original signature is it a reproduction of your bug. (The protocol error may still be worth a note: if the block can be driven that way in the real system, it is a second bug.)
Because the point of the reduction is to run the same RTL in a smaller world. A copy can drift: a package edited to make it compile, a parameter changed, a fix made in the project but not in the copy. Then a pass or a fail in the reduced bench says nothing about the real design. Compiling the same files means any fix you test there is the fix you will commit.
dbglab at HEAD 49c5f04, Icarus Verilog 13.0, Verilator 5.048, 2026-10-17. Reduced bench dma_alone_tb.sv as shown (81 lines), compiled with iverilog -g2012 -Wno-timescale -o alone.vvp $R/dbglab_pkg.sv $R/sync_fifo.sv $R/ram_slave.sv $R/dma_engine.sv dma_alone_tb.sv (R = dbglab's rtl/) and run with vvp -n alone.vvp, +LEN=255, +LEN=256, +SWEEP (1.44 s) and +PACED +LEN=48. A two-line probe module compiled beside it printed 8 reads before the first write for LEN 256 (4 for LEN 255).
Timings with /usr/bin/time -p: full design vvp -n build/dbglab.vvp +TEST=random +SEED=10 0.06 s, reduced vvp -n alone.vvp under 0.01 s; Verilator build and run, full (tb/run.sh random 10 verilator after rm -rf build/vl) 8.35 s, reduced (verilator --binary --timing ... --top-module dma_alone_tb) 3.45 s; full design, random seeds 1 to 60 in Icarus, 10.6 s. Line counts with wc -l.
Trap: tb/run.sh dma_burst_backpressure 1 fails with the WATCHDOG at cycle 2165; the reduced bench with +PACED +LEN=48 passes.
Limits: dbglab is small, so its wall-clock gains are fractions of a second; the cycle counts and the sweep are the real gain here. On a large design the same cut saves far more, but that is not measured here.