Dump Strategy

Part 2 · Observe · Working (W)

On this page

Ninety thousand cycles in, one bad word

A long soak run of dbglab fails after 90,623 cycles:

Terminal

$ tb/run.sh random 1 icarus +OPS=15000
dbglab: test=random seed=1
FAIL test=random seed=1 cycle=90623: SCOREBOARD: RAM word 256 (address 00001400) read 669136e1, expected ae007788 (last written by DMA run #591)

You need to see what DMA run #591 did. dbglab has a dump switch, DUMP=1, that dumps the whole bench from time zero. On this run that file is 36.8 MB, and a bigger design on a longer test would be gigabytes. What should you dump, and from when?

What people usually do, and why it fails

The common habit is $dumpvars(0, top) at time zero, often left on for every run of the regression "so the waves are there when we need them". It fails three ways. It is slow: every value change of every signal is written, all run long. It is big: disks fill and viewers take minutes to open the file. And it is still incomplete. A full dump feels like it holds everything, but in most simulators it holds no memories at all, which is often exactly what the question is about.

The opposite habit fails too: "dump the last 200 cycles before the failure." It is small and fast, but 200 is a guess. If the cause happened 217 cycles before the failure, the window shows the symptom and not the cause.

The method: scope and window come from the question

Dumping is part of the Observe step. Its job is to record exactly what the question needs, and nothing else:

Off by default. A dump is turned on per run, by a plusarg, for the run you are debugging. Regressions run without one. (The RTL course shows the plusarg pattern in controlled dumps and trigger capture.)
Scope from the question. Dump the module that owns the suspect value and its interfaces. A question about the DMA needs the DMA, not the peripheral's clock domain.
Window from the question. Open the dump at the event your question is about, such as the start of the suspect transaction, and close it at the failure.
Run it twice. A deterministic failure costs one cheap run without a dump to learn the failing cycle and the suspect, then one run with the right window.
Trade scope against time. Size is roughly scope times time. One signal can be dumped for the whole run; the whole design only for a short window.

The picture: two windows before the same failure

The last 260 cycles of the failing run. A window of the last 200 cycles opens 17 cycles after DMA run #591 started; a trigger on the run's start catches it.

Read it left to right. The run that matters starts at cycle 90406, when its counter loads 38 for a 294-word copy. The grey window, "200 cycles before the failure", opens at 90423 and misses that moment by 17 cycles. The accent window opens on the event itself.

Do it: measure, then dump what the question needs

A dump you can steer from the command line

dbglab's own +DUMP is all or nothing. A small module compiled next to the design gives you scope, a cycle window and an event trigger. It only reads the design:

dump_ctl.sv

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// dump_ctl: chooses what to dump and when, from the command line.
//   +DSCOPE=all|dut|dma   whole bench | dut's own signals, one level | the DMA engine and below
//   +DFROM=<c> +DTO=<c>   dump only from bus cycle c to cycle c (default: the whole run)
//   +DRUN=<n>             trigger: start the dump when DMA run n starts
`timescale 1ns/1ps
module dump_ctl;
  string scope;
  int unsigned from_c, to_c, run_n, runs = 0;
  bit on = 1'b0;
  initial begin
    if (!$value$plusargs("DSCOPE=%s", scope)) scope = "";
    if (!$value$plusargs("DFROM=%d", from_c)) from_c = 0;
    if (!$value$plusargs("DTO=%d", to_c))     to_c   = 32'hFFFF_FFFF;
    if (!$value$plusargs("DRUN=%d", run_n))   run_n  = 0;
    if (scope != "") begin
      $dumpfile("scoped.vcd");
      if      (scope == "all") $dumpvars(0, dbglab_tb);
      else if (scope == "dut") $dumpvars(1, dbglab_tb.dut);
      else if (scope == "dma") $dumpvars(0, dbglab_tb.dut.u_dma);
      if (from_c > 0 || run_n > 0) $dumpoff;     // header and one snapshot only, until the window opens
      else on = 1'b1;
    end
  end
  always @(posedge dbglab_tb.clk) begin
    if (dbglab_tb.dut.u_dma.start && !dbglab_tb.dut.u_dma.busy) runs++;
    if (scope != "" && !on && dbglab_tb.cycle >= from_c && (run_n == 0 || runs >= run_n)) begin
      $dumpon; on = 1'b1;
      $display("dump_ctl: dump on at cycle %0d", dbglab_tb.cycle);
    end
    if (on && dbglab_tb.cycle > to_c) begin
      $dumpoff; on = 1'b0; from_c = 32'hFFFF_FFFF;
      $display("dump_ctl: dump off at cycle %0d", dbglab_tb.cycle);
    end
  end
endmodule

What each choice costs

The same failing run, 90,623 cycles, with six dump choices. Times are the simulation alone, without compiling:

Dump	VCD size	Icarus run time
none	none	1.02 s
whole bench from time zero (`+DSCOPE=all`, same as dbglab's `+DUMP`)	36.8 MB	1.66 s
the DUT's own signals, one level (`dut`)	10.1 MB	1.20 s
the DMA engine and below (`dma`)	5.7 MB	1.16 s
whole bench, last 200 cycles (`all`, `+DFROM=90423`)	96.7 kB	1.08 s
DMA, from run #591's start (`dma`, `+DRUN=591`)	14.6 kB	1.07 s

Read the table two ways. Down the size column, scope and window each cut the file by an order of magnitude or more, and together they cut it by more than two thousand times. Down the time column, a full dump adds 63% to the run; a windowed dump adds about 5%, because the simulator still tracks the signals while the dump is off. Verilator, with tracing compiled in, gives the same picture: 0.07 s without a dump, 0.11 s with DUMP=1, and a 32.1 MB file.

These times are small because dbglab is small. The ratios are what carry over to a real block: a full dump of a large design over a long test is the difference between a run that finishes over lunch and one that finishes tomorrow.

Open the window at the right event

First run without a dump, to get the failing cycle. Then rerun with a window. Here is the tempting version, 200 cycles before the failure:

Terminal

$ fail=$(tb/run.sh random 1 icarus +OPS=15000 | sed -n 's/^FAIL.*cycle=\([0-9]*\).*/\1/p'); echo $fail
90623
$ ../probes/run-dump.sh random 1 +OPS=15000 +DSCOPE=all +DFROM=$((fail-200))
dump_ctl: dump on at cycle 90423
FAIL test=random seed=1 cycle=90623: SCOREBOARD: RAM word 256 (address 00001400) read 669136e1, expected ae007788 (last written by DMA run #591)
real 1.08
scoped.vcd 96737 bytes

It worked, and it is the wrong window. The failure message names DMA run #591, so the question is about that run, and you need its start, where the run's length is loaded. The debug loop lesson's DMA probe gives the cycles:

Terminal

$ ../probes/run-probe.sh ../probes/dma_probe.sv random 1 +OPS=15000 +RUN=591
dbglab: test=random seed=1
cycle 90406: DMA run #591 start  LEN=294 SRC=000017e4 DST=00001078  left_q=38
cycle 90578: DMA run #591 done   words written=38  dst_q=00001110
FAIL test=random seed=1 cycle=90623: SCOREBOARD: RAM word 256 (address 00001400) read 669136e1, expected ae007788 (last written by DMA run #591)

The run started at 90406, 17 cycles before the window opened. Opening the dump on the event itself, with the DMA's scope only, catches it and is smaller still:

Terminal

$ ../probes/run-dump.sh random 1 +OPS=15000 +DSCOPE=dma +DRUN=591
dump_ctl: dump on at cycle 90405
FAIL test=random seed=1 cycle=90623: SCOREBOARD: RAM word 256 (address 00001400) read 669136e1, expected ae007788 (last written by DMA run #591)
real 1.07
scoped.vcd 14584 bytes

This dump holds the start of run #591, the load of left_q with 38 for a 294-word copy, and every beat until the failure: the whole story of bug B3, in 14.6 kB.

One word, the whole run

Sometimes the question is about one storage location over a long time: was RAM word 256 ever written by run #591? That is the opposite trade: almost no scope, all of the time. The full dump cannot answer it, because it holds no memory contents (375 variables, and neither the RAM nor the bench's model). Icarus will dump one array word if you name it:

dump_word.sv

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
`timescale 1ns/1ps
module dump_word;
  initial begin
    $dumpfile("word.vcd");
    $dumpvars(0, dbglab_tb.dut.u_ram.mem[256]);
  end
endmodule

Terminal

$ ../probes/run-probe.sh ../probes/dump_word.sv random 1 +OPS=15000
dbglab: test=random seed=1
VCD info: dumpfile word.vcd opened for output.
VCD warning: array word dbglab_tb.dut.u_ram.mem[256] will conflict with an escaped identifier.
FAIL test=random seed=1 cycle=90623: SCOREBOARD: RAM word 256 (address 00001400) read 669136e1, expected ae007788 (last written by DMA run #591)
$ sed -n '/enddefinitions/,$p' word.vcd
...
#66000
b1000111011110101001011011000001 !
...
#811665000
b1100110100100010011011011100001 !
#906226000

Times are in picoseconds. The first value is the bench's preload at 66 ns. The last change is at 811,665 ns, cycle 81,167, and the value it wrote, 669136e1 in binary, is what the CPU read at the failure. So nothing wrote word 256 for more than 9,000 cycles before run #591 even started. The run that the model says wrote it never touched it. A 623-byte file, no extra run time, and a clean answer.

What the evidence says

The full dump was the most expensive choice and the only one that could not answer the one-word question. The cheapest useful dumps were the ones shaped by a question: the DMA from the start of the suspect run, or one word for the whole run. The 200-cycle window was cheap and plausible and missed the cause, and nothing in its output warned you. Only knowing when the suspect transaction began showed the gap.

For a failure that does not repeat, you cannot run twice. There you need a trigger that is armed before the event, or a ring buffer that keeps the last N cycles and is written out when a check fires. Those belong to Part 8: intermittent failures.

Traps

Interview

Asked: "A test fails after two hours of simulation. How do you get a waveform of the failure?"

Weak answer: I would turn on full dumping for the whole design and rerun it overnight, so I have everything when I open the waveform.

Strong answer: First I check the failure is deterministic: same seed, same cycle. Then I use the failure message to decide what the question is about, such as one transaction, and I dump only the scope that owns it, opening the dump at that transaction's start rather than a fixed number of cycles before the failure. A two-hour run without a dump tells me the cycle; a second run with a small window gives the waveform. If the question is about one memory location over the whole run, I dump just that word, or log writes to it. If it does not repeat, I arm a trigger or a ring buffer on the check that fails.

Follow-ups: (1) How would you find when the suspect transaction began? (2) What does a ring buffer cost in an emulator or on silicon?

Check yourself A colleague wants the regression to dump the whole design on every run, "to save a rerun". Using the numbers above, what would you tell them?

On this run a full dump made the simulation 63% slower and wrote 36.8 MB, for every run, pass or fail, to save one cheap rerun on the few that fail. It still would not hold the RAM. Keep dumps off, and make each failure print a rerun command with a window: the rerun of a deterministic failure costs one extra run of one test.

Check yourself Your question is "which master last wrote address 0x1400 before the failure at cycle 90,623?" Which dump do you choose, and why?

Not a window around the failure: the write could be thousands of cycles earlier, and here it was more than 9,000. Either dump the one RAM word for the whole run (623 bytes here), which shows when it changed but not who wrote it, or dump the bus master interfaces and nothing else for the whole run, or log every write to that address. The question asks who, so the bus interfaces or a write log answer it; the single word only answers when.

How this was checked

dbglab at HEAD 49c5f04, 2026-10-17. tb/run.sh random 1 icarus +OPS=15000 fails at cycle 90623 (RAM word 256, DMA run #591). Times are /usr/bin/time -p vvp -n build/dump.vvp +TEST=random +SEED=1 +OPS=15000 ... (Icarus Verilog 13.0, compile excluded), each run three times with the same result to 0.06 s; sizes from ls -l.

dump_ctl.sv compiled next to dbglab (iverilog -g2012 -Wno-timescale -I tb), run with +DSCOPE=all|dut|dma, +DFROM=90423 and +DRUN=591. Verilator 5.048: tb/run.sh random 1 verilator +OPS=15000 with and without DUMP=1, after a first run that compiled the model. dma_probe.sv (from the debug-loop lesson) gave run #591's start and done cycles; dump_word.sv dumped mem[256] alone.

$dumpvars(0, dbglab_tb) in Icarus wrote 375 variables and neither the RAM array nor the bench's model array.

Limits: one small design on one machine; absolute times are tiny and scale with design size, while the ratios are what carries over. Larger designs, other simulators and compressed waveform formats differ. Ring-buffer capture for failures that do not repeat is described, not run.

Navigate

Ninety thousand cycles in, one bad word

What people usually do, and why it fails

The method: scope and window come from the question

The picture: two windows before the same failure

Do it: measure, then dump what the question needs

A dump you can steer from the command line

What each choice costs

Open the window at the right event

One word, the whole run

What the evidence says

Traps

Related topics

Dump Strategy

Ninety thousand cycles in, one bad word

What people usually do, and why it fails

The method: scope and window come from the question

The picture: two windows before the same failure

Do it: measure, then dump what the question needs

A dump you can steer from the command line

What each choice costs

Open the window at the right event

One word, the whole run

What the evidence says

Traps

Related topics

Ninety thousand cycles in, one bad word

What people usually do, and why it fails

The method: scope and window come from the question

The picture: two windows before the same failure

Do it: measure, then dump what the question needs

A dump you can steer from the command line

What each choice costs

Open the window at the right event

One word, the whole run

What the evidence says

Traps

Related topics

Up next

Ninety thousand cycles in, one bad word

What people usually do, and why it fails

The method: scope and window come from the question

The picture: two windows before the same failure

Do it: measure, then dump what the question needs

A dump you can steer from the command line

What each choice costs

Open the window at the right event

One word, the whole run

What the evidence says

Traps

Related topics

Up next