A nightly random regression of the whole system fails a dozen seeds in sixty with a corrupted RAM word, thousands of cycles in. At full-chip scale that run would take hours, its waveform would be gigabytes, and every experiment would wait for the next night. The team starts adding dumps to it.
The bug is in one block, and it needs one condition to show. Why debug it in the most expensive place it can be seen?
They debug where the failure was found, because that is where the repro is and moving it feels like extra work. Every rerun then costs a full regression slot, every probe a rebuild of the whole system, and the waveform is too big to read with a question in mind. The opposite mistake is to jump straight to a unit test of the suspect block, with the unit test's usual stimulus, and conclude from its pass that the block is fine. That cheaper stage did not contain the ingredient the bug needs.
Name the failure's ingredients. From the first wrong thing, not the symptom: which block, which input condition, which timing, which software, which initial state. For a silicon failure, add the things simulation lacks: real timing, real power-up values, analogue effects.
List what each stage contains. A block bench has the block and any stimulus you like; a system simulation adds the other blocks and real traffic; gate-level simulation adds cells and, with a delay file, timing; emulation and FPGA add speed and real software; silicon adds physics. Each step up costs time per run and visibility.
Move the failure to the cheapest stage that contains every ingredient. Write the smallest test there that supplies them.
Prove the move. The failure must show at the new stage with the same signature, or a signature the hypothesis predicts. A different symptom may be a different bug.
Use the cheap stage's power. Sweep the space the expensive stage only sampled: every length, every mode, every corner. This is where the extent of a bug is found.
Prove the fix at both ends. At the cheap stage (the whole sweep) and at the stage where it was found (the original seed or test).
Stage | Contains | Cost per run at full-chip scale (illustrative) | Visibility |
|---|---|---|---|
Block simulation | one block, any stimulus | seconds | every signal |
System RTL simulation, directed | all blocks, chosen traffic | minutes | every signal |
System RTL simulation, random regression | all blocks, random traffic | hours per seed set | every signal, huge dumps |
Formal on a block | all inputs at once, bounded depth | minutes to hours | a counterexample trace |
Gate-level simulation | cells, timing with a delay file | hours to days | every net, slowly |
Emulation or FPGA | real software, long runs | minutes per run, days to build | triggers and trace windows |
Silicon | physics, real environment | hours per lab experiment | pass or fail, registers, dumps |
Formal, gate-level and emulation each have their own lesson; the numbers in the cost column are orders of magnitude, not measurements, and the cost of a bug by stage explains why they grow.
On the debug loop this is Reproduce in the service of Localise: a better repro, at a better stage, makes every later step cheaper.
$ for s in $(seq 1 60); do tb/run.sh random $s; done | grep -c '^FAIL'
19
$ /usr/bin/time -p tb/run.sh random 10
dbglab: test=random seed=10
FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)
real 0.09
...Nineteen of sixty seeds fail at HEAD, from two different bugs, and seed 10 fails 4,895 cycles in. The debug loop already found the first wrong thing with a probe: DMA run #29 was asked for 308 words and wrote 52, which is 308 minus 256. So the ingredients are one DMA run longer than 255 words. Nothing else: no particular traffic, no peripheral, no timing. The random test makes such a run once in 400, which is why only a few seeds see it.
one_copy_tb instantiates all of dbglab_top, preloads the RAM, programs one copy of +LEN words through the CPU port, waits for the DMA and compares the destination.
$ ./one.sh
one copy of 300 words: 256 words wrong, DMA done at cycle 370
$ ./one.sh +LEN=200
one copy of 200 words: 0 words wrong, DMA done at cycle 1618The signature survived the move: 300 words asked, 44 copied (300 minus 256), so 256 wrong, and the DMA reports done early, at cycle 370 instead of 4,895 cycles into a random run. A 200-word copy is clean. The failure needs no other traffic, so the next move can drop the rest of the system.
Reducing the design takes this step in full: dma_engine alone with an ideal memory, the same signature, and a sweep of every length in a second and a half. The sweep is the payoff. Every length from 256 up is wrong and none below; the random test samples 257 to 320 once in 400 runs, and dma_copy never goes above 64. And LEN 256, a length no dbglab test produces, shows a second symptom from the same line: the right number of words, 4 of them wrong. A similar bench here, dma_unit_tb (the engine alone, every length from +LO to +HI, default 0 to 1023), gives the same answers: LEN 300 writes 44 words in 323 cycles, and 768 lengths are wrong, the first 256. A formal check of the counter alone (formal counterexamples as traces) is cheaper still and finds the lengths above 256, but it models the counter and not the FIFO, so it cannot see LEN 256. Each stage sees what it contains.
$ iverilog -g2012 -o uf.vvp ../fixed3/rtl/dbglab_pkg.sv ../fixed3/rtl/sync_fifo.sv ../fixed3/rtl/dma_engine.sv dma_unit_tb.sv
...
$ /usr/bin/time -p vvp -n uf.vvp
lengths 0..1023: none wrong; 3800833 cycles
...
real 8.82
$ D=../fixed3 ./one.sh
one copy of 300 words: 0 words wrong, DMA done at cycle 2418With the counter widened to 10 bits (localparam int CW = 10;, in a scratch copy), every length is right and the system test is clean; the sweep now takes 3.8 million cycles instead of 955,582, because every copy runs to its full length. The last proof is the original seed at the original stage, which Part 9 covers along with the seed sweep.
Two moves took the failure from the 29th DMA run of a random seed to a test of one block with one input, and kept its signature at every step. The cheap stage then answered the question the expensive one never could: for which inputs is this wrong? All lengths from 256 up, with two different symptoms. That answer is the test to keep: the block-level sweep belongs in the regression, where it runs in seconds and covers what the random test only samples.
A block bench with the usual stimulus is cheaper still, and it passes: here dma_unit_tb stopped at 255, like a unit test written from the common case.
$ D=../dbglab; iverilog -g2012 -o u.vvp $D/rtl/dbglab_pkg.sv $D/rtl/sync_fifo.sv $D/rtl/dma_engine.sv dma_unit_tb.sv
...
$ vvp -n u.vvp +HI=255
lengths 0..255: none wrong; 237505 cyclesEvery length up to 255 is right, so a unit test that stops there says the block is fine. It is the right stage with the wrong stimulus: the ingredient, a run longer than 255 words, was left behind. The same trap at other stages: a two-state simulator that starts flip-flops at 0 hides a bug that needs X or a random power-up value, and a zero-delay simulation hides one that needs timing (timing-only bugs).
A failure that needs real timing, power-up values or the board cannot be shown in RTL simulation at any price. Its cheapest stage may be a model of the mechanism (when simulation cannot reproduce it, marginal failures) or a replay of a silicon dump (the first failing test on silicon). Moving it there is still moving it down.
The ingredient was not in the input you drove: something else at system level matters, such as back-pressure, a configuration register or a second master; check by moving one stage up (a directed system test) and removing parts until it stops failing. The block bench differs from the system: an ideal memory, different timing of ready, a different reset; check the bench's models against the system's. Or the bug is not in that block: the first wrong thing was misread; check by probing the block's inputs and outputs in the failing system run.
RTL simulation has no delays, so a temperature-dependent failure (usually timing) cannot appear there, at any seed. The cheapest stage is one with timing in it: static timing at the hot corner to find the path, a timing model of the suspect path, or gate-level simulation with the hot delay file on a short directed test. Moving it there is still moving it down from the lab, and every experiment becomes repeatable.
dbglab at HEAD 49c5f04, Icarus Verilog 13.0, 2026-10-18. /usr/bin/time -p tb/run.sh random 10: FAIL at cycle 4895, real 0.09 s; tb/run.sh random <s> for s = 1 to 60: 19 FAIL lines (bugs B3 and B4 together at HEAD), 12.7 s in all. one_copy_tb.sv (dbglab_top, one directed copy): 300 words, 256 wrong, DMA done at cycle 370; 200 words, 0 wrong. dma_unit_tb.sv (dma_engine alone): LEN 300 writes 44 words in 323 cycles; LEN 0 to 255 none wrong; LEN 0 to 1023: 768 wrong, first 256, 955,582 cycles, 2.20 s; LEN 256 writes 256 words with 4 wrong; a probe (chunk_probe.sv) counted 8 reads in its first chunk and 4 pushes into a full FIFO.
Fix check: with localparam int CW = 10; in a copy of dma_engine.sv, dma_unit_tb gives none wrong for 0 to 1023 (8.82 s) and one_copy_tb 0 wrong. The two harnesses are not part of dbglab; they are compiled with its RTL as the commands show.
Limits: dbglab is small, so every stage here runs in seconds and the differences are in cycles and in what each stage can ask, not in hours. The cost of each stage at full-chip scale in the table is illustrative. No formal, gate-level, emulation or silicon run is shown here; those stages have their own lessons.