The nightly sweep of dbglab's random test, seeds 1 to 60, has 19 failures. Ten read a wrong real value from RAM; nine read or deliver an unknown value. The random test mixes ten kinds of traffic: CPU writes and reads of RAM, register writes, words pushed into the peripheral, idle gaps, and five kinds of DMA run. Shrinking one seed gave one tiny test. But which kind of traffic triggers each failure, and are these two bugs, or five?
The usual move is to shrink or debug the first failing seed and assume the other eighteen are the same bug. Or to read the failure messages and guess from the words: "last written by DMA run" appears in all ten value failures, so "the DMA is broken", which is true and useless, because the DMA has a read path, a write path, a counter, a fill mode and a paced mode. A shrunk test for one seed shows one path that fails; it does not show which paths never fail, and that is where localisation comes from.
The stimulus is a space you can halve, like time or the hierarchy. The useful axis is not the position of an operation in the list (Part 1 did that) but its class: who issued it and what kind it is.
Record the stimulus as a list that the bench can replay exactly (recording everything to replay); check that the full list fails exactly like the original.
Tag every operation with a class. Start with the source (CPU or DMA), then the kind (read, write, copy, fill, a run into the peripheral, an error run). Setup that every test needs (the first control write, the polls that close a run) is kept in every candidate.
Halve the classes. Keep half of them and drop every operation of the others; replay. Same failure: the dropped half is not needed. Pass or a different failure: some of it is. Part 1's ddmin does exactly this when you hand it class names instead of operations.
Make dropping honest. Removing a class must leave the kept operations as they were. If the replay skips a DMA start because the previous run has not finished, dropping CPU traffic (and the time it took) drops DMA runs too. The replay must wait, not skip.
Split further by an attribute once a class is found: length, address range, mode bits.
Do it for every failing seed and group the seeds by the classes they need.
Read it left to right. Two signatures lead to three trigger classes, and the three classes lead to two shared conditions. Not one failing seed needed any CPU traffic.
The script tags each recorded line, then hands the list of class names to Part 1's ddmin.py. A candidate keeps every operation of the classes it names; the ie line (the first control write) and the poll lines are always kept, and a fill line travels with the DMA run after it.
# not-run: drives the scratch replay bench through vvp and imports Part 1's ddmin.py (command in the Evidence box)
"""split_classes.py: which classes of stimulus does a failure need?
usage: python3 split_classes.py <seed> <ops file> <signature regex> (run where build/dbglab.vvp is)
Tags every recorded operation with a class, then runs ddmin over the classes, not the operations:
a candidate keeps every operation of the classes it names and drops the rest."""
import re, subprocess, sys
from ddmin import ddmin
DMA_KIND = [(12, "dma-periph"), (24, "dma-err"), (25, "dma-zero"), (33, "dma-pattern"), (100, "dma-copy")]
CPU = {"wr": "cpu-ram-wr", "rd": "cpu-ram-rd", "scratch": "cpu-reg", "ctrl": "cpu-reg",
"per": "cpu-per", "idle": "idle"}
def tag(lines):
"""[(class, line)]; 'ie' and 'poll' are always kept, a 'fill' goes with the DMA run after it."""
out, pending = [], []
for line in lines:
op = line.split()[0]
if op == "fill":
pending.append(line)
continue
if op == "dma":
k = int(line.split()[1])
cls = next(name for top, name in DMA_KIND if k < top)
out += [(cls, l) for l in pending + [line]]
pending = []
else:
out.append((CPU.get(op, "keep"), line))
return out
seed, src, sig = sys.argv[1], sys.argv[2], re.compile(sys.argv[3])
tagged = tag(open(src).read().splitlines())
classes = sorted({c for c, _ in tagged if c != "keep"})
runs, seen = 0, {}
def fails(keep):
global runs
key = tuple(sorted(keep))
if key not in seen:
runs += 1
with open("cand.ops", "w") as f:
f.write("\n".join(l for c, l in tagged if c == "keep" or c in keep) + "\n")
out = subprocess.run(["vvp", "-n", "build/dbglab.vvp", "+TEST=replay", f"+SEED={seed}", "+STIM=cand.ops"],
capture_output=True, text=True).stdout
line = next((l for l in out.splitlines() if l.startswith("FAIL")), "PASS")
seen[key] = bool(sig.search(line))
print(f"run {runs:2d}: keep {', '.join(key):60s} {'same failure' if seen[key] else line[:60]}")
return seen[key]
assert fails(classes), "the full list must fail with the signature"
need = ddmin(classes, fails, log=lambda m: None)
print(f"seed {seed}: needs {', '.join(need)} ({len(classes)} classes, {runs} runs)")The first number in a dma line is the random test's draw that picks the kind of run (below 12 into the peripheral, 12 to 23 an error run, 24 a zero fill, 25 to 32 a pattern fill, the rest a RAM copy), so the class comes straight from the recording.
$ python3 ../probes/split_classes.py 10 seed10.ops 'SCOREBOARD.*(read|holds) [0-9a-f]{8}, expected [0-9a-f]{8} \(last written by DMA run'
run 1: keep cpu-per, cpu-ram-rd, cpu-ram-wr, cpu-reg, dma-copy, dma-err, dma-pattern, dma-periph, idle same failure
run 2: keep cpu-per, cpu-ram-rd, cpu-ram-wr, cpu-reg PASS
run 3: keep dma-copy, dma-err, dma-pattern, dma-periph, idle same failure
run 4: keep dma-copy, dma-err same failure
run 5: keep dma-copy same failure
seed 10: needs dma-copy (9 classes, 5 runs)Read it line by line. Seed 10 used nine of the ten classes. Run 1 is the full list. The class names sort with the CPU's first, so the first cut is CPU against DMA: CPU traffic alone passes (run 2), DMA traffic with the idle gaps fails (run 3). Two more halvings and the answer is one class: RAM-to-RAM DMA copies. Five replays, under a second.
A small shell loop picks the signature from each seed's failure (an unknown value, or a wrong real value left by a DMA run) and runs the splitter:
$ ../probes/split_all.sh ../sweep60.txt
value seed 10: needs dma-copy (9 classes, 5 runs)
value seed 13: needs dma-copy (8 classes, 5 runs)
X seed 16: needs dma-zero (8 classes, 6 runs)
X seed 17: needs dma-zero (10 classes, 8 runs)
X seed 19: needs dma-zero (7 classes, 6 runs)
value seed 20: needs dma-copy (10 classes, 8 runs)
X seed 23: needs dma-zero (9 classes, 8 runs)
X seed 29: needs dma-zero (9 classes, 8 runs)
X seed 31: needs dma-zero (10 classes, 8 runs)
value seed 34: needs dma-pattern (10 classes, 6 runs)
X seed 41: needs dma-zero (9 classes, 8 runs)
X seed 43: needs dma-zero (10 classes, 8 runs)
value seed 46: needs dma-copy (8 classes, 5 runs)
value seed 48: needs dma-copy (10 classes, 8 runs)
value seed 50: needs dma-copy (9 classes, 5 runs)
X seed 51: needs dma-zero (7 classes, 6 runs)
value seed 52: needs dma-pattern (10 classes, 6 runs)
value seed 58: needs dma-copy (10 classes, 8 runs)
value seed 59: needs dma-copy (10 classes, 8 runs)Nineteen seeds, 130 replays, under seven seconds. Every unknown-value seed needs a zero fill and nothing else. Eight value seeds need a copy, and two, seeds 34 and 52, need a pattern fill. A pattern fill reads nothing from RAM, so whatever breaks those two is not on the DMA's read path.
What does a pattern fill share with a copy? Both have a length. The random test makes one run in 400 longer than 256 words, so split the DMA runs at 255 words and replay each half with everything else kept:
$ awk '$1!="dma" || $3<=255' seed10.ops > s10-short.ops; awk '$1!="dma" || $3>255' seed10.ops > s10-long.ops
...
$ for s in 10 34 52; do for half in short long; do printf "seed %s %s: " $s $half; vvp -n build/dbglab.vvp +TEST=replay +SEED=$s +STIM=s$s-$half.ops | grep -E "^(PASS|FAIL)"; done; done
seed 10 short: PASS test=replay seed=10 cycles=4499 time=44986 ns
seed 10 long: FAIL test=replay seed=10 cycle=1906: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #1)
seed 34 short: PASS test=replay seed=34 cycles=11390 time=113896 ns
seed 34 long: FAIL test=replay seed=34 cycle=4507: SCOREBOARD: RAM word 830 (address 00001cf8) read 9ef7e786, expected d2dc7f79 (last written by DMA run #1)
seed 52 short: PASS test=replay seed=52 cycles=3677 time=36766 ns
seed 52 long: FAIL test=replay seed=52 cycle=1558: SCOREBOARD: RAM word 430 (address 000016b8) read 6d37a6cb, expected 10fdb14a (last written by DMA run #1)In each seed the long runs alone fail and the short runs alone pass. The recorded lists say what the long runs are: a 308-word copy in seed 10, a 273-word pattern fill in seed 34 and a 291-word pattern fill in seed 52.
Finding | What it rules out | Where it points |
|---|---|---|
CPU-only traffic never fails | the CPU port, the CSR read-back, the peripheral path for CPU words | the DMA |
Value failures need a copy or a pattern fill | the DMA's read path (fills do not read) | logic copies and fills share |
Only runs over 255 words fail | addresses, modes, back-pressure | the length: its register, its counter |
Unknown values need only a zero fill | copies, pattern fills, the peripheral | what a zero fill uses that a pattern fill sets first: FILL |
Two bugs, two places, and no waveform yet. The value bug lives where a length over 255 is handled; the first wrong cycle walks one of these seeds back to the DMA's beat counter. The unknown-value bug lives where a zero fill gets its value; bisection in time finds seed 23's first unknown word written by a zero fill whose fill value was already unknown. And because the split was done per seed, you also know the grouping is honest: all ten value seeds behave alike, and so do all nine unknown ones.
The first version of the replay bench, from Part 1, skipped a recorded DMA start if the previous run was still busy. In the full list that never happens: the recording only writes a start when the DMA is idle. In a split it happens all the time. Here is seed 10 on that bench:
$ python3 ../probes/split_classes.py 10 seed10.ops 'SCOREBOARD.*(read|holds) [0-9a-f]{8}, expected [0-9a-f]{8} \(last written by DMA run'
run 1: keep cpu-per, cpu-ram-rd, cpu-ram-wr, cpu-reg, dma-copy, dma-err, dma-pattern, dma-periph, idle same failure
run 2: keep cpu-per, cpu-ram-rd, cpu-ram-wr, cpu-reg PASS
run 3: keep dma-copy, dma-err, dma-pattern, dma-periph, idle PASS
...
run 13: keep dma-copy PASS
...
run 17: keep cpu-reg, dma-copy same failure
seed 10: needs cpu-reg, dma-copy (9 classes, 17 runs)It claims the bug needs register writes. It does not: with the CPU's traffic gone, nothing gave each copy time to finish before the next start, and of the 21 copies in the copy-only list only 9 started, the longest 23 words. The 308-word copy was silently skipped. Register writes "mattered" only because they took time. The fix is two lines in the replay test: wait for the running DMA instead of skipping.
// snippet: the scratch replay test after the change; before it, both lines were "if (!r_active)" skips
end else if (op == "fill") begin
n = $fscanf(fd, "%h", v);
while (r_active) r_poll_dma(); // split: wait for the run, never drop the write
csr_wr(CSR_FILL, v);
end else if (op == "dma") begin
n = $fscanf(fd, "%d %d %d %d %h", a, b, c, d, v);
while (r_active) r_poll_dma(); // split: wait for the run, never drop a startAfter it, every one of the 19 full lists still replays to its identical FAIL line, and the split gives the honest answer in 5 runs.
Fill mode is what seed 34 happened to use; eight other seeds with the same signature need copies instead. The conclusion should be about what copies and pattern fills share. Splitting by an attribute settles it: long runs alone fail and short runs alone pass, in both kinds, so the bug is in length handling, not in fill mode.
That the bug is in an interaction: something the two sources share or contend for, such as the peripheral's FIFO, its flow control, or the bus arbitration between them. Neither path is wrong on its own. Look at the logic both reach, and make the next split by an attribute of the interaction, such as whether the two overlap in time.
dbglab at HEAD 49c5f04, Icarus Verilog 13.0, Python 3.14.3, 2026-10-17, on a scratch copy of Part 1's replay bench (the random test records its operations with +REC=<file>; the replay test runs a list). One change for this lesson: a dma or fill line waits for the running DMA instead of being skipped. All 19 seeds of 1 to 60 that fail random were recorded, and each full list replays to the identical FAIL line.
python3 split_classes.py 10 seed10.ops '<signature>' (ddmin from Part 1 over classes): 5 runs, needs dma-copy. With the old skipping replay: 17 runs, needs cpu-reg and dma-copy; its copy-only list started 9 of its 21 DMA runs, the longest 23 words. split_all.sh over the 19 seeds: 6.7 s, output as shown. Length split (awk on the lists, replayed): seeds 10, 34 and 52 pass with runs of 255 words or fewer only, and fail with the long runs only.
Limits: the replay bench is a scratch extension, not part of dbglab. Ten classes chosen by hand; a different split could need more runs. The split shows what triggers a failure, not why the design fails.