A failure with nothing behind it

A colleague was trying an idea in dbglab's random test on her laptop, saw a failure, and pasted it into the tracker before leaving for two weeks:

FAIL test=random seed=2 cycle=6964: SCOREBOARD: RAM word 186 (address 000012e8) read 58fab7f8, expected a0da7c74 (last written by DMA run #41)

You check out the main branch and run seed 2. It passes. The line names a test, a seed and a cycle. What else would you need to see this failure again? (The story is illustrative; every run in this lesson is real.)

What people usually do, and why it fails

The usual record of a failure is the FAIL line, maybe with a branch name. Then the hunt for the rest begins: which commit was the branch on that day, was anything edited and not committed, which simulator, which switches? Every question is answered from memory, and memory is where reproductions go to die. Here the answer was a one-line edit on the laptop that made long DMA runs common. Nothing in the tracker says so, and seed 2 passes without it.

The previous lesson listed the inputs that decide a run. This one makes sure they are written down, by a script, at the moment the run fails.

The method: a replay record for every failure

And the rules that make the record trustworthy:

1. Write it automatically, at the moment of failure. A wrapper around the run command, not a habit. Nobody fills in a form at 2 a.m.

2. Record facts, not names. A full commit hash, the version string the tool prints, the exact command line.

3. Save the full log next to it. The FAIL line is the signature; the log is the context.

4. Replay in a fresh checkout. Never in your working tree, which has moved on and may have its own edits.

5. Compare the signature. A replay passes only if it prints the same FAIL line. Any other result means something is missing.

The picture

Do it: record and replay on dbglab

The recorder

record-run.sh wraps dbglab's tb/run.sh. It passes the result through unchanged, and on a failure writes failures/<test>-<seed>-<sim>/replay.txt with a copy of the log. Keep it outside the repository, so checking out another commit does not change it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/bin/sh
# record-run.sh: run one dbglab test; on failure, write everything needed to replay it.
# usage: record-run.sh <test> <seed> [icarus|verilator] [plusargs...]   (run from the repository root)
TEST=$1 SEED=$2 SIM=${3:-icarus}
[ $# -gt 3 ] && shift 3 || shift $#
out=$(tb/run.sh "$TEST" "$SEED" "$SIM" "$@"); status=$?
echo "$out"
[ $status -eq 1 ] || exit $status                  # record failures only
dir=failures/$TEST-$SEED-$SIM
mkdir -p "$dir"
case $SIM in
  icarus)    ver=$(iverilog -V 2>&1 | head -1) ;;
  verilator) ver=$(verilator --version) ;;
esac
dirty=no
if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
  dirty=yes; git diff HEAD > "$dir/local.patch"     # uncommitted edits are part of the recipe
fi
{
  echo "when=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "commit=$(git rev-parse HEAD)"
  echo "dirty=$dirty"
  echo "cmd=tb/run.sh $TEST $SEED $SIM${*:+ $*}"
  echo "simulator=$ver"
  echo "env=DUMP=${DUMP:-} XRAND=${XRAND:-}"
  echo "host=$(uname -srm)"
  echo "result=$(echo "$out" | grep '^FAIL')"
} > "$dir/replay.txt"
cp "build/$SIM-$TEST-$SEED.log" "$dir/"
echo "recorded: $dir/replay.txt"
exit 1

It exits with the same status as tb/run.sh (0 pass, 1 fail, 125 no build), so it can sit inside a regression script or git bisect run unchanged.

Two failures, two records

First, seed 10 on a clean tree at HEAD. Then the colleague's case: the one-line edit that makes one DMA run in ten long instead of one in 400, and seed 2.

$ ../replay/record-run.sh random 10 > /dev/null
$ cat failures/random-10-icarus/replay.txt
when=2026-10-17T16:13:52Z
commit=49c5f04d1124aaeab1dc902884d49b7d1b4b8e75
dirty=no
cmd=tb/run.sh random 10 icarus
simulator=Icarus Verilog version 13.0 (stable) (v13_0)
env=DUMP= XRAND=
host=Darwin 25.6.0 arm64
result=FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)
$ git diff
...
-  else            r_len = (urange(0, 399) == 0) ? urange(257, 320) : urange(1, 24);  // 1 in 400 runs is long
+  else            r_len = (urange(0, 9) == 0) ? urange(257, 320) : urange(1, 24);  // 1 in 400 runs is long
...
$ ../replay/record-run.sh random 2
dbglab: test=random seed=2
FAIL test=random seed=2 cycle=6964: SCOREBOARD: RAM word 186 (address 000012e8) read 58fab7f8, expected a0da7c74 (last written by DMA run #41)
recorded: failures/random-2-icarus/replay.txt
$ grep -E 'dirty|cmd' failures/random-2-icarus/replay.txt; ls failures/random-2-icarus
dirty=yes
cmd=tb/run.sh random 2 icarus
icarus-random-2.log
local.patch
replay.txt

The second record says what the tracker entry did not: the tree was dirty, and local.patch holds the edit. (Its comment still says "1 in 400"; the patch records the code, not what the author meant.)

The replayer

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/bin/sh
# replay.sh: rebuild the recorded state in a fresh worktree, rerun, compare the result line.
# usage: replay.sh <dir>/replay.txt   (run from the repository root)
rec=$(cd "$(dirname "$1")" && pwd)/$(basename "$1")      # absolute: we cd into the worktree
get() { sed -n "s/^$1=//p" "$rec"; }
commit=$(get commit); cmd=$(get cmd); want=$(get result)
sim=$(echo "$cmd" | cut -d' ' -f4)
case $sim in
  icarus)    ver=$(iverilog -V 2>&1 | head -1) ;;
  verilator) ver=$(verilator --version) ;;
esac
[ "$ver" = "$(get simulator)" ] || echo "WARNING: simulator is '$ver', recorded '$(get simulator)'"
wt=$(mktemp -d)/replay
git worktree add -q --detach "$wt" "$commit" || exit 2
if [ "$(get dirty)" = yes ] && ! git -C "$wt" apply "$(dirname "$rec")/local.patch"; then
  git worktree remove --force "$wt"; echo "replay: local.patch does not apply"; exit 2   # never run without it
fi
got=$(cd "$wt" && env $(get env) $cmd | grep -E '^(PASS|FAIL)')
git worktree remove --force "$wt"
echo "recorded: $want"
echo "replayed: $got"
if [ "$got" = "$want" ]; then echo "REPRODUCED"; exit 0; else echo "DIFFERENT"; exit 1; fi

git worktree add gives a second checkout of the same repository at the recorded commit, without touching yours. The script applies the patch (and stops if it does not apply), runs the recorded command with the recorded environment, and compares the replay's PASS or FAIL line with the recorded one.

Replay from somewhere else

Now move on, as real life does: drop the local edit and check out an older commit, 12. Rerunning the commands by hand gives two passes. The records do better:

$ git checkout -q tb/tests.svh; git checkout -q bc2bb34
$ tb/run.sh random 10
dbglab: test=random seed=10
PASS test=random seed=10
$ tb/run.sh random 2
dbglab: test=random seed=2
PASS test=random seed=2
$ ../replay/replay.sh failures/random-10-icarus/replay.txt
recorded: FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)
replayed: FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)
REPRODUCED
$ ../replay/replay.sh failures/random-2-icarus/replay.txt
recorded: FAIL test=random seed=2 cycle=6964: SCOREBOARD: RAM word 186 (address 000012e8) read 58fab7f8, expected a0da7c74 (last written by DMA run #41)
replayed: FAIL test=random seed=2 cycle=6964: SCOREBOARD: RAM word 186 (address 000012e8) read 58fab7f8, expected a0da7c74 (last written by DMA run #41)
REPRODUCED

• The manual reruns passed because both inputs had changed: the commit (12 is before the bug) and, for seed 2, the missing edit.

• The replays rebuilt both: commit 13 or later from the hash, and the edit from local.patch. Same line, same cycle.

An input you cannot see: the environment

$ git checkout -q main
$ XRAND=1 ../replay/record-run.sh random 17 verilator
dbglab: test=random seed=17
FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)
recorded: failures/random-17-verilator/replay.txt
$ grep env failures/random-17-verilator/replay.txt
env=DUMP= XRAND=1
$ tb/run.sh random 17 verilator
dbglab: test=random seed=17
PASS test=random seed=17 cycles=18442 time=184416 ns
$ /usr/bin/time -p ../replay/replay.sh failures/random-17-verilator/replay.txt
recorded: FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)
replayed: FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)
REPRODUCED
real 8.93
...

The command line is identical in the failing and the passing run; only an environment variable differs. XRAND=1 makes dbglab's script build Verilator with random initial values. The record kept it, so the replay failed the same way. The 8.9 seconds are mostly Verilator compiling the fresh checkout.

What the evidence says

All three records replayed exactly, from a working tree that by then passed all three seeds. Each record needed a different field: seed 10 needed the commit, seed 2 the patch, seed 17 the environment. A record that drops any one of them would have turned a real bug into "cannot reproduce". Once the replay prints REPRODUCED, you are at the next step of the loop: the failure is on demand, and the work moves to making it small.

Traps

DIFFERENT can mean the replay is broken

The first version of replay.sh read the record and the patch by the relative paths it was given, after changing into the worktree, and it printed only a FAIL line. The patch failed to apply, so it ran seed 2 without the edit; that run passed, and the comparison reported a difference:

$ ../replay/replay.sh failures/random-2-icarus/replay.txt
error: can't open patch 'failures/random-2-icarus/local.patch': No such file or directory
sed: failures/random-2-icarus/replay.txt: No such file or directory
recorded: FAIL test=random seed=2 cycle=6964: SCOREBOARD: RAM word 186 (address 000012e8) read 58fab7f8, expected a0da7c74 (last written by DMA run #41)
replayed:
DIFFERENT

The error lines are the giveaway, not the empty "replayed" line: that version printed the same empty line for any passing replay. A replay tool that cannot read its own record must stop, not compare. The version above makes the paths absolute, exits when the patch does not apply, and prints the replay's PASS or FAIL line; run against this record without its patch, it now prints replay: local.patch does not apply and exits with status 2. Test the tool on a fresh record, straight after writing it, while you know it must reproduce. A replay tool that has never printed REPRODUCED has not been tested.

Related topics

• Part 1 · Reproduce

• Next: shrinking a failing test

• Determinism and seeds

• Bisection in history: the same exit codes, used by git bisect run

• UVM: regression triage