Test list times seeds

A regression is a matrix: a test list of which tests run with which knobs, crossed with seeds that select one random stream per run. The dispatcher expands the matrix into jobs, farms them out in parallel, then scrapes each log for its banner and exit code.

Everything earlier in this topic exists so this flow runs without humans: banners, seeds in headers, knobs, and grep-able logs.

test list                seeds
---------                -----
smoke_test     x1        random per run,
burst_test     x20       recorded in log
err_inject     x10
soak_test      x5
      |
      v
DISPATCH: N jobs in parallel
  job = (test, seed, knobs) -> sim -> run.log
      |
      v
AGGREGATE per job:
  PASS banner + exit 0     -> pass
  FAIL banner / exit != 0  -> fail + first-problem signature
  no banner / timeout      -> infra trouble (counts as fail)
      |
      v
TRIAGE: bucket fails by first-problem signature
RERUN failures with logged seeds -> reproduce -> debug -> fix
      repeat until green

Aggregate and bucket

Thirty failing runs rarely mean thirty bugs. The aggregator takes each run's first-problem line, masks volatile fields such as time, addresses and ids, and clusters identical signatures into buckets. A run passes only with a PASS banner and exit status 0, which the dispatcher saved next to the log.

Each bucket is one suspected bug. Engineers debug one representative run per bucket, not all thirty.

# Aggregate: one verdict line per run. The dispatcher wrote each
# run's exit status to exit_code next to its run.log.
for dir in results/*; do
  log="$dir/run.log"
  if grep -q "^PASS:" "$log" && [ "$(cat "$dir/exit_code")" = "0" ]; then
    echo "PASS  $log"
  else
    # first-problem signature; time, numbers and ids masked for clustering
    sig=$(grep -m1 -E "LOG_ERROR|^FAIL:" "$log" |
          sed 's/^\[[0-9]*\]/[T]/; s/0x[0-9a-fA-F]*/0xN/g; s/#[0-9]*/#N/g')
    if [ -z "$sig" ]; then sig="NO_BANNER_OR_CRASH"; fi
    echo "FAIL  $log  $sig"
  fi
done | tee summary.txt

# Triage buckets: cluster failures by signature
grep "^FAIL" summary.txt | cut -d' ' -f4- | sort | uniq -c | sort -rn

The masking turns thirty runs with different addresses into one bucket line. The representative seed for each bucket sits in its log header, ready for replay.

Tiers and the rerun loop

Not every change deserves the full matrix. Healthy projects layer regressions like a pyramid: tiny and fast at the commit gate, broad and slow at night, exhaustive before milestones.

The wide base runs constantly, so it must be fast and rock-stable. A flaky gate trains people to bypass it, which is worse than no gate at all.

Coverage merge belongs to the nightly and milestone tiers. Single-run coverage means little; merged sweeps show closure.

1. Aggregate the overnight run and cluster failures into signature buckets.

2. Pick one representative test and seed per bucket from the log headers.

3. Replay it with verbosity up and waves on, then debug and fix the RTL or the testbench.

4. Rerun just the failing pairs to confirm each fix.

5. Rerun the full matrix, since fixes can unmask new failures downstream.

Common mistakes