A new engineer joins the team and writes a regression script: for each test, compile the bench, then run it. With 2,000 runs a night and a two-minute compile, the regression spends almost three days compiling before it simulates anything. Meanwhile a merge on Tuesday breaks the APB driver, and nobody notices until Thursday's nightly, by which time six more changes have landed on top of it. Both problems have the same fix: the right structure around the bench. What does that structure look like?

The idea: build once, run many, and gate by tier

A UVM bench is compiled once into an executable that contains every test class. +UVM_TESTNAME picks which one to construct at run time, and the seed picks its random choices. So a regression is one build followed by any number of cheap runs, which is exactly how this course's bench already works. The second idea is tiers: not every test runs on every event. Small, fast, fixed-seed tiers run often and block bad changes; large random tiers run overnight and on weekends to find new bugs.

An airport works the same way. Every passenger goes through the quick security check (the smoke tier) before boarding, every time, with the same rules. Deeper customs inspections (the nightly) are run on a large random sample. Nobody proposes rebuilding the airport for each flight.

The picture: from a commit to a sign-off number

Build it: a tiered regression for the DMA bench

The test list

A test list names each test, its tier and how many seeds it gets. Keeping it in a file under version control means the regression's contents are reviewed like code, and a test can be promoted from nightly to smoke with a one-line change.

# tests.txt
# test                  tier     seeds
dma_smoke_test          smoke    1
dma_rand_test           smoke    1
dma_truncate_test       sanity   1
dma_reset_test          sanity   1
dma_backpressure_test   sanity   1
dma_check_test          nightly  5
dma_rand_test           nightly  50
dma_zero_len_test       nightly  1

The runner

The runner takes a tier and one compiled executable, runs every list entry at or below that tier, and writes a manifest: one line per run with its verdict, test, seed switch and log path. Smoke and sanity entries always use seeds 1 to N; nightly entries get fresh seeds from the clock, recorded in the manifest. Like the triage script, it treats a log with no summary as a failure.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#!/usr/bin/env bash
# regress.sh <tier> <simv> <outdir>: one build, many runs. Runs every list entry at or below <tier>.
set -u
tier=$1; simv=$2; out=$3; mkdir -p "$out"
rank() { case $1 in smoke) echo 0;; sanity) echo 1;; nightly) echo 2;; weekly) echo 3;; esac; }
base=$(date +%s)                                  # fresh nightly seeds every run, all recorded
grep -v '^#' tests.txt | while read -r test t n; do
  [ "$(rank "$t")" -le "$(rank "$tier")" ] || continue
  for i in $(seq 1 "$n"); do
    case $t in smoke|sanity) seed=$i;; *) seed=$((base + i));; esac   # fixed vs fresh
    log="$out/$test.$seed.log"
    "$simv" +UVM_TESTNAME="$test" +verilator+seed+"$seed" > "$log" 2>&1
    e=$(sed -n 's/^UVM_ERROR : *//p' "$log"); f=$(sed -n 's/^UVM_FATAL : *//p' "$log")
    [ "${e:-x}" = 0 ] && [ "${f:-x}" = 0 ] && v=PASS || v=FAIL     # no summary is a FAIL
    echo "$v $test +verilator+seed+$seed $log" >> "$out/manifest.txt"
  done
done

Watch it run

$ ./regress.sh smoke ./obj/Vtop runs/smoke
$ cat runs/smoke/manifest.txt
PASS dma_smoke_test +verilator+seed+1 runs/smoke/dma_smoke_test.1.log
PASS dma_rand_test +verilator+seed+1 runs/smoke/dma_rand_test.1.log

$ time ./regress.sh nightly ./obj/Vtop runs/nightly
real 5.04
$ head -7 runs/nightly/manifest.txt
PASS dma_smoke_test +verilator+seed+1 runs/nightly/dma_smoke_test.1.log
PASS dma_rand_test +verilator+seed+1 runs/nightly/dma_rand_test.1.log
PASS dma_truncate_test +verilator+seed+1 runs/nightly/dma_truncate_test.1.log
PASS dma_reset_test +verilator+seed+1 runs/nightly/dma_reset_test.1.log
PASS dma_backpressure_test +verilator+seed+1 runs/nightly/dma_backpressure_test.1.log
PASS dma_check_test +verilator+seed+1792213449 runs/nightly/dma_check_test.1792213449.log
PASS dma_check_test +verilator+seed+1792213450 runs/nightly/dma_check_test.1792213450.log

The numbers make the case for compile-once on their own. Building the bench took about 100 seconds of wall-clock time (the UVM library is most of it). The 61 nightly runs together took about 5 seconds. Compiling per test would have made the same nightly take well over an hour and a half. On a real SoC bench the build can take an hour and a single test several hours, so the ratio changes, but the rule does not: build once per configuration, then fan the runs out.

Farms: running thousands of jobs in parallel

A nightly of thousands of runs does not run on one machine. It is submitted to a compute farm through a job scheduler: IBM Spectrum LSF (bsub), Grid Engine (qsub), Slurm (sbatch), or a cloud batch service. Each run becomes one job: the scheduler finds a free machine with a simulator licence, runs the command from the manifest, and stores the log on shared storage. Three practical limits shape every farm regression:

• Licences, not CPUs, are usually the bottleneck. A simulator licence is checked out per running job, so the number of parallel runs is capped by the licence count. Short tests and early exits on the first fatal are worth real money.

• Wall-clock limits kill jobs silently. The scheduler ends a job that runs too long, with no UVM summary in the log. That is why the triage script treats a missing summary as a failure, and why a test needs its own timeout (set_timeout or +UVM_TIMEOUT) well below the farm's limit.

• Every job must be self-contained. Absolute paths to one engineer's home directory, files written to the current directory by every job at once, and environment variables set in someone's shell all break on the farm. Verilator's coverage-instrumented bench, for example, writes coverage.dat into the current directory by default; on a farm every run would overwrite the same file, so each run names its own with +verilator+coverage+file+<test>.<seed>.dat.

CI for DV: gate merges on the smoke tier

Continuous integration (CI) runs a defined job on every change. For a DV team the job is: build the bench, run the smoke tier, and report the result on the pull request. The repository then refuses the merge if the job failed. The gate must be fast (minutes, so engineers wait for it rather than bypass it), deterministic (fixed seeds) and owned (a broken gate is fixed first, before any other work). A sketch of such a job, in the style of a CI configuration file:

1
2
3
4
5
6
7
8
9

# sketch: CI job that gates merges on the smoke tier
on: pull_request
jobs:
  smoke:
    steps:
      - build:  ./build.sh                      # one compile of the bench
      - run:    ./regress.sh smoke ./obj/Vtop runs/smoke
      - check:  "! grep -q '^FAIL' runs/smoke/manifest.txt"
      - keep:   runs/smoke/                     # logs and manifest, for every run

The nightly is a scheduled job on the merged main branch, not a gate. Its failures go to triage, and a new signature in the nightly points at the merges since the last clean night.

Plan and regression management tools

Commercial flows wrap all of this in a verification management tool. Cadence's vManager (now part of its Verisium platform), Synopsys' Verdi Planner with its execution manager, and Siemens' Questa Verification Run Manager (VRM) all work at the same level: they read a test list or a regression description, launch runs on a farm, collect logs and coverage databases, group failures, and link the results to a verification plan, so that each plan item shows the tests and coverage that back it. The details and file formats differ by vendor; what they automate is exactly what the scripts on this page do by hand.

Artifacts: what to keep from every run

What breaks: the green gate that was not running

A team's merge gate stayed green for a month while the nightly kept finding basic failures. The gate's script ran the smoke tests and grepped their logs for UVM_ERROR lines; a refactor of the build had renamed the executable, so every run printed "command not found" and no log contained an error line. Each run "passed" in about a millisecond. The fix was the rule the runner above already follows: a run passes only when its log contains a summary with zero errors and zero fatals, and the gate also fails if the number of runs is not what the test list asks for.

Next: the runs are on the farm and a bucket needs debugging. Waveforms at Scale gets you the waves for one run without dumping all of them.

Related topics

• Up: Regression and Sign-off

• Heartbeats and watchdogs

• Factory and configuration