The nightly regression reports 100% passing. In the morning someone notices that dma_smoke_tst is on the list, a typo that has been there for a week, and that it passed every night. The test it names does not exist. How can a test that does not exist pass, and what should the regression script have checked?

Answering that needs the whole path from the command line to the test object. This lesson follows it, then adds typed knobs, and ends with what a failure report must contain so anyone can replay it.

The idea: the test is chosen by name, at run time

Compiling the bench takes minutes; running one test takes seconds. So a regression compiles once and runs the same binary many times, each time with a different test. That works because run_test creates the test by name: it asks the factory for a component whose registered type name matches a string, and gives it the instance name uvm_test_top. The string comes from the +UVM_TESTNAME= plusarg if there is one, and otherwise from the argument to run_test. The plusarg wins; run_test and timeouts is the home of that rule.

A theatre with a repertory company works this way. The stage, the lights and the actors are the same every night (the compiled binary). The poster outside (the plusarg) says which play is on. If the poster names a play the company does not know, the show is cancelled at the door, which is exactly what happens to a misspelled test.

The picture

Build it

Selecting the test

The bench's top calls run_test("dma_smoke_test"), so running the binary with no plusarg runs the smoke test. Everything else is a plusarg:

./simv +UVM_TESTNAME=dma_rand_test
./simv +UVM_TESTNAME=dma_vseq_test +UVM_VERBOSITY=UVM_MEDIUM
./simv +UVM_TESTNAME=dma_backpressure_test +UVM_VERBOSITY=UVM_MEDIUM

• The name must be the registered name, which uvm_component_utils(T) sets to the class name T. A class without the macro cannot be selected.

• Given twice, the first +UVM_TESTNAME is used. A library built with its DPI support also prints a MULTTST warning listing them all; the course's Verilator runs are built without DPI, so they use the first one silently.

• If an override for the test's own type is registered before run_test creates it, the RNTST line says via factory override for test "...". That is rare, but it explains a confusing log.

Typed knobs, parsed once

Test selection is one string. Scenario knobs, such as how many jobs dma_rand_test runs, need typed values. SystemVerilog's own $value$plusargs covers most cases; UVM's uvm_cmdline_processor adds lookups that return every match, which helps when a knob may be given several times. Either way, parse in one place, the base test's build_phase, before anything is built from the value, and log the effective values so the log alone can reproduce the run.

1
2
3
4
5
6
7
8
9
10

// snippet
// In the base test (or a derived test's build_phase), before the env is created.
function void read_knobs();
  uvm_cmdline_processor clp = uvm_cmdline_processor::get_inst();
  string val;
  void'($value$plusargs("DMA_JOBS=%d", n_jobs));          // plain SystemVerilog
  if (clp.get_arg_value("+DMA_MAX_LEN=", val))            // UVM's processor
    max_len = val.atoi();
  `uvm_info("KNOBS", $sformatf("n_jobs=%0d max_len=%0d", n_jobs, max_len), UVM_LOW)
endfunction

Two details save debugging time. $test$plusargs and $value$plusargs match by prefix: $test$plusargs("DMA_FAST") is also true for +DMA_FAST_RESET. Keep the = in value formats, and pick flag names that are not prefixes of each other. And knobs that change what gets built (agent modes, which scoreboard) must be read in build_phase; reading them in run_phase is too late, because the components were built from the old value.

Built-in plusargs worth knowing

The random seed is not a UVM plusarg: each simulator has its own switch (+ntb_random_seed= on VCS, -sv_seed on Questa, -svseed on Xcelium). Regression tools wrap it.

The replay bundle

A failure report is only useful if someone else can rerun it exactly. It must carry the test name, the seed, every plusarg, the build identity (commit or tag, and the UVM library version) and the simulator version. Logging the knobs in build_phase, as above, means the log itself carries most of the bundle.

Watch it run: the test that does not exist

Here is the real log of the bench run with the typo from the opening, trimmed:

UVM_WARNING @ 0: reporter [BDTYP] Cannot create a component of type 'dma_smoke_tst' because it is not registered with the factory.
UVM_FATAL @ 0: reporter [INVTST] Requested test from command line +UVM_TESTNAME=dma_smoke_tst not found.
--- UVM Report Summary ---
** Report counts by severity
UVM_INFO :    2
UVM_WARNING :    2
UVM_ERROR :    0
UVM_FATAL :    1

• BDTYP comes from the factory: no class is registered under that name.

• INVTST ends the run at time 0 and says where the bad name came from: the command line or the run_test argument.

• `UVM_ERROR : 0`. That line is the whole story of the opening.

What breaks: the regression that only counted errors

The team's pass criterion was a single grep: the log contains UVM_ERROR : 0. The typo'd test died at time 0 with a fatal, and the summary still printed zero errors, so the script marked it as passing, every night. Nobody saw it because a test that runs for zero seconds does not stand out in a regression of thousands.

The fix had three parts. Pass requires both UVM_ERROR : 0 and UVM_FATAL : 0. Pass also requires the line Running test <expected name>, which proves the intended test ran. And pass requires a positive sign of work, the scoreboard's words matched line with a non-zero count, because a test that compared nothing proves nothing. The third check also catches the test that ends at time 0 because it never raised an objection.

Next: the test is chosen; the stage it runs on is built by 40 lines nobody reviews. Top Module and run_test Startup reads them.

Related topics

• run_test and timeouts

• Command-line overrides

• Custom plusargs

• Top Module and run_test Startup