A regression script decided pass or fail by counting lines that start with UVM_ERROR in each log. It worked for two years. Then the farm started killing jobs that ran out of memory, and a killed simulation writes no errors at all. For a week, every test that crashed was reported as passed, because a log with no error lines looked exactly like a clean one.

This lesson builds the opposite: one verdict line that a run prints on every way out, plus a machine-readable summary, so the regression keys off a positive statement of PASS and treats its absence as a failure. It also covers the report server's other controls, actions, the quit count and log files, at the level a senior engineer is expected to know them.

The idea: the referee's scorecard

Every message, from every component, ends at one object: the global report server. It is the referee holding the only scorecard. It counts messages by severity and by ID, applies the quit count, and at the end calls report_summarize(), which prints the familiar summary. Because there is only one server and it sees everything, it is the one place where a verdict can be computed correctly.

The library calls report_summarize() on two paths: at the end of run_test() after the report phases, and from uvm_root::die(), which runs on a UVM_FATAL, on the quit count, and on any message with the UVM_EXIT action, just before $finish. A line printed there survives a fatal. A banner printed in report_phase does not, because a fatal skips the remaining phases.

The picture: three ways a run ends

Build it: the bench's verdict server

dma_report_server extends uvm_default_report_server and adds one line after the standard summary:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// snippet: from the course DMA bench
// A report server that always ends the log with one verdict line, even when the
// run stops on a fatal (the library calls report_summarize on the way out).
class dma_report_server extends uvm_default_report_server;
  function new(string name = "dma_report_server");
    super.new(name);
  endfunction

  virtual function void report_summarize(UVM_FILE file = 0);
    int n_err = get_severity_count(UVM_ERROR);
    int n_fat = get_severity_count(UVM_FATAL);
    super.report_summarize(file);
    $display("TEST VERDICT: %s (UVM_ERROR=%0d, UVM_FATAL=%0d)",
             (n_err == 0 && n_fat == 0) ? "PASS" : "FAIL", n_err, n_fat);
  endfunction
endclass

install_verdict_server() makes it the global server. uvm_report_server::set_server() copies the counts from the server it replaces, so messages reported before the install are not lost. The phase-7 tests call it in their constructor, the earliest point a test can run code:

1
2
3
4
5

// snippet: from dma_pkg in the course DMA bench (dma_tb.sv)
function automatic void install_verdict_server();
  dma_report_server srv = new();
  uvm_report_server::set_server(srv);
endfunction

1
2
3
4
5

// snippet: from dma_verdict_test in the course DMA bench (dma_tb.sv)
function new(string name, uvm_component parent);
  super.new(name, parent);
  install_verdict_server();
endfunction

Watch it run: PASS, and FAIL after a fatal

+UVM_TESTNAME=dma_verdict_test
UVM_INFO dma_tb.sv(411) @ 655000: uvm_test_top.env.sb [SB] 8 words matched, 0 mismatched
UVM_ERROR :    0
UVM_FATAL :    0
TEST VERDICT: PASS (UVM_ERROR=0, UVM_FATAL=0)

+UVM_TESTNAME=dma_verdict_fatal_test
UVM_FATAL base/uvm_phase_hopper.svh(598) @ 20000000: reporter [PH_TIMEOUT] Explicit timeout of 20000000 hit, indicating a probable testbench issue
UVM_ERROR :    0
UVM_FATAL :    1
TEST VERDICT: FAIL (UVM_ERROR=0, UVM_FATAL=1)

• The passing run ends with the standard counts and then the verdict line.

• The fatal run never reached report_phase; the timeout fatal went through die(), which still called report_summarize(), so the verdict line is there and says FAIL.

Actions: what the server does with a message

Each message carries an action, a bit mask the component's report handler looked up from its severity and ID. The server executes it:

The server counts every message it executes, whatever the action, which is why warnings appear in the summary counts. UVM_COUNT is only about the quit count. UVM_STOP is useful interactively: set it on one ID with +uvm_set_action=uvm_test_top.env.sb,SB_MISMATCH,UVM_ERROR,UVM_DISPLAY|UVM_STOP and the simulator stops at the first mismatch with the waveform at the right time.

The quit count: stop after N

set_max_quit_count(n) on the server (or +UVM_MAX_QUIT_COUNT=n,NO, where NO forbids later overrides) ends the run after n messages with UVM_COUNT. The quit count goes through die(), so the verdict line appears. The scratch run caps a run that would report six mismatches at three, and also routes that ID to its own log file:

1
2
3
4
5
6

// snippet: scratch run (sync_tb.sv), quit_test's end_of_elaboration_phase
uvm_report_server srv = uvm_report_server::get_server();
srv.set_max_quit_count(3);
sb_log = $fopen("sb_mismatch.log", "w");
set_report_id_file("SB_MISMATCH", sb_log);
set_report_id_action("SB_MISMATCH", UVM_DISPLAY | UVM_LOG | UVM_COUNT);

scratch run: +UVM_TESTNAME=quit_test
UVM_ERROR sync_tb.sv(287) @ 10000: uvm_test_top [SB_MISMATCH] word mismatch at 10000
UVM_ERROR sync_tb.sv(287) @ 20000: uvm_test_top [SB_MISMATCH] word mismatch at 20000
UVM_ERROR sync_tb.sv(287) @ 30000: uvm_test_top [SB_MISMATCH] word mismatch at 30000
--- UVM Report Summary ---
Quit count reached!
Quit count :     3 of     3
UVM_ERROR :    3
TEST VERDICT: FAIL (UVM_ERROR=3, UVM_FATAL=0)

The same three lines were also written to sb_mismatch.log. The set_report_*_file calls exist per severity, per ID and per severity-and-ID, plus set_report_default_file, each with a _hier form for a whole subtree. Setting UVM_LOG without a file writes to the default file, which is 0, meaning standard output: the message lands in the main log anyway. Set the file and the action together.

A machine-readable summary

A regression dashboard wants fields, not prose. Extending report_summarize() once more writes a JSON record per run. The seed comes from the launcher as a plusarg: the launcher chose it, and there is no portable IEEE call that returns the simulator's seed ($get_initial_random_seed is a vendor extension).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

// snippet: verified in a scratch run (sync_tb.sv)
class json_report_server extends uvm_default_report_server;
  string json_file = "uvm_summary.json";
  function new(string name = "json_report_server");
    super.new(name);
  endfunction
  virtual function void report_summarize(UVM_FILE file = 0);
    string ids[$];
    string test = "unknown", seed = "unknown", id_list = "";
    int n_info = get_severity_count(UVM_INFO);       // read before the summary adds its own info
    int n_warn = get_severity_count(UVM_WARNING);
    int n_err  = get_severity_count(UVM_ERROR);
    int n_fat  = get_severity_count(UVM_FATAL);
    string verdict = (n_err == 0 && n_fat == 0) ? "PASS" : "FAIL";
    int fd;
    super.report_summarize(file);
    void'($value$plusargs("UVM_TESTNAME=%s", test));
    void'($value$plusargs("SEED=%s", seed));          // the launcher's copy of the seed, for the record only;
                                                     // the simulator's own seed switch is tool-specific
    get_id_set(ids);
    foreach (ids[i])
      id_list = {id_list, (i == 0) ? "" : ", ", $sformatf("\"%s\": %0d", ids[i], get_id_count(ids[i]))};
    fd = $fopen(json_file, "w");
    $fdisplay(fd, "{\"test\": \"%s\", \"seed\": \"%s\", \"verdict\": \"%s\", \"end_time\": \"%0t\",",
              test, seed, verdict, $time);
    $fdisplay(fd, " \"counts\": {\"info\": %0d, \"warning\": %0d, \"error\": %0d, \"fatal\": %0d},",
              n_info, n_warn, n_err, n_fat);
    $fdisplay(fd, " \"ids\": {%s}}", id_list);
    $fclose(fd);
    $display("TEST VERDICT: %s (UVM_ERROR=%0d, UVM_FATAL=%0d)", verdict, n_err, n_fat);
  endfunction
endclass

scratch run: +UVM_TESTNAME=json_test +SEED=1234   (uvm_summary.json; the library's own IDs trimmed)
{"test": "json_test", "seed": "1234", "verdict": "FAIL", "end_time": "20000",
 "counts": {"info": 4, "warning": 3, "error": 1, "fatal": 0},
 "ids": {"CFG": 1, "RNTST": 1, "SB_MISMATCH": 1, "UVM/RELNOTES": 1, "UVM/REPORT/SERVER": 1}}

The per-ID counts are the triage key: a regression of 2,000 tests that groups failures by their first error ID turns 140 red tests into four bugs. A CSV row (test,seed,verdict,errors,fatals,first_error_id) appended to one shared file is the lowest-tech version of the same idea. Escape any quote characters in IDs before writing real JSON, and write the file from report_summarize(), so a fatal run produces it too: the heartbeat scratch run in the next lesson ends on a fatal and its JSON was written.

Why the regression keys off one line

• Absence means failure. The script looks for TEST VERDICT: PASS. A crash, a kill or a truncated log has no verdict line and is a FAIL by construction, which is the bug from the opening.

• The counts, not the text. Grepping for UVM_ERROR also matches message text and misses demoted or caught messages. The server's counts already reflect overrides and catchers, which is the definition of the test result.

• One definition, in one place. Whether warnings fail a test, or a non-zero catcher summary needs review, is decided once in the server class every test installs, not in a dozen scripts.

• Also check the scoreboard's own evidence. A verdict of PASS with zero words compared is still a false pass; the bench's SB_EMPTY error exists for that reason.

What breaks: the PASS banner that vanished

A team printed *** TEST PASSED *** or *** TEST FAILED *** from the base test's report_phase, and the regression grepped for PASSED. It took a while to notice that a test ending on a uvm_fatal printed neither: the fatal called die(), the report phases never ran, and the script classified the test as "no result", which a colleague's filter then silently dropped from the report. Fatals were exactly the failures they most needed to see. Moving the verdict into a report server's report_summarize(), as in dma_verdict_fatal_test above, made every exit print it.

Next: a verdict is only useful if the run ends. The next lesson ends hung runs in microseconds: Heartbeats and Watchdogs.

Related topics

• Up: Synchronisation and Reporting

• run_test, timeouts and how a simulation ends

• Command-line control

• Seed control and replay