Two stories from the same week. In the first, a test drops its objection as soon as the last input word is accepted; the scoreboard compares five words out of six, reports no error, and the test passes with a word still inside the DMA. In the second, a hung test hits its timeout and dies with a fatal; the regression dashboard shows no PASS/FAIL banner for it at all, because the banner was printed in report_phase, which never ran. One is a check that looked at the wrong thing, the other a verdict that lived in the wrong phase. What exactly has to be true before a test may pass, and where does each check belong?

The idea: closing a restaurant for the night

A restaurant does not close when the last customer walks in; it closes when every order has been served, every bill is paid, nothing is left on the pass, and the till has been counted. And a night with zero customers is not a great night with no complaints; it is a reason to ask whether the doors were locked. A test's end is that audit. Run-time checks catch wrong items as they happen. End-of-test checks catch what can only be known when traffic has stopped: things still owed, things never seen, and the suspicious absence of anything at all.

The picture

Build it: the checklist

Two rules follow from the table. Waiting and judging are different jobs: phase_ready_to_end decides whether the run phase may end yet; check_phase decides whether what happened was right. And monitors never raise objections: a monitor only observes, so end of test is owned by the test and, when it must wait for outstanding items, by a scoreboard through phase_ready_to_end.

Waiting: owed work comes from what the DUT was asked to do

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

// snippet: from the course DMA bench
// A scoreboard that owns end of test. It learns how many words each job owes
// from the observed LEN and GO register writes, and when every objection drops
// while words are still owed, it raises its own objection until they arrive.
// (Counting on the expected queue alone is not enough: the last input word may
// not have been published yet when the test drops its objection.)
class dma_scoreboard_eot extends dma_scoreboard;
  `uvm_component_utils(dma_scoreboard_eot)
  int unsigned len_reg, owed;
  event        drained;

  function new(string name, uvm_component parent);
    super.new(name, parent);
  endfunction

  virtual function void write_apb(apb_item t);
    super.write_apb(t);
    if (!t.write || t.slverr) return;
    if (t.addr == LEN)                 len_reg = t.data[15:0];
    if (t.addr == CTRL && t.data[0])   owed += len_reg;   // GO: this job owes len_reg words
  endfunction

  virtual function void write_out(axis_item t);
    super.write_out(t);
    if (owed > 0) owed--;
    if (owed == 0) -> drained;
  endfunction

  function void phase_ready_to_end(uvm_phase phase);
    if (phase.get_imp() != uvm_run_phase::get() || owed == 0) return;
    phase.raise_objection(this, $sformatf("%0d word(s) still owed by the DMA", owed));
    fork begin
      @drained;
      phase.drop_objection(this, "all words out");
    end join_none
  endfunction
endclass

The scoreboard learns the job size from the observed LEN and GO writes, adds it to owed, and counts output words down. When the run phase's objection count reaches zero while words are still owed, phase_ready_to_end raises an objection and forks a thread that drops it when the count reaches zero. The mechanism itself, including the iteration cap and the get_imp() comparison, is taught in phase_ready_to_end.

+UVM_TESTNAME=dma_drain_test
UVM_INFO dma_tb.sv(411) @ 375000: uvm_test_top.env.sb [SB] 6 words matched, 0 mismatched
UVM_ERROR :    0

Judging: check_phase runs once, after everything

check_phase is a function phase: it takes no time, runs after the run phase (and every runtime phase) has finished, and visits components bottom-up. It is the right home for any check whose answer is only final when traffic has stopped. It is the wrong home for anything that could have been reported earlier: an unexpected word reported at end of test has lost its time stamp.

Reporting: summaries, not verdicts

report_phase is where the bench prints 8 words matched, 0 mismatched. Keep it for summaries. Two facts about the report server matter for verdicts. First, it counts every severity, warnings included: every run on this machine shows UVM_WARNING : 2 from the library's own notes about running without DPI, and the scratch latency run above showed 4. The UVM_COUNT action only decides which messages count toward the quit count (set_max_quit_count), not which are counted. Second, it prints the summary of those counts even when a fatal ends the run.

Why a fatal skips report_phase

A UVM_FATAL ends the simulation at once. The library calls pre_abort() on the components, prints the report summary and finishes; extract, check, report and final never run. dma_timeout_test, which waits for an interrupt that is never enabled, shows it:

+UVM_TESTNAME=dma_timeout_test
UVM_FATAL uvm_phase_hopper.svh(598) @ 20000000: reporter [PH_TIMEOUT] Explicit timeout of 20000000 hit, indicating a probable testbench issue
UVM_FATAL :    1
[PH_TIMEOUT]     1

There is no [SB] ... words matched line: the scoreboard's report_phase never ran, and neither did its check_phase. Anything that must appear for every test, pass or fail, therefore cannot live there. The robust verdict is the report summary itself (regression scripts decide on UVM_ERROR : and UVM_FATAL :), and when you need a custom verdict line, put it in an override of the report server's report_summarize or in a test's pre_abort() as well as in report_phase.

Documenting an exception: the zero-length job

Some tests legitimately compare nothing. dma_zero_len_test starts a job with LEN = 0 to close a coverage hole: the DMA must report DONE at once with COUNT = 0, and no data moves. SB_EMPTY would fail it. The test does not disable the check, does not remove it from the scoreboard, and does not raise the global error budget. It demotes one ID, on one component, in one test, with a comment saying why:

1
2
3
4
5
6
7

// snippet: from dma_zero_len_test
// No data moves, so the scoreboard's "nothing was compared" error is expected
// here, and only here: demote that one ID on that one component, on purpose.
function void end_of_elaboration_phase(uvm_phase phase);
  super.end_of_elaboration_phase(phase);
  env.sb.set_report_severity_id_override(UVM_ERROR, "SB_EMPTY", UVM_INFO);
endfunction

+UVM_TESTNAME=dma_zero_len_test
UVM_INFO dma_tb.sv(1718) @ 275000: uvm_test_top [ZERO] zero-length job finished at once with DONE
UVM_INFO dma_tb.sv(407) @ 325000: uvm_test_top.env.sb [SB_EMPTY] no words were compared: the test checked nothing
UVM_INFO dma_tb.sv(411) @ 325000: uvm_test_top.env.sb [SB] 0 words matched, 0 mismatched
UVM_ERROR :    0
UVM_FATAL :    0

The waiver is visible: the message still prints, now as UVM_INFO, so anyone reading the log sees that the check fired and was waived. The test replaces the waived check with checks of its own (DONE set, ERR clear, COUNT = 0), so it still proves something. That is the standard for any exception: narrow, commented, visible in the log, and replaced by a check that fits the case.

What breaks: the check that passed with a word missing

This is the first opening story, and it happened while the bench was being built. The first end-of-test scoreboard decided whether to extend the run phase by looking at its expected queue: if predictions were waiting, words were still owed. dma_drain_test dropped its objection in the same timestep the last input word was accepted, before the input monitor had published that word. At that instant the queue looked empty. Nothing was extended, the run phase ended, and the last word was never predicted and never compared. Five of six words matched, no leftovers, UVM_ERROR : 0.

The fix is in the code above: the scoreboard learns how many words each job owes from the observed LEN and GO register writes, and holds the phase until they have all come out. The principle generalises to every end-of-test check: "nothing pending" must come from what the DUT was asked to do, not from what the testbench happens to have seen so far. A queue can only tell you about predictions already made.

Next: the model inside dma_scoreboard has done its job for this topic. Reference Models takes it out into a predictor component of its own, calls a C model through DPI-C, and shows when the library's built-in comparators help and when they lie.

Related topics

• Next topic: Reference Models

• phase_ready_to_end

• End-of-test strategy

• Reporting and verbosity