Monday morning, the overnight regression has 37 tests marked TIMEOUT, each one having burned the farm's full four-hour limit. Nobody touched the testbench on Friday, except one change to an interrupt sequence. You have the logs and nothing else. Where do you start, and how quickly can you name the line of code that is holding every one of those tests open?

This lesson is the playbook. It covers the three ways the end of a test goes wrong, the two pieces of evidence that tell them apart, and the fix for each, using the bench's dma_timeout_test as the hang you can reproduce.

The idea: three failures, one counter

A test's end is decided by one number, uvm_top's objection total (the mechanics are in Objection Hierarchy Propagation). Every end-of-test failure is that number doing something unexpected:

• Hang: the total never reaches zero. Someone raised and never dropped, or a wait between raise and drop never returns. The run ends only at the timeout.

• Instant pass: the total was never above zero. Nobody raised before the first blocking statement, so the run phase ended at time 0.

• Early end: the total reached zero while work was still in flight, typically a drop right after fork ... join_none, or a test that let go before the DUT finished.

Think of a hospital ward's sign-out sheet. A hang is a patient signed in and never signed out: the ward cannot close. An instant pass is a ward that closed because nobody signed anyone in. An early end is a nurse signing a patient out while they are still in surgery. Each needs a different question, but you answer all three from the same sheet.

The picture: triage in four questions

Build it: make hangs cheap

A hang costs whatever the timeout costs, so the first fix is a short one. The UVM default is 9200 s of simulated time, which on a real design is hours of wall clock. The bench's dma_timeout_test sets 20 µs; the interrupt it waits for is never enabled, so it hangs on purpose:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// snippet: from the course DMA bench
// A test that would hang: the interrupt is never enabled, so it never fires.
// The global timeout ends the run with a fatal instead of burning the regression slot.
class dma_timeout_test extends dma_base_test;
  `uvm_component_utils(dma_timeout_test)

  function new(string name, uvm_component parent);
    super.new(name, parent);
  endfunction

  function void build_phase(uvm_phase phase);
    super.build_phase(phase);
    uvm_root::get().set_timeout(20us, 1);   // replaces a +UVM_TIMEOUT=...,YES value; ,NO locks the command line
  endfunction

  task run_phase(uvm_phase phase);
    dma_job_cfg job = dma_job_cfg::type_id::create("job");
    bit [31:0] status;
    phase.raise_objection(this);
    if (!job.randomize() with { len == 4; irq_en == 0; }) `uvm_fatal("RAND", "dma_job_cfg randomize failed")
    run_job(job, 4, status);                 // waits for an interrupt that never comes
    phase.drop_objection(this);
  endtask
endclass

+UVM_TIMEOUT=<time>,<YES|NO> on the command line does the same; the second argument says whether a later call may override it. The last call before the run starts wins unless a call locked the value. The command-line value is applied in uvm_root's build_phase, before the test's build_phase, so a test that calls set_timeout(20us, 1) there replaces a +UVM_TIMEOUT=...,YES value. To make the command line win, pass ,NO: that locks it against the test's call.

The second fix is evidence. A timeout tells you that a test hung, not who hung it. Dump the objection table just before the timeout fires. A small thread in the base test does it:

1
2
3
4
5
6
7
8
9

// snippet
// In the base test: dump every holder shortly before the timeout.
task run_phase(uvm_phase phase);
  fork begin
    #19us;                                       // just under the 20 us timeout
    phase.get_objection().display_objections();  // lists sequences too
  end join_none
  super.run_phase(phase);
endtask

With +UVM_PHASE_TRACE, the library's own timeout watchdog also prints the outstanding objections of each phase before the fatal.

Watch it run: reading the evidence

The real end of the timeout test's log, with that dump:

UVM_INFO uvm_objection.svh(1319) @ 19000000: run_objection [UVM/OBJ/DISPLAY] The total objection count is 1
Source  Total
Count   Count   Object
0       1       uvm_top
1       1         uvm_test_top
UVM_FATAL uvm_phase_hopper.svh(598) @ 20000000: reporter [PH_TIMEOUT] Explicit timeout of 20000000 hit, indicating a probable testbench issue

--- UVM Report Summary ---
** Report counts by severity
UVM_ERROR :    0
UVM_FATAL :    1

• The fatal time is the configured timeout, so this is a hang, not an early end.

• The table has one row with a non-zero source count: uvm_test_top. The test itself holds the run phase.

• No `[SB]` line: report_phase did not run after the fatal. The summary is the only verdict.

From here it is a code read: the test raised, called run_job, and run_job waits for irq_vif.irq, which can never rise because irq_en is 0. The objection trace gives the same answer in a different form: the last raised line for the test has no matching dropped line. When the table shows a sequence, its row sits under the sequencer it runs on, so the indentation tells you which agent to open.

For an instant pass, the evidence is the absence of a raise: the run phase ends at time 0 and a zero-compare check (SB_EMPTY in the bench) is the only error. For an early end, the trace shows the last drop at a time when the monitors were still publishing words, and the scoreboard reports leftovers.

The hang catalog

The unified-drop shape for the first two rows is in Objection Counting Rules: one raise, the work raced against a timeout inside an isolated fork begin ... end join with disable fork, and one drop after it. disable fork kills every child of the current process, so the isolating wrapper is what keeps it from killing unrelated threads.

What breaks: two worked cases

The overnight hang

Back to the 37 timeouts. The dump in every log showed the same holder: a sequence row under irq_agt.sqr. Friday's change had made the interrupt handler sequence wait for a second interrupt edge before finishing, and in these 37 tests the DUT raised only one. The sequence's automatic objection held the run phase open forever. The fix bounded the wait and reported a uvm_error when the second edge did not come. The process fix mattered more: the farm's timeout dropped from four hours to a per-test value, and every base test gained the pre-timeout dump. The next hang was diagnosed from its log in five minutes.

The instant pass that looked like a speed-up

A refactor made a family of tests run in 1% of their usual time, and someone celebrated a faster regression. The log showed the run phase ending at 0 ns. A new base-class run_phase waited for reset release before calling the derived test's scenario, and the derived tests raised their objection inside the scenario, after that wait. Moving the raise to the first line of run_phase, before any blocking statement, restored the tests; SB_EMPTY-style checks in every scoreboard now turn the same mistake into an immediate failure.

Guards that stop repeats

Next: the last lesson of this part looks at the tool people reach for when the built-in schedule seems too small: Custom phases.

Related topics

• Custom phases

• Objection counting rules

• Sequence handshake hangs