The regression farm kills a job after two hours of wall-clock time. Its log ends in the middle of nothing: no error, no summary, no verdict. Somebody reruns it with waves on and it hangs again, this time with a 30 GB waveform file. A hang is the most expensive failure in verification, not because it is hard to fix, but because nothing about it is visible until you make it visible. How do you turn a two-hour silence into a five-minute diagnosis?

A hang is a wait that nothing will ever satisfy

Every UVM hang has the same shape. Somebody holds an objection, so the run phase cannot end, and the thread that would drop it is blocked on a wait: for an item, a grant, an event, a signal. The thing it waits for will never happen. Think of a phone call on hold where the other side has already hung up. The line is open, the hold music plays, and you will wait forever unless something has a clock.

So hang debug is two questions in order: who holds the objection, and what is that holder waiting for. Before either, rule out a different animal. If simulation time has stopped advancing (a forever loop with no delay, a combinational loop in the design), it is not a UVM hang at all: the simulator is spinning at one time step, and a UVM timeout, which is measured in simulated time, can never fire. A job whose log shows time frozen needs the simulator's own interrupt or a wall-clock limit, and then a stack dump.

The picture: hang triage

Step one: make every hang end on its own

The library's default timeout is 9200 s of simulated time, long enough for a farm to burn hours on one stuck job. Every bench should set a timeout that fits its longest legitimate test, from the command line (+UVM_TIMEOUT=5000000,YES on a commercial simulator, in the time unit of the run) or in code with uvm_root::get().set_timeout(5us, 1). The timeout gives you an end; a watchdog gives you an early end. The course bench has one: it subscribes to all three monitors and, if there has been no traffic for 2 µs while the run phase is still held open, raises a fatal that says when the last activity was and what it was.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

// snippet: from the course DMA bench
// Ends a stalled test early: if no monitor reports any traffic for quiet_limit
// while someone still holds the run phase open, it is a hang, not a slow test.
class dma_activity_watchdog extends uvm_component;
  `uvm_component_utils(dma_activity_watchdog)
  uvm_analysis_imp_wd_apb #(apb_item,  dma_activity_watchdog) apb_export;
  uvm_analysis_imp_wd_in  #(axis_item, dma_activity_watchdog) in_export;
  uvm_analysis_imp_wd_out #(axis_item, dma_activity_watchdog) out_export;
  time quiet_limit = 2us;
  time last_activity;
  string last_what = "nothing yet";

  function new(string name, uvm_component parent);
    super.new(name, parent);
    apb_export = new("apb_export", this);
    in_export  = new("in_export", this);
    out_export = new("out_export", this);
  endfunction

  virtual function void write_wd_apb(apb_item t);  last_activity = $time; last_what = t.convert2string(); endfunction
  virtual function void write_wd_in(axis_item t);  last_activity = $time; last_what = "input beat";       endfunction
  virtual function void write_wd_out(axis_item t); last_activity = $time; last_what = "output beat";      endfunction

  task run_phase(uvm_phase phase);
    forever begin
      #(quiet_limit / 4);
      if ($time - last_activity > quiet_limit &&
          phase.get_objection().get_objection_total(uvm_root::get()) > 0)
        `uvm_fatal("WATCHDOG", $sformatf("no bus activity for %0t while the test is still running; last activity at %0t: %s",
                   $time - last_activity, last_activity, last_what))
    end
  endtask
endclass

dma_stall_watchdog_test starts a job and never sends data. The watchdog ends it at 2.5 µs, with a message that already contains the diagnosis: the last thing on any bus was the CTRL write that started the job, so the DMA is waiting for input that never came.

+UVM_TESTNAME=dma_stall_watchdog_test
UVM_FATAL dma_tb.sv(1816) @ 2500000: uvm_test_top.wd [WATCHDOG] no bus activity for 2345000 while the test is still running; last activity at 155000: WR addr=00 data=00000003
UVM_ERROR :    0
UVM_FATAL :    1
TEST VERDICT: FAIL (UVM_ERROR=0, UVM_FATAL=1)

Two details make a watchdog trustworthy. It checks the objection total, so it never fires during a legitimate idle after the test has finished. And its quiet limit should be a per-test knob. The bench's quiet_limit is a plain 2 us default; a test that waits for a 10 µs timer inside the DUT sets a longer limit on its watchdog (or reads one from config_db), rather than disabling the watchdog.

The broken bench: a driver that forgets item_done

dma_hang_noack_test is the smoke test with one change: a factory type override swaps the stream driver for axis_driver_noack, which drives each beat correctly but never calls item_done().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

// snippet: from the course DMA bench
// A driver that forgets item_done. Used only by the hang playbook.
class axis_driver_noack extends axis_driver;
  `uvm_component_utils(axis_driver_noack)
  function new(string name, uvm_component parent);
    super.new(name, parent);
  endfunction
  task run_phase(uvm_phase phase);
    vif.tvalid <= 1'b0;
    @(posedge vif.rst_n);
    forever begin
      seq_item_port.get_next_item(req);
      @(posedge vif.clk);
      vif.tdata  <= req.data;
      vif.tlast  <= req.last;
      vif.tvalid <= 1'b1;
      do @(posedge vif.clk); while (!vif.tready);
      vif.tvalid <= 1'b0;
      // BUG: no seq_item_port.item_done() here
    end
  endtask
endclass

// Hang playbook: the stream driver forgets item_done.
class dma_hang_noack_test extends dma_smoke_test;
  `uvm_component_utils(dma_hang_noack_test)
  function new(string name, uvm_component parent);
    super.new(name, parent);
    install_verdict_server();
  endfunction
  function void build_phase(uvm_phase phase);
    axis_driver::type_id::set_type_override(axis_driver_noack::get_type());
    super.build_phase(phase);
    uvm_root::get().set_timeout(5us, 1);
  endfunction
endclass

Watch it hang

+UVM_TESTNAME=dma_hang_noack_test
UVM_ERROR @ 175000: uvm_test_top.env.in_agt.sqr [uvm_test_top.env.in_agt.sqr] Get_next_item called twice without item_done or get in between
UVM_ERROR @ 195000: uvm_test_top.env.in_agt.sqr [uvm_test_top.env.in_agt.sqr] Get_next_item called twice without item_done or get in between
UVM_ERROR @ 215000: uvm_test_top.env.in_agt.sqr [uvm_test_top.env.in_agt.sqr] Get_next_item called twice without item_done or get in between
   ... the same error every 20 ns, 8 in all, the last at 315000 ...
UVM_FATAL base/uvm_phase_hopper.svh(598) @ 5000000: reporter [PH_TIMEOUT] Explicit timeout of 5000000 hit, indicating a probable testbench issue
UVM_ERROR :    8
UVM_FATAL :    1
[PH_TIMEOUT]     1
[uvm_test_top.env.in_agt.sqr]     8
TEST VERDICT: FAIL (UVM_ERROR=8, UVM_FATAL=1)

Read the log the way the triage diagram says. The run ended on PH_TIMEOUT at 5 µs, so simulated time was advancing and this is a real UVM hang. The first error names the cause: the sequencer saw get_next_item twice with no item_done between. Because the first item was never completed, the sequencer hands the driver the same item again on each call, and the driver re-drives the same word until the DMA, having taken its eight words, stops accepting. After that the driver is blocked on TREADY and the log goes quiet for 4.7 µs.

Notice what is missing: the scoreboard's [SB] ... words matched line. A fatal ends the run, so report_phase never runs. The verdict line is still there, because the bench's report server prints it from report_summarize, which the library calls on the way out.

Who holds the objection?

Before the timeout fires, ask the objection who is holding it. A scratch test forks one thread that calls display_objections() just before the 5 µs timeout:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// snippet: scratch test (dbg_tb.sv)
class dbg_hang_holders_test extends dma_hang_noack_test;
  `uvm_component_utils(dbg_hang_holders_test)
  function new(string name, uvm_component parent);
    super.new(name, parent);
  endfunction
  task run_phase(uvm_phase phase);
    fork
      begin
        #4900ns;                                     // just before set_timeout(5us)
        phase.get_objection().display_objections(null, 1);
      end
    join_none
    super.run_phase(phase);
  endtask
endclass

UVM_INFO base/uvm_objection.svh(1319) @ 4900000: run_objection [UVM/OBJ/DISPLAY] The total objection count is 1
---------------------------------------------------------
Source  Total
Count   Count   Object
---------------------------------------------------------
0       1       uvm_top
1       1         uvm_test_top
---------------------------------------------------------
UVM_FATAL base/uvm_phase_hopper.svh(598) @ 5000000: reporter [PH_TIMEOUT] Explicit timeout of 5000000 hit, indicating a probable testbench issue

The holder is uvm_test_top itself (source count 1). So the test's run_phase has not reached its drop_objection, and the question becomes where it is blocked. The smoke test's run phase calls pkt.start(), the sequence's body is in finish_item(), and finish_item() returns only when the driver calls item_done(). The chain is complete: test, sequence, finish_item, missing item_done. Restore seq_item_port.item_done() at the end of the driver loop, as in axis_driver, and the same test ends with 8 words matched, 0 mismatched.

The usual suspects, in the order to check them

Two of these leave no error at all, which is why the watchdog's "last activity" message is worth so much: it tells you which interface went quiet first. A bounded wait in a driver is the same idea at protocol level. Instead of do @(posedge clk); while (!pready); forever, count clocks and raise a named error after a protocol-sensible limit, so the hang turns into a message that names the bus and the address.

1
2
3
4

// snippet: waiting on state, so a trigger that already happened is not missed
uvm_event done_ev = uvm_event_pool::get_global("dma_irq");
done_ev.wait_on();          // returns at once if the event is already on
// done_ev.wait_trigger();  // waits for the NEXT trigger: hangs if it fired before this line

What breaks: the timeout that never fired

A team set a sensible set_timeout on every test and still had jobs killed by the farm after hours. The log of the killed job ended at the same simulated time in every rerun. A monitor had a forever loop that waited on a clocking-block event inside an if that, after reset, was never taken, so the loop spun with no delay at all. Simulated time stopped, and a timeout measured in simulated time cannot expire if time does not move. The diagnosis came from the farm's wall-clock kill, a rerun with the simulator's interactive interrupt, and a stack dump showing the monitor's loop. The fix was a loop whose every path waits on a clock edge, and a wall-clock limit per job so the next one dies in minutes.

Next: a test that ends with an answer is easier than one that never ends, but the answer can still mislead. Scoreboard Mismatch Playbook starts from one wrong word and finds whose bug it is.

Related topics

• Driver handshake hangs

• Objection leak triage

• lock vs grab

• run_test and timeouts