A nightly SoC regression has 1,800 tests with a 4-hour timeout each. Eleven of them hang every night, on different seeds. Each hung test holds a license and a farm slot for the full 4 hours, then reports PH_TIMEOUT with no hint of what stopped. That is 44 license-hours a night spent on silence, and a triage queue that starts every morning from zero.

A timeout is an upper bound on how long a test may run. What you actually want to know is that the testbench stopped moving. This lesson builds both kinds of detector on the bench: the library's uvm_heartbeat, which is objection-based and easy to configure wrongly, and the bench's simpler activity watchdog.

The idea: an alarm clock versus a smoke detector

The global timeout is an alarm clock: it rings at a fixed time whether or not anything is wrong, so you set it late and it tells you little. A watchdog is a smoke detector: it watches for a specific sign of trouble (no traffic, no progress) and goes off soon after the trouble starts, with a message about what it saw. You keep both. The alarm clock catches what the detector cannot see; the detector catches most hangs in a fraction of the time and says where to look.

UVM's built-in detector is uvm_heartbeat. It watches a dedicated objection that components raise and drop each time they make progress, and it checks, once per window, that the right components did so. A window is the time between two triggers of an event you supply.

The picture: how a heartbeat decides

Build it: a heartbeat that actually fires

The example below was run as a scratch test. Two components beat every 30 ns; the env triggers the heartbeat event every 100 ns; the monitor stops at 400 ns.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

// snippet: verified in a scratch run (sync_tb.sv)
class hb_worker extends uvm_component;
  `uvm_component_utils(hb_worker)
  uvm_callbacks_objection hb_obj;
  time period  = 30ns;
  time stop_at = 0;                        // 0 = never stops
  function new(string name, uvm_component parent);
    super.new(name, parent);
  endfunction
  task run_phase(uvm_phase phase);
    forever begin
      #(period);
      if (stop_at != 0 && $time >= stop_at) wait (0);   // stuck
      hb_obj.raise_objection(this);        // one beat = one raise and one drop
      hb_obj.drop_objection(this);
    end
  endtask
endclass

class hb_env extends uvm_env;
  `uvm_component_utils(hb_env)
  hb_worker               drv, mon;
  uvm_callbacks_objection hb_obj;
  uvm_heartbeat           hb;
  uvm_event               hb_ev;
  uvm_heartbeat_modes     mode   = UVM_ALL_ACTIVE;
  time                    window = 100ns;
  function new(string name, uvm_component parent);
    super.new(name, parent);
  endfunction
  function void build_phase(uvm_phase phase);
    hb_obj = new("hb_obj");                // its own objection, never a phase objection
    drv = hb_worker::type_id::create("drv", this);
    mon = hb_worker::type_id::create("mon", this);
    drv.hb_obj = hb_obj;
    mon.hb_obj = hb_obj;
    hb_ev = new("hb_ev");
    hb    = new("hb", this, hb_obj);       // context = this env
  endfunction
  task run_phase(uvm_phase phase);
    uvm_component watched[$];
    watched.push_back(drv);
    watched.push_back(mon);
    void'(hb.set_mode(mode));
    hb.set_heartbeat(hb_ev, watched);      // registers the list and starts monitoring
    forever begin
      hb_ev.trigger();                     // closes one window, opens the next
      #(window);
    end
  endtask
endclass

Each detail in it is there because the heartbeat silently does nothing, or fires on the wrong thing, without it. These rules come from the 1800.2 source of uvm_heartbeat and its callback:

• Use a dedicated objection, of type uvm_callbacks_objection. In UVM 1.2 the heartbeat constructor requires that type; in IEEE 1800.2 it is a typedef of uvm_objection, so the same code compiles on both. Never use a phase objection: every beat would also be an end-of-test vote.

• The context must be an ancestor of the beating components. The callback counts a beat only when the propagating objection reaches the context object. A context outside the beaters' branch sees nothing, and UVM_ALL_ACTIVE then fails every component at the first check.

• The first trigger only opens a window. Checking starts at the second trigger. With a 100 ns window, the earliest possible failure is 200 ns.

• Any component that beats is added to the list. The callback creates a count for every source it sees, so a component that beats once becomes a required participant in UVM_ALL_ACTIVE mode from then on.

• The window must exceed the longest legitimate idle gap. A DMA stalled 28 clocks by backpressure is not hung. Derive the window from the protocol's worst case, not from a typical run.

Call hb.stop() around phases where silence is legal (a long reset, a power-down window) and hb.start() afterwards; add() and remove() change the list while it runs.

The bench's simpler alternative: an activity watchdog

A heartbeat needs every watched component to beat, which means editing them, or adding subscriber shims. dma_activity_watchdog needs nothing from the agents. It subscribes to the three monitors' analysis ports, remembers when traffic was last seen and what it was, and fires only while someone still holds the run phase open:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

// snippet: from the course DMA bench
// Ends a stalled test early: if no monitor reports any traffic for quiet_limit
// while someone still holds the run phase open, it is a hang, not a slow test.
class dma_activity_watchdog extends uvm_component;
  `uvm_component_utils(dma_activity_watchdog)
  uvm_analysis_imp_wd_apb #(apb_item,  dma_activity_watchdog) apb_export;
  uvm_analysis_imp_wd_in  #(axis_item, dma_activity_watchdog) in_export;
  uvm_analysis_imp_wd_out #(axis_item, dma_activity_watchdog) out_export;
  time quiet_limit = 2us;
  time last_activity;
  string last_what = "nothing yet";

  function new(string name, uvm_component parent);
    super.new(name, parent);
    apb_export = new("apb_export", this);
    in_export  = new("in_export", this);
    out_export = new("out_export", this);
  endfunction

  virtual function void write_wd_apb(apb_item t);  last_activity = $time; last_what = t.convert2string(); endfunction
  virtual function void write_wd_in(axis_item t);  last_activity = $time; last_what = "input beat";       endfunction
  virtual function void write_wd_out(axis_item t); last_activity = $time; last_what = "output beat";      endfunction

  task run_phase(uvm_phase phase);
    forever begin
      #(quiet_limit / 4);
      if ($time - last_activity > quiet_limit &&
          phase.get_objection().get_objection_total(uvm_root::get()) > 0)
        `uvm_fatal("WATCHDOG", $sformatf("no bus activity for %0t while the test is still running; last activity at %0t: %s",
                   $time - last_activity, last_activity, last_what))
    end
  endtask
endclass

dma_stall_watchdog_test starts a job and never sends data; its global timeout is 20 µs:

+UVM_TESTNAME=dma_stall_watchdog_test
UVM_FATAL dma_tb.sv(1816) @ 2500000: uvm_test_top.wd [WATCHDOG] no bus activity for 2345000 while the test is still running; last activity at 155000: WR addr=00 data=00000003
UVM_ERROR :    0
UVM_FATAL :    1
TEST VERDICT: FAIL (UVM_ERROR=0, UVM_FATAL=1)

The run ended at 2.5 µs instead of 20, and the message says what the last activity was: the CTRL write that started the job. That message is the head start a triage engineer needs. The check every quiet_limit / 4 means it fires between 2 µs and 2.5 µs after the last traffic, and the objection test keeps it quiet once the test has finished.

Which detector, and the timeout policy around them

• Set the timeout per test class, in the base test, never relying on the 9,200 s default. Long tests override it upward; debug reruns can raise it with +UVM_TIMEOUT=<time>,YES when the setter left it overridable.

• Express quiet limits in clocks from the config object, so a slower clock or a longer packet does not need a code change.

• Give each hang detector its own ID (PH_TIMEOUT, HBFAIL, WATCHDOG), so the regression can bucket hangs by kind without reading logs.

Watch it run

scratch run: +UVM_TESTNAME=hb_test   (UVM_ALL_ACTIVE, 100 ns window, mon stops at 400 ns)
UVM_INFO sync_tb.sv(305) @ 420000: uvm_test_top.env.mon [HB_WORKER] stuck: no more beats
UVM_FATAL @ 500000: uvm_test_top.env [HBFAIL] Did not recieve an update of hb_obj for component uvm_test_top.env.mon since last event trigger at time 400000 : last update time was 390000
UVM_FATAL :    1
TEST VERDICT: FAIL (UVM_ERROR=0, UVM_FATAL=1)

scratch run: +UVM_TESTNAME=hb_any_test   (same, UVM_ANY_ACTIVE)
UVM_INFO sync_tb.sv(305) @ 420000: uvm_test_top.env.mon [HB_WORKER] stuck: no more beats
UVM_FATAL :    0
TEST VERDICT: PASS (UVM_ERROR=0, UVM_FATAL=0)

• The monitor's last beat was at 390 ns. The window 300–400 ns was satisfied; the window 400–500 ns had no beat from it, so the check at 500 ns failed and named it. (The misspelling is the library's own text.)

• The fatal still printed the verdict, through the report server from the previous lesson.

• ANY_ACTIVE passed with the monitor stuck, because the driver kept beating. That is the next section's story.

What breaks: the mode that hid the stuck monitor

A team added a heartbeat to a four-channel stream subsystem and chose UVM_ANY_ACTIVE, because channels legitimately idle. Months later a monitor bug made one channel's monitor stop sampling after a mid-test reset. The heartbeat stayed quiet, since the other three channels kept beating, and the scoreboard's leftover check was the first thing to notice, at the end of a 3-hour test. The scratch run above shows the same blind spot in miniature.

The fix was to match the mode to the question. UVM_ANY_ACTIVE asks "is the subsystem alive?", which is a fine question for a long SoC test. "Is every channel's monitor alive while its channel has traffic?" needs a per-channel check: one heartbeat per channel in UVM_ALL_ACTIVE, started and stopped by that channel's traffic, or an activity watchdog per channel.

Next: detectors tell you that something stopped. To see what the transactions were doing before it stopped, record them: Transaction Recording.

Related topics

• Up: Synchronisation and Reporting

• End-of-test strategy

• Objection leak triage

• Hang triage