Where memory growth usually comes from

Many UVM memory issues are retention bugs, not allocator bugs: queues that never drain, associative arrays that are never deleted, and references kept in debug containers long after use.

[PERF] common retention sources

  scoreboard exp queues not drained/deleted
  subscriber history buffers unbounded
  sequence item handles stored in debug maps forever
  per-test static globals accumulating across iterations

Leak symptom pattern

[REG] memory trend across long regression

  test index -> RSS MB
    1   -> 1400
    20  -> 1525
    40  -> 1660
    80  -> 1930

  steady upward trend without plateau => investigate retained structures

Queue hygiene and bounded buffers

Any queue used for history or diagnostics should have clear bounds or eviction policy. Scoreboard expected stores must delete entries after compare and report leftovers in check_phase.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

class bounded_history #(type T = int);
  localparam int MAX_ENTRIES = 2048;
  T q[$];

  function void push(T item);
    q.push_back(item);
    if (q.size() > MAX_ENTRIES)
      void'(q.pop_front());
  endfunction
endclass

function void write_act(bus_txn act);
  if (!exp_by_id.exists(act.id)) begin
    unexpected_count++;
    return;
  end
  if (!act.compare(exp_by_id[act.id]))
    mismatch_count++;
  exp_by_id.delete(act.id); // critical to avoid retained growth
endfunction

[CHECK] memory-safe scoreboard rules

  insert expected exactly once
  consume and delete on compare
  drain leftovers in check_phase
  clear temporary debug caches between tests

Practical leak investigation workflow

Use long repeat loops with fixed seed to detect monotonic memory growth. Then instrument suspected containers (size and high-water marks) at periodic intervals.

1
2
3
4
5
6
7

# Repeat same test to expose retention growth
for i in $(seq 1 100); do
  simv +UVM_TESTNAME=axi_long_random +ntb_random_seed=22291 > "repeat_${i}.log"
done

# Extract queue/map telemetry
rg "SCB_MEM|HWMARK|exp_q|exp_by_id" repeat_*.log

1
2
3
4
5
6
7
8
9
10
11
12
13

int exp_q_hwm = 0;
int exp_map_hwm = 0;

function void update_mem_stats();
  if (exp_q.size() > exp_q_hwm) exp_q_hwm = exp_q.size();
  if (exp_by_id.num() > exp_map_hwm) exp_map_hwm = exp_by_id.num();
  `uvm_info(
    "SCB_MEM",
    $sformatf("exp_q=%0d exp_q_hwm=%0d exp_map=%0d exp_map_hwm=%0d",
      exp_q.size(), exp_q_hwm, exp_by_id.num(), exp_map_hwm),
    UVM_MEDIUM
  )
endfunction

End-of-test cleanup hooks

• Reset large transient containers in final_phase if lifetime is per-test.

• Avoid static globals for test-local state.

• Release references in custom pools when test ends.

Key takeaways

• Memory regressions are often container-retention bugs.

• Bound history buffers and always delete consumed expected entries.

• Track queue/map high-water marks in long-run tests.

• Add explicit cleanup policy for per-test temporary structures.

Common pitfalls

• Assuming simulator leak when scoreboard containers keep growing.

• Unbounded debug history queues left enabled in farm runs.

• No memory trend monitoring in CI until late project stage.