Skip to content
DenseChip
Learn
PracticeCode labInterview

DenseChip

Navigate

Menu

LearnPracticeCode labInterview

Tools

All toolsSV editorCode repoResume
DenseChip

Practice, courses, and chip-design tools in one focused workspace.

LearnPracticeCode labInterviewToolsLearning paths

© 2026 DenseChip

AboutContactPrivacyTerms
Skip to lesson

Course contents

  • Start Here6
  • SystemVerilog Language Foundations28
  • Object-Oriented SystemVerilog for Verification30
  • Constraint Randomization — Interview-Critical Deep Dive36
  • SystemVerilog Assertions — Interview-Critical Deep Dive40
  • Functional Coverage in SystemVerilog22
  • Testbench Architecture in SystemVerilog23
  • Advanced Language & Integration25
  • Senior & Interview Preparation21
  • Principal-Level Verification28
    • Reusable Testbench Architecture
    • Checking Out-of-Order Traffic
      • ID-Tagged Scoreboard
      • Multi-Stream Scoreboard
      • Drain and Leak Accounting
      • Memory-Model Checker
    • Assertion Depth
    • Coverage Modelling for Real Features
    • Capstone: Out-of-Order Memory Controller
    • Principal Judgment
Loading lesson

‹Checking Out-of-Order Traffic
  1. Topics
  2. SystemVerilog
  3. Principal-Level Verification
  4. Checking Out-of-Order Traffic
  5. Drain and Leak Accounting

Part 9 · Principal-Level Verification · Advanced

Drain and Leak Accounting

End-of-test accounting: issue times, in-flight watchdogs, bounded drain, and a report that names leaked tags and hung streams.

On this page

    Most escaped checking bugs are at the end

    A scoreboard that compares every response it sees can still pass a broken design. If a request is never answered, there is nothing to compare. The test ends, the queues are not empty, and nobody looks. End-of-test accounting closes that hole.

    Three faults need three different reports:

    Fault

    What you see

    Report

    Leak

    One request never answered; the DUT keeps working

    The ID or tag, when it was issued, and how long the DUT ran past it

    Hang (stuck stream)

    No responses at all for a long time while work is outstanding

    Time since last progress and the IDs waiting

    Drop with in-order IDs

    A later response on the same ID mismatches

    Data mismatch on that ID (the previous lessons)

    The third row is why this lesson uses unique tags. With a pool of tags, each tag has at most one request outstanding, so a dropped request shows up as a missing response instead of shifting every later comparison on that ID.

    What to record

    • Issue time per outstanding item, in a queue shaped like the expected queue.

    • Last progress time: the cycle of the most recent response of any kind.

    • A watchdog that warns once when an item passes the maximum latency. It finds the leak at the time it happens, not 10,000 cycles later.

    • A bounded drain: after stimulus stops, wait until nothing is outstanding or a limit passes. Never wait forever.

    The drivers must not hang either. A driver that waits forever for a free tag turns a DUT leak into a simulator timeout with no message. The driver below gives up after 100 cycles and says so.

    Leak or hang: one rule

    At the end of the drain, classify every item still outstanding:

    • If the DUT kept answering other requests for more than the maximum latency after this item was issued (last_progress - issued_at > max_lat), the item was leaked. The DUT was alive and skipped it.

    • Otherwise the item is stuck behind a hang. If nothing has come back for longer than the stall limit, the stream is stuck.

    Measuring against last_progress, not against the current time, is the whole trick. Every item looks old once the stream hangs; only the leaked one was old while the DUT was still working.

    Runnable example: a leaked tag and a hung stream

    The engine has eight tags, a latency of 2 to 9 cycles from the data, and answers the lowest ready tag first. The bench drives two copies: a healthy one, and one with two planted faults. On the bad copy, the third request on tag 5 is accepted and never answered, and from cycle 100 no response is sent at all. The drain scoreboard extends the ID scoreboard from the first lesson with issue times and classification.

    tag_if.sv
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    interface tag_if (input logic clk);
      logic       req_valid = 0;
      logic [2:0] req_tag   = 0;
      logic [7:0] req_data  = 0;
      logic       rsp_valid;
      logic [2:0] rsp_tag;
      logic [7:0] rsp_data;
    endinterface
    
    // Tagged engine: 8 tags, latency 2..9 cycles from the data, lowest ready tag answers first.
    // LEAK_TAG >= 0: the 3rd request on that tag is accepted and never answered.
    // HANG_AT  >= 0: from that cycle on, no response is ever sent.
    module tag_engine #(parameter int LEAK_TAG = -1, parameter int HANG_AT = -1) (
      input logic clk, rst_n,
      tag_if      bus
    );
      logic       busy [8];
      logic [3:0] timer[8];
      logic [7:0] data [8];
      logic [1:0] uses [8];
      int         cycle;
    
      always_ff @(posedge clk or negedge rst_n) begin
        if (!rst_n) begin
          for (int t = 0; t < 8; t++) begin busy[t] <= 0; timer[t] <= 0; uses[t] <= 0; end
          bus.rsp_valid <= 0; bus.rsp_tag <= 0; bus.rsp_data <= 0; cycle <= 0;
        end else begin
          automatic logic found = 0;
          cycle <= cycle + 1;
          bus.rsp_valid <= 0;
          for (int t = 0; t < 8; t++) if (busy[t] && timer[t] != 0) timer[t] <= timer[t] - 1;
          if (HANG_AT < 0 || cycle < HANG_AT)
            for (int t = 0; t < 8; t++)
              if (!found && busy[t] && timer[t] == 0) begin
                found = 1;
                busy[t] <= 0;
                bus.rsp_valid <= 1; bus.rsp_tag <= 3'(t); bus.rsp_data <= ~data[t];
              end
          if (bus.req_valid) begin
            uses[bus.req_tag] <= uses[bus.req_tag] + 1;
            data[bus.req_tag] <= bus.req_data;
            timer[bus.req_tag] <= 4'(bus.req_data[2:0]) + 2;
            busy[bus.req_tag] <= !(int'(bus.req_tag) == LEAK_TAG && uses[bus.req_tag] == 2);
          end
        end
      end
    endmodule
    
    class id_scoreboard #(type T = logic [7:0]);
      string name;
      protected T exp_q[int][$];
      int n_added, n_matched, n_errors;
      function new(string name); this.name = name; endfunction
      virtual function void add_expected(int id, T item);
        exp_q[id].push_back(item);
        n_added++;
      endfunction
      virtual function void check_actual(int id, T item);
        T exp;
        if (!exp_q.exists(id) || exp_q[id].size() == 0) begin
          n_errors++;
          $display("%s: UNEXPECTED response for ID %0d with nothing outstanding", name, id);
          return;
        end
        exp = exp_q[id].pop_front();
        if (exp_q[id].size() == 0) exp_q.delete(id);
        if (item !== exp) begin
          n_errors++;
          $display("%s: MISMATCH ID %0d expected %h got %h", name, id, exp, item);
        end else n_matched++;
      endfunction
      function int outstanding();
        int n = 0;
        foreach (exp_q[id]) n += exp_q[id].size();
        return n;
      endfunction
    endclass
    
    // Adds issue times, progress tracking and end-of-test classification.
    class drain_scoreboard #(type T = logic [7:0]) extends id_scoreboard #(T);
      protected longint issued_at[int][$];   // same shape as exp_q
      longint last_progress;
      int     max_lat, stall_limit;
      int     leaked[$], stuck[$];          // filled by classify()
      bit     stream_stuck;
    
      function new(string name, int max_lat, int stall_limit);
        super.new(name);
        this.max_lat = max_lat;
        this.stall_limit = stall_limit;
      endfunction
    
      function void add_at(longint now, int id, T item);
        add_expected(id, item);
        issued_at[id].push_back(now);
      endfunction
    
      function void check_at(longint now, int id, T item);
        check_actual(id, item);
        if (issued_at.exists(id)) begin
          void'(issued_at[id].pop_front());
          if (issued_at[id].size() == 0) issued_at.delete(id);
        end
        last_progress = now;
      endfunction
    
      // Called every cycle: report an item the first time it passes max_lat.
      function void watchdog(longint now);
        foreach (issued_at[id])
          if (now - issued_at[id][0] == max_lat)
            $display("%s: WARN ID %0d has waited %0d cycles (issued at %0d)", name, id, max_lat, issued_at[id][0]);
      endfunction
    
      // End of test: a leak is an item the DUT outlived by more than max_lat while
      // still answering others; everything else left behind is stuck in a hung stream.
      function void classify(longint now);
        foreach (issued_at[id])
          foreach (issued_at[id][k])
            if (last_progress - issued_at[id][k] > max_lat) leaked.push_back(id);
            else stuck.push_back(id);
        stream_stuck = stuck.size() != 0 && (now - last_progress) > stall_limit;
        foreach (leaked[i]) $display("%s: LEAK ID %0d never answered; DUT kept working for %0d+ cycles after it", name, leaked[i], max_lat);
        if (stream_stuck)
          $display("%s: STUCK no response for %0d cycles; IDs waiting: %p", name, now - last_progress, stuck);
        if (outstanding() == 0) $display("%s: drained clean, %0d matched", name, n_matched);
      endfunction
    endclass
    
    class tag_agent;
      virtual tag_if vif;
      drain_scoreboard #(logic [7:0]) sb;
      int    free_tags[$];
      longint cycle;
      bit    starved;
    
      function new(virtual tag_if vif, string name);
        this.vif = vif;
        sb = new(name, 40, 60);
        for (int t = 0; t < 8; t++) free_tags.push_back(t);
      endfunction
    
      // Monitor: one process per agent, runs for the whole test.
      task monitor();
        forever begin
          @(posedge vif.clk);
          cycle++;
          if (vif.rsp_valid) begin
            sb.check_at(cycle, vif.rsp_tag, ~vif.rsp_data);
            free_tags.push_back(vif.rsp_tag);
          end
          sb.watchdog(cycle);
        end
      endtask
    
      // Driver: decides on the falling edge, so the monitor has already freed
      // this cycle's tag. It never waits forever for a free tag.
      task drive(int n_req);
        for (int n = 0; n < n_req; n++) begin
          int waited = 0;
          int t;
          logic [7:0] d;
          @(negedge vif.clk);
          vif.req_valid <= 0;
          while (free_tags.size() == 0 && waited < 100) begin @(negedge vif.clk); waited++; end
          if (free_tags.size() == 0) begin
            starved = 1;
            $display("%s: driver gave up after %0d cycles with no free tag", sb.name, waited);
            return;
          end
          t = free_tags.pop_front();
          d = 8'($urandom);
          vif.req_valid <= 1; vif.req_tag <= 3'(t); vif.req_data <= d;
          sb.add_at(cycle + 1, t, d);      // the DUT samples it on the next edge
        end
        @(negedge vif.clk) vif.req_valid <= 0;
      endtask
    
      // Drain: wait for the last response, but never longer than the limit.
      task drain(int limit);
        int waited = 0;
        while (sb.outstanding() != 0 && waited < limit) begin @(posedge vif.clk); waited++; end
        sb.classify(cycle);
      endtask
    endclass
    

    The bench passes. The good engine drains clean with 100 matches. On the bad one, the watchdog warns about tag 5 at cycle 63, long before the end, then the driver gives up, and the drain report says:

    Diagram
    bad: LEAK ID 5 never answered; DUT kept working for 40+ cycles after it
    bad: STUCK no response for 301 cycles; IDs waiting: '{'h0, 'h1, 'h2, 'h3, 'h4, 'h6, 'h7}

    Two mutants were run and both fail the bench:

    • Classify against now: change last_progress - issued_at[id][k] to now - issued_at[id][k] in classify(). Every outstanding item then counts as a leak: the report names eight leaked tags and no stuck stream.

    • Plant a leak in the good engine (LEAK_TAG(2) on u_good). Its drain is no longer clean and the bench fails.

    In UVM, this logic belongs in check_phase or report_phase, and the bounded drain maps to the objection drain time and phase_ready_to_end. The classification rule is the same.

    Common pitfall

    Checking only outstanding() == 0 at the end, with a drain that waits forever. A leak then becomes a global timeout, and the log shows nothing but the timeout. Bound every wait and print what is left, by name.

    Choosing the limits

    • max_lat comes from the spec or the design's worst case under the test's traffic, plus margin. Here the worst case is about 16 cycles (9 cycles of latency plus seven tags ahead), so 40 is safe.

    • stall_limit is longer than any legal quiet period: refresh, power state exits, back-pressure the test applies on purpose.

    • Scale both with configuration. A fixed number tuned on one config gives false leaks on a slower one.

    Interview

    Question: "Your regression times out once a week with no error. How do you make the bench tell you why?" Answer: record issue time per outstanding item, add a watchdog against the maximum latency, bound the drain, and at the end print every outstanding item with its ID, issue time and age, split into leaked items (the DUT kept working) and stuck items (the DUT stopped).

    Check yourself Why does the leaked tag 5 not appear in the stuck list, even though it is also still outstanding?

    It was issued at cycle 23, and the DUT kept answering until about cycle 100. That is more than 40 cycles of progress after it, so the rule classifies it as leaked, not stuck.

    Check yourself With in-order IDs (lesson 1) instead of unique tags, what does a dropped request look like?

    The next response on that ID mismatches, because the scoreboard compares it with the dropped item. The end-of-test count is one short on that ID. Unique tags turn the same fault into a clean, named leak.

    Related topics

    • Previous: Multi-Stream Scoreboard

    • Next: Memory-Model Checker

    • Compare and End-of-Test Checks (Part 5)

    • Debugging the Testbench

    Previous

    239 / 259

    Next: Memory-Model Checker

    Up next

    Up nextMemory-Model Checker

    Sign in to save progress

    Practice this lessonReport an error

    See also

    • ID-Tagged Scoreboard