Measure before reasoning

Distribution bugs never fail a test. Every call returns 1 with legal values, while a corner case appears once per million transactions instead of once per fifty.

The only witness is measurement. Bump counters in post_randomize over a short standalone loop, and keep a covergroup on rand fields so every regression watches the spread.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

class txn;
  rand bit       is_err;
  rand bit [3:0] len;
  constraint err_c { is_err dist { 1 := 1, 0 := 9 }; }  // intend 10% errors

  static int unsigned hist_err[2];
  static int unsigned hist_len[16];

  function void post_randomize();
    hist_err[is_err]++;
    hist_len[len]++;
  endfunction

  static function void report();
    int unsigned total = hist_err[0] + hist_err[1];
    $display("err count: %0d of %0d", hist_err[1], total);
    foreach (hist_len[i])
      $display("len=%0d : %0d", i, hist_len[i]);
  endfunction
endclass

module top;
  initial begin
    txn x = new();
    repeat (2000) if (!x.randomize()) $fatal(1, "txn randomize failed");
    txn::report();
    // A tolerance band, never an exact count: 10% of 2000 is about 200.
    if (txn::hist_err[1] < 160 || txn::hist_err[1] > 240)
      $fatal(1, "FAIL: %0d errors, outside the 160..240 band", txn::hist_err[1]);
    $display("PASS: error rate %0d of 2000 is inside the band", txn::hist_err[1]);
    $finish;
  end
endmodule

err count: 189 of 2000
len=0 : 129
len=1 : 148
len=2 : 123
len=3 : 133
len=4 : 114
len=5 : 126
len=6 : 104
len=7 : 110
len=8 : 131
len=9 : 136
len=10 : 121
len=11 : 124
len=12 : 101
len=13 : 132
len=14 : 148
len=15 : 120
PASS: error rate 189 of 2000 is inside the band

For permanent monitoring, sample a covergroup in post_randomize instead. Skew then shows up as cold bins in the coverage report of every regression, not just when someone remembers to look.

Verilator 5.048 accepts covergroups but does not collect them, so this fragment is an illustration and the course checks skip it. Functional Coverage shows what does run.

1
2
3
4
5
6
7
8
9
10
11
12
13

// snippet
class txn_cov_wrap;
  txn t = new();
  covergroup cg;
    cp_err: coverpoint t.is_err { bins err = {1}; bins ok = {0}; }
    cp_len: coverpoint t.len {
      bins zero = {0}; bins mid = {[1:14]}; bins max = {15};
    }
    x_err_len: cross cp_err, cp_len;
  endgroup
  function new(); cg = new(); endfunction
  task gen_one(); if (!t.randomize()) $fatal(1, "randomize failed"); cg.sample(); endtask
endclass

Implication skew and solve-before

The LRM asks for a uniform pick over the joint solution space, but tools only approximate it, so measure. Constraint structure also shapes marginal distributions in non-obvious ways.

The classic case is an implication where one branch admits far more solutions. is_small -> val < 4 on a 10-bit val leaves 4 tuples with is_small == 1 and 1024 with it clear, so a uniform pick sets is_small about once in 257 calls. Bidirectional Solving measures exactly this shape and its fixes.

Eaten defaults and clipped ranges

A soft constraint that conflicts with any hard constraint is dropped entirely and silently. In layered benches the base author believes the default holds while the test author believes the override is narrow.

Width and sign bugs masquerade as skew. Check declared ranges before blaming the solver: unrepresentable values clip without warning.

The bench measures all three, plus the worked error-rate example from the next section, so the counts sit on one screen.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

class bus_txn;
  rand bit       err;
  rand bit [7:0] addr;
  constraint err_rate_c { err dist { 1 := 1, 0 := 9 }; }    // intended: 10% errors
  constraint err_addr_c { err -> addr == 8'hFF; }
endclass

class bus_txn_ordered extends bus_txn;
  constraint order_c { solve err before addr; }
endclass

class clipped;
  rand bit [3:0] len;                      // 4 bits: 0..15
  constraint c { len inside {[8:20]}; }    // 16..20 cannot exist
  rand byte signed_off;                    // signed: -128..127
  constraint c2 { signed_off < 10; }
endclass

class cfg;
  rand bit [7:0] qdepth;
  constraint typical_c { soft qdepth == 8; }
endclass

err rate, dist 1:9 plus implication: 88 of 1000; with solve err before addr: 102
len inside [8:20] on 4 bits: largest len seen 15
signed_off < 10 on a signed byte: 848 of 1000 negative
soft qdepth == 8: 200 of 200 at 8 alone; 0 of 200 once qdepth > 8 is added
PASS: the clipped range, the negative byte and the eaten soft default all showed up in the counts

The negative share came out at 85%, where a uniform pick over the 138 legal values gives 93%. Nothing in the class asked for negative values: the type did. Read the declaration before blaming the solver.

A worked skew and its fix

An error-injection rate can measure below its dist target when an implication gives one side far fewer completions: err allows one address, a clear err allows 256. How a solver combines the two is tool-dependent, so measure.

On Verilator 5.048 the bench above measured 88 errors in 1000 calls with the dist alone and 102 with solve err before addr: close to the 10% target both times. Another tool may fall much further short; the histogram is the proof, not the theory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

// snippet
class bus_txn;
  rand bit       err;
  rand bit [7:0] addr;
  constraint err_rate_c { err dist { 1 := 1, 0 := 9 }; }
  constraint err_addr_c { err -> addr == 8'hFF; }
endclass
// err=1 admits 1 addr value, err=0 admits 256. On some solvers this
// imbalance pulls the measured err rate below 10%. Measure it.

class bus_txn_fixed;
  rand bit       err;
  rand bit [7:0] addr;
  constraint err_rate_c { err dist { 1 := 1, 0 := 9 }; }
  constraint err_addr_c { err -> addr == 8'hFF; }
  constraint order_c    { solve err before addr; }
  // err is picked first honoring its dist, then addr solves inside
  // the chosen branch. The histogram confirms the rate.
endclass

Related topics

• Finding Contradictions

• The Randomization Debug Checklist