A testbench sends perfect traffic: every source offers a flit every cycle, every sink accepts at once and returns credits immediately. FIFOs never fill, credits never run out, the router never idles. The design's handshake logic, its full flags, its credit counters at zero, its sleep entry in the middle of traffic: all of it sits unexercised, while the regression reports thousands of packets delivered.

What stimulus finds bugs that only appear when events collide or stall?

The obvious answer, and why it breaks

The obvious answer is more and more varied packets. But many bugs depend on when things happen, not on what is sent: a credit returned in the same cycle as a send, a push and a pop together, an input that pauses for eight cycles mid-packet, a sink that stops accepting. Smooth, polite timing never creates these coincidences. The data can be random while the timing stays perfectly regular.

The idea: randomise timing as well as data

Build it: identical packets, smooth or stalled

Eight rounds of good traffic, every input to random outputs. Each round gives every source an idle profile (none, 40% or 92% idle) and every sink a drain profile (always, 40% or 4% per cycle). After the traffic, PKT_CNT is compared with the packets seen, and a credit audit empties every sink, then stalls one at a time and counts the flits it receives: exactly 4 must arrive. Compiling with +define+SMOOTH sends the same packets with no gaps and sinks that never stall.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

// uses: dcrouter
// Lesson 4.8: random timing and back-pressure. The same good traffic (every input to every
// output, random lengths) runs in eight rounds. Each round gives every source an idle
// profile (none, some, or long pauses between flits) and every sink a drain profile (fast,
// medium, or almost stalled), so FIFOs fill, credits run out, the router falls asleep in
// the middle of traffic, and sends, returns, pushes and pops collide in the same cycles.
// A final audit stalls each sink and counts the flits it receives: exactly 4 credits each.
// Compile with +define+SMOOTH to run the identical packets with no gaps and no stalls.
module tb;
  `include "dcr_env.svh"
  int gaps [3] = '{0, 40, 92};
  int drains [3] = '{100, 40, 4};
  int n0;
  initial begin
    reset_dut();
    for (int r = 0; r < 8; r++) begin
      for (int k = 0; k < 4; k++) begin
`ifdef SMOOTH
        gap_pct[k] = 0;   drain_pct[k] = 100;
`else
        gap_pct[k] = gaps[$urandom_range(0, 2)];   drain_pct[k] = drains[$urandom_range(0, 2)];
`endif
      end
      for (int n = 0; n < 6; n++)
        for (int i = 0; i < 4; i++) queue_pkt(i, $urandom_range(0, 3), 0, $urandom_range(1, 15));
      wait_drain(40000);
    end
    // every packet seen at an output was counted there (R10)
    for (int k = 0; k < 4; k++) check_reg(8'h20 + 8'(4 * k), got_pkts[k], $sformatf("PKT_CNT%0d", k));
    // credit audit: empty every sink, then stall one at a time; no credit lost or invented (R7)
    gap_pct = '{0, 0, 0, 0};
    drain_pct = '{100, 100, 100, 100};
    while (occ.sum() > 0) @(posedge clk);
    repeat (2) @(posedge clk);
    for (int o = 0; o < 4; o++) begin
      drain_pct[o] = 0;
      n0 = got_flits[o];
      queue_pkt(0, o, 0, 15);
      repeat (60) @(posedge clk);
      if (got_flits[o] - n0 != 4) fail($sformatf("credit audit: %0d flits left output %0d into a stalled sink, expected 4", got_flits[o] - n0, o));
      drain_pct[o] = 100;
      wait_drain();
    end
    finish($sformatf("%0d packets through eight rounds of random stalls, all intact and counted; credits audited", got_pkts[0] + got_pkts[1] + got_pkts[2] + got_pkts[3]));
  end
endmodule

How to read the waveform: in0_vld stays high while in0_rdy is low, so input 0 must hold its flit; the driver never changes valid data while stalled. On output 1, four flits leave back to back using its four credits; after that, each flit leaves the cycle after out1_crd returns a credit. That is R7 working under pressure, a state smooth traffic never reaches.

Run it against the Bug Zoo, both ways

With random stalls:

clean design: PASS
  BUG_1  caught  ERROR @1465: out0: flit 9038, expected 5aae from input 1
  BUG_2  caught  ERROR @95: out3: flit 0000, expected aca5 from input 0
  BUG_3  caught  ERROR @305: out3: out_err 1 on flit c5c3, expected 0
  BUG_8  caught  ERROR @55: out3: out_err 1 on flit da00, expected 0
  BUG_9  caught  ERROR @119675: credit audit: 1 flits left output 0 into a stalled sink, expected 4
  BUG_10 caught  ERROR @109515: credit audit: 3 flits left output 0 into a stalled sink, expected 4
  BUG_11 caught  ERROR @142475: PKT_CNT1: read 30, expected 31
  BUG_12 caught  ERROR @21475: out0: out_err 1 on flit 0080, expected 0
  BUG_17 caught  ERROR @400025: 66 flits still pending after 40000 cycles (lost or stuck)
  BUG_20 caught  ERROR @16325: out2: unexpected flit 89ef
  BUG_22 caught  ERROR @143185: credit audit: 1 flits left output 0 into a stalled sink, expected 4
  BUG_24 caught  ERROR @555: out2: unexpected flit da02
caught 12 of 24 bugs

The same packets with +define+SMOOTH:

clean design: PASS
  BUG_1  caught  ERROR @315: out2: unexpected flit a8a1
  BUG_2  caught  ERROR @145: out1: flit a9ad, expected 38b4 from input 0
  BUG_3  caught  ERROR @65: out3: out_err 1 on flit c5c3, expected 0
  BUG_8  caught  ERROR @55: out0: out_err 1 on flit 0500, expected 0
  BUG_9  caught  ERROR @19255: credit audit: 1 flits left output 0 into a stalled sink, expected 4
  BUG_10 caught  ERROR @10925: credit audit: 3 flits left output 0 into a stalled sink, expected 4
  BUG_11 caught  ERROR @10245: PKT_CNT1: read 2e, expected 30
  BUG_12 caught  ERROR @3345: out1: out_err 1 on flit 0075, expected 0
  BUG_17 caught  ERROR @400025: 78 flits still pending after 40000 cycles (lost or stuck)
  BUG_24 caught  ERROR @205: out1: unexpected flit c401
caught 10 of 24 bugs

How to read it: 12 caught with stalls against 10 smooth. The difference is BUG_20 and BUG_22: sleeping with an output locked mid-packet needs a source to pause while everything else is quiet, and losing credit returns while asleep needs a slow sink still holding flits when the router sleeps. Smooth traffic creates neither situation. Both runs catch BUG_9 and BUG_10 only through the final credit audit; without it, lost credits would just make the router slower (lesson 3.2).

What timing randomisation should include

• Profiles, not just jitter: long pauses and near-stalls, not only occasional single-cycle gaps.

• Changing profiles: rounds or phases with different settings, so the design moves between full and empty.

• Per-port independence: coincidences across ports are where concurrency bugs live (lesson 1.4).

• An audit at the end: hidden state disturbed by timing (credits, counters) must be checked once traffic stops.

• Legal behaviour only: stalls and gaps the interface allows; a driver that drops valid data while stalled is a testbench bug.

How real teams randomise timing

What breaks

Even with stalls, this test misses 12 bugs, including all the error-path bugs (it sends no faults) and BUG_21 (nothing arranges for the router to sleep and then be woken by input 3 alone). Timing randomisation multiplies what traffic can find; it does not replace error injection or scenarios.

Key takeaways

• Identical packets exercise different logic under different timing.

• Randomise timing profiles per port and per phase: pauses, stalls, bursts, coincidences.

• Keep stalls legal, and audit hidden state such as credits when traffic stops.

Related topics

• Part 4 overview: Stimulus