Every test is green, and the link runs at half speed

A streaming block was retimed to close timing: a register stage went into each of its three pipeline steps. All functional tests pass. Two weeks later the system team measures the link and gets 0.49 words per cycle instead of 1.

Nothing is wrong with any value. Where did half the bandwidth go?

What people usually do, and why it fails

The usual response is to suspect the measurement, then the traffic, then "the system". When the block is finally suspected, people scroll a waveform of a busy stream, where a bubble every other cycle looks like normal traffic. Without a number to compare against, nobody can say the block is slow, and tests that check values cannot fail on timing.

The signature

• Throughput collapse: a clean ratio, very often exactly one half, one third or one Nth of the expected rate. A ratio that simple points at a structural cause, such as a stage that cannot overlap, rather than at contention.

• Bubbles: in a waveform, valid low in alternate cycles downstream of a stage, with its input stalled in the cycles between; in counters, input stalls and output bubbles that rise together, one per word.

• A regression in cycles: the same directed test, same seed, takes more cycles after a commit, with every check still passing. Only a test that reports its cycles can show it.

The first three checks

1. Measure, do not estimate. Count transfers and cycles at the input and the output of the block for one directed test. Compute words per cycle and compare with the specification. If the number is a clean fraction, look for a structural cause.

2. Walk the stall counters stage by stage. At each stage boundary count cycles with valid && !ready (stalls) and cycles with !valid while work is pending (bubbles). The source is the first stage that stalls its input in a cycle when its own output is being taken: it had room and refused.

3. Compare against the last good version. Run the same test on the commit before and after and diff the cycles. If the regression came with a change, bisect it with a script that fails on cycles, exactly as you would on a wrong value.

The picture

Read it as two stages fed the same stream; count words per slot.

• Slots 0 to 7, bad stage: in_ready_bad alternates 0 and 1, and out_valid_bad alternates 1 and 0. It holds a word in one cycle and is empty the next, so it can never take a word in the cycle its old word leaves.

• Slots 0 to 7, good stage: in_ready_good and out_valid_good stay at 1, and out_data_good advances every slot.

• After 8 slots the bad stage has delivered words 1 to 4, the good stage words 1 to 8: exactly half.

One real reproduction

Three of these stages in a row, a source that always has data and a sink that is always ready. The bench has a scoreboard, counters for input stalls and output bubbles, and a cycle budget: 64 words may take at most 64 + 8 cycles from the first word in to the last word out. FIX=1, the default, is the stage that also accepts when its word is leaving.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// pipe_stage: one register stage on a valid/ready path; adds 1 to the data.
// FIX=0 is the bug: the stage only takes a word when it is empty, so a full stage
// cannot take a new word in the same cycle the next stage takes the old one.
module pipe_stage #(parameter bit FIX = 1) (
  input  logic       clk, rst_n,
  input  logic       in_valid,
  output logic       in_ready,
  input  logic [7:0] in_data,
  output logic       out_valid,
  input  logic       out_ready,
  output logic [7:0] out_data
);
  assign in_ready = FIX ? (!out_valid || out_ready) : !out_valid;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin out_valid <= 1'b0; out_data <= '0; end
    else if (in_ready)  begin out_valid <= in_valid; out_data <= in_data + 8'd1; end
    else if (out_ready) out_valid <= 1'b0;          // the word left; the stage is empty next cycle
endmodule

$ iverilog -g2012 -o p.vvp bubble_tb.sv && vvp -n p.vvp
PERF: 64 words in 67 cycles (0.95 words/cycle), input stalls=0, output bubbles=0
PASS: 64 words in order, within the cycle budget (FIX=1)
...
$ iverilog -g2012 -Pbubble_tb.FIX=0 -o pb.vvp bubble_tb.sv && vvp -n pb.vvp
PERF: 64 words in 130 cycles (0.49 words/cycle), input stalls=63, output bubbles=63
FAIL: PERF: 130 cycles for 64 words, budget 72
...

• Fixed: 67 cycles for 64 words, which is one word per cycle plus three cycles to fill three stages. No stalls, no bubbles.

• Bug: 130 cycles, 0.49 words per cycle. The scoreboard did not fail: every word arrived, in order, with the right value. Only the budget line fails the run.

• 63 and 63: one input stall and one output bubble per word after the first. Counters that move in lockstep like this mean one structural cause, not random contention. A probe copy of this bench with the counters per stage puts all 63 stalls at the first stage's input and none at the other two: the first half-rate stage sets the rate, and the stages after it never fill. All three share the bug, so fixing only the first would move the stalls to the second.

This half-rate stage is a known style with a known cost; the RTL course measures it beside a skid buffer. As a bug, it usually arrives the way it did in the scene: someone adds a register to fix timing and copies the simplest stage.

dbglab: a fix that is a 30% regression

dbglab's history has both directions. Commit 11 made dma_copy faster (15366 ns to 11816 ns; the counters lesson measures where the time went) and introduced the deadlock. The safe fix for the deadlock puts the old arbitration back, and with it the old cost. Since commit 16 every PASS line reports its cycles, so a small script can hold each directed test to a baseline:

1
2
3
4
5
6
7
8
9
10
11

#!/bin/sh
# cycle-budget.sh <baseline-file> <tree>: run each "test seed cycles" line of the baseline
# in <tree> and fail if a test takes more than 5% more cycles than its baseline.
bad=0
while read -r test seed base; do
  now=$(cd "$2" && tb/run.sh "$test" "$seed" | sed -n 's/.*cycles=\([0-9]*\).*/\1/p')
  if [ -z "$now" ]; then echo "FAIL $test seed $seed: no PASS line"; bad=1; continue; fi
  if [ "$now" -gt $((base * 105 / 100)) ]; then echo "FAIL $test seed $seed: $now cycles, baseline $base"; bad=1
  else echo "ok   $test seed $seed: $now cycles, baseline $base"; fi
done < "$1"
exit $bad

$ cat baseline.txt
smoke 1 76
dma_copy 1 1182
periph_stream 1 687
$ ./cycle-budget.sh baseline.txt dbglab        # HEAD
ok   smoke seed 1: 76 cycles, baseline 76
ok   dma_copy seed 1: 1182 cycles, baseline 1182
ok   periph_stream seed 1: 687 cycles, baseline 687
$ ./cycle-budget.sh baseline.txt fixb2         # HEAD with assign accept  = rsp_fire;
ok   smoke seed 1: 76 cycles, baseline 76
FAIL dma_copy seed 1: 1537 cycles, baseline 1182
ok   periph_stream seed 1: 687 cycles, baseline 687

The fix passes every functional test and costs dma_copy 355 cycles, 30%. That may be the right trade (a hang is worse than a slow copy), but it should be a decision, made with the number in front of you, not a surprise in the system lab. The case study's lens shows a fix that keeps the speed-up: lock only the chunks that cannot wait on the outside world.

The fix pattern

• Bubbles in a pipeline: let a stage accept when it is empty or its word is leaving (!valid || ready), or use a skid buffer where ready must be registered.

• Throughput lost to arbitration or locking: measure wait cycles per master first, then change the policy; every such change trades one master's latency for another's.

• A regression in cycles: treat it like any regression. Bisect with a script that fails on cycles, then decide whether the cost is accepted, and update the baseline only with that decision written down.

Preventing it

Three habits catch this class early. A cycle budget for every directed test, checked in every regression, like the script above or the BUDGET check in the bench. Throughput assertions on streams that must run at full rate: for example, a stage never stalls its input in a cycle when its output is taken (in_valid && out_valid && out_ready |-> in_ready). In the probe copy it fires in all 63 stall cycles, all at the first stage, and never with the fix. And a review question for every new register stage: can this stage take a word in the same cycle it hands one on?

Traps

Related topics

• Part 5 · Bug Classes

• RTL: three pipeline stage styles, measured

• Counters and performance probes

• Protocol First Principles: bubbles and half throughput

• Case study: the deadlock under back-pressure

• Architecture: performance counters