It is in your block now

Tuesday. The verification engineer who took row S3 from the triage has walked it back from the stream pins to the power domain: a signal goes unknown while the domain is off. The power-domain owner sits in another team, eight hours away. The last time a bug crossed that boundary it took three days of messages before anyone looked at the RTL (an illustrative story, but a common one).

What must travel with this bug so that the power-domain owner starts at the first wrong cycle tomorrow morning, not at the regression page?

What people usually do, and why it fails

The usual hand-off is a reassignment: the ticket gets a new owner and a comment, "looks like a power-domain issue, see the log". The receiver opens the log, finds one FAIL line, rebuilds the repro, wonders which simulator was used, and starts the walk-back from the symptom again. If they reach a different conclusion, the ticket goes back. Two people now half-own it, and neither is working on it.

Two habits cause most of the waste. Ownership by block: the bug "belongs" to whoever owns the module where the symptom shows, so it moves when someone has a guess, not when the evidence says so. Handing over the failure instead of the hunt: the receiver gets the symptom, but not what the giver learned, which is the expensive part.

The method: one owner, a package, an acceptance

Who owns the bug

• One person at a time, by name. A team cannot own a bug; a team's queue is where bugs wait.

• The owner is whoever acts next. During Reproduce and Localise that is often verification; once the first wrong cycle is inside a block, the block's designer; for a timing question, physical design; for a part on a bench, the lab.

• Ownership moves by acceptance. The giver offers, the receiver checks the package, then takes it. Until then the giver still owns it and still answers questions.

• The tracker records each move with the step of the debug loop the bug is on, so anyone can see where it stands.

What the package contains

Most of these fields already exist if the giver kept a debug log and a replay record; the package is their summary, written for one reader.

When the hand-off is done

1. The receiver builds a fresh tree from the package's commit and runs its command: the same FAIL line.

2. The receiver runs the probe or opens the window: the same first wrong cycle.

3. Only then is ownership moved. Anything that differs goes back to the giver as a question, the same day.

The picture

Do it: the package for S3

Signature in both simulators

$ tb/run.sh periph_power 1
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
$ tb/run.sh periph_power 1 verilator
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008

The same bug prints two different signatures: an unknown value in Icarus, a hang in Verilator, whose two-state supply model drives noise instead of X. If the package names only one, the receiver who runs the other simulator sees "a different bug". Both lines go in.

History

$ git bisect start HEAD v0.1
$ git bisect run tb/run.sh periph_power 1
...
$ git bisect log
...
# bad: [122e88a7c8e74f9e3b29db8fad03dbe3fd3b7e9b] periph_pd: isolate only the outputs that leave the peripheral
git bisect bad 122e88a7c8e74f9e3b29db8fad03dbe3fd3b7e9b
# good: [b357bd84b7252b54097cac920e6a14cadda7503c] csr: no reset on the DMA address and data registers
git bisect good b357bd84b7252b54097cac920e6a14cadda7503c
# first bad commit: [122e88a7c8e74f9e3b29db8fad03dbe3fd3b7e9b] periph_pd: isolate only the outputs that leave the peripheral
$ git show 122e88a
...
    rinc only feeds our own async FIFO, so it does not need a clamp.
...
-  assign rinc      = iso_en ? 1'b0  : rinc_p;
+  assign rinc      = rinc_p;          // internal: only feeds our own async FIFO

First wrong cycle, with the probe that shows it

The giver's probe prints the power controls, the sink's read-increment and the FIFO's read side on every edge of the peripheral clock. It is compiled next to the design and changes nothing (the exact command is in the Evidence box). It goes in the package as a file, because a cycle number the receiver cannot reproduce is only a claim.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
// pd_probe: one line per pclk edge from +FROM to +TO (bus cycles): the power controls,
// the sink's read-increment and the FIFO's read side. Reads the design, changes nothing.
`timescale 1ns/1ps
module pd_probe;
  int unsigned from_c, to_c;
  initial begin
    if (!$value$plusargs("FROM=%d", from_c)) from_c = 0;
    if (!$value$plusargs("TO=%d", to_c))     to_c   = 0;
    $display("  ns  cycle pwr_on iso_en rinc rempty rbin");
  end
  always @(posedge dbglab_tb.pclk) begin
    #0.5;
    if (dbglab_tb.cycle >= from_c && dbglab_tb.cycle <= to_c)
      $display("%4.0f  %5d    %b      %b     %b     %b    %h", $realtime, dbglab_tb.cycle,
        dbglab_tb.dut.u_per.u_pd.pwr_on, dbglab_tb.dut.u_per.u_pd.iso_en, dbglab_tb.dut.u_per.rinc,
        dbglab_tb.dut.u_per.rempty, dbglab_tb.dut.u_per.u_fifo.rbin);
  end
endmodule

$ ../probes/run-probe.sh ../probes/pd_probe.sv periph_power 1 +FROM=70 +TO=95
dbglab: test=periph_power seed=1
  ns  cycle pwr_on iso_en rinc rempty rbin
 722     72    1      1     0     1    04
 759     76    0      1     x     1    04
 796     80    0      1     x     1    04
 833     83    0      1     x     0    04
 870     87    0      1     x     x    xx
 907     91    0      1     x     x    xx
 944     94    0      1     x     x    xx
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown

Line by line: at 722 ns the domain is on and isolated, and rinc is 0. At 759 ns (cycle 76) power goes off and rinc becomes X while `iso_en` is 1: that is the first wrong value. At 833 ns words arrive and the FIFO is no longer empty, so the X increment reaches the read pointer, which is X from 870 ns. The STREAM failure at cycle 431 is 355 cycles of consequence later.

The waveform window

$ DUMP=1 tb/run.sh periph_power 1
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
$ ls -l dbglab.vcd | awk '{print $5}'
55057

Here the whole dump is 55 KB, so it could be attached; on a real project the package gives the command, the window (700 to 950 ns) and the signals (pwr_on, iso_en, rinc, rempty, rbin, pstream_valid), and the receiver makes their own. Six lanes and 250 ns is a picture anyone can read; "see the dump" is not.

The package as sent

Notice what the package does not say: "the bug is the missing clamp". It states the first wrong value, the commit that changed it and the question only the owner can answer. The receiver may know a reason the clamp was removed that the giver does not.

The receiver accepts

Next morning, the power-domain owner builds the package's commit in a new folder and runs the three commands it names:

$ R=$(mktemp -d)/recv && build-history.sh $R > /dev/null && cd $R && git checkout -q 49c5f04
$ tb/run.sh periph_power 1; echo "exit $?"
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
exit 1
$ tb/run.sh periph_power 1 verilator
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008
$ run-probe.sh pd_probe.sv periph_power 1 +FROM=72 +TO=87
dbglab: test=periph_power seed=1
  ns  cycle pwr_on iso_en rinc rempty rbin
 722     72    1      1     0     1    04
 759     76    0      1     x     1    04
 796     80    0      1     x     1    04
 833     83    0      1     x     0    04
 870     87    0      1     x     x    xx
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown

Same lines, same cycle, same first wrong value, on a tree built from nothing. Now the tracker's owner changes, with a note: accepted, first wrong cycle reproduced, step: Fix. The receiver starts from cycle 76 and one line of RTL.

Hand-offs between disciplines

The package above is an RTL-to-RTL hand-off inside simulation. Across disciplines the fields stay the same, but what "replay" and "first wrong cycle" mean changes:

What the evidence says

The receiver spent three commands and a minute to get exactly where the giver stopped, and took the bug with nothing to ask. The package cost the giver perhaps twenty minutes, most of it already in their log. Compare that with the three-day thread: the difference is not effort, it is whether the evidence travels.

Traps

Related topics

• Part 11 · Debug as a Team

• Triage meetings that converge

• Recording everything to replay

• Bisection in space: the same failure localised to the power domain

• Bring-up: failure triage in the lab