The periph_power failure is fixed. The debug log tracked the hunt to one line of periph_pd.sv, the clamp on rinc is back, and both simulators pass. The ticket has one field left, Root cause, and the first draft says: Engineer removed the isolation clamp on rinc by mistake. Action: be more careful with power domains.
Everything in that sentence is true, and none of it will stop the next missing clamp. What made this bug possible, and what would have stopped it?
Most root-cause analyses end early, in one of three places. Each one feels like an answer, and each one leaves the process exactly as it was.
Blame the tool. "The X comes from the supply model's pessimism; real silicon has no X." It ends the analysis at the symptom, and here it is plainly wrong: Verilator has no X and hangs instead.
Stop at the first technical cause. "rinc had no clamp; clamp added." That is the fix, not the cause. It protects this one port and says nothing about the next one.
Human error. "The author made a mistake." Always true, never useful. People make mistakes at a steady rate; the question is what let this one reach the main branch.
A fourth failure looks thorough: twelve "contributing factors" with no evidence, ending in improve communication. Nobody can check it, so nobody acts on it.
Start from the symptom as observed, the exact failing line, not from the fix you already know.
Each answer is a fact with evidence next to it: a run, a diff, a log line, a file. If you cannot show it, write guess beside it and go and get the evidence before you ask the next why.
Follow two chains. The mechanism chain asks why the hardware did this. The escape chain asks why the bug got through. They fork at the commit that introduced it.
Do not stop at a person. When the answer is someone, ask what let their mistake survive: what they could not see, and what did not check them.
Stop when the answer is something you can change, then test it: would changing it have stopped this bug, and would it stop others like it? If only this one, go one level deeper.
Say what you do not know. History records commits, not intentions. A guess presented as a cause poisons every action built on it.
Five is a habit, not a rule. Some chains end at three; some fork into two causes, which is fine as long as each one has its evidence.
Each rung answers the one above and carries its own evidence. The marks on the right are where most analyses stop.
$ tb/run.sh periph_power 1
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
$ tb/run.sh periph_power 1 verilator
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008Two symptoms, and the first evidence against blame the tool: a two-state simulator with no X still fails, by hanging.
Answer: the FIFO's read pointer is X from cycle 87, while the domain is off and 340 cycles before any output is wrong; the X enters through rinc with isolation on (the debug log has the probe and its output). The outputs fail because the FIFO already has.
$ git blame -L 33,34 rtl/periph_pd.sv
a60b6f71 (dbglab 2026-03-08 10:00:00 +0000 33) // Isolation: every output of the domain is clamped to 0 while iso_en is high.
122e88a7 (dbglab 2026-03-16 10:00:00 +0000 34) assign rinc = rinc_p; // internal: only feeds our own async FIFOAnswer: rinc leaves the switched domain with no clamp, since commit 15 on 2026-03-16. Line 33, a week older, still says every output is clamped: the code and its own comment disagree. This is the first technical cause, and the fix. The draft ticket stopped here, plus one sentence about the engineer.
$ git show 122e88a
...
periph_pd: isolate only the outputs that leave the peripheral
rinc only feeds our own async FIFO, so it does not need a clamp.
...
- assign rinc = iso_en ? 1'b0 : rinc_p;
+ assign rinc = rinc_p; // internal: only feeds our own async FIFO
...
$ grep -n "u_fifo (\|u_pd (" rtl/periph_slave.sv
52: async_fifo #(.DW(32), .AW(AW)) u_fifo (
60: periph_pd u_pd (.pclk(pclk), .prst_n(prst_n), .pwr_on(pctrl_q[PCTRL_PWR_ON]), .iso_en(pctrl_q[PCTRL_ISO_EN]),Answer: the author believed rinc stayed inside the domain. In the module hierarchy it does: the FIFO is "our own", instantiated in the same periph_slave. In power it does not: u_fifo sits beside u_pd, not inside it, so its read side is always on. The word internal mixed up two different boundaries.
This is where human error stops. It is a good finding, because it names the confusion exactly. But a confusion an honest engineer can make will be made again. Ask what let it survive.
Answer: nothing in the repository says which signals cross the power boundary, or which supply powers each reader, except comments. There is no list of boundary ports, and the bench never checks the isolation rule, only the stream pins far downstream. A reviewer of commit 15 had what the author had: a one-line diff and a convincing message. The evidence here is an absence, checked by reading periph_pd.sv, periph_slave.sv and tb/.
periph_power fails on every seed at commit 15 and passes at commit 14. So the regression saw it the day it landed:
$ git checkout -q b357bd8 && tb/regress.sh icarus 1 2 3 > ../runs/regress-b357bd8.txt; tail -1 ../runs/regress-b357bd8.txt
6 failing run(s)
$ git checkout -q 122e88a && tb/regress.sh icarus 1 2 3 > ../runs/regress-122e88a.txt; tail -1 ../runs/regress-122e88a.txt
9 failing run(s)
$ git log --format="%h %ad %s" --date=short 2e63466..HEAD
49c5f04 2026-03-17 tb: report cycles and sim time in the PASS line
122e88a 2026-03-16 periph_pd: isolate only the outputs that leave the peripheral
b357bd8 2026-03-15 csr: no reset on the DMA address and data registers
03ebdc1 2026-03-14 dma: one width for the beat counters
bc2bb34 2026-03-13 dma: fill mode
c4a71b5 2026-03-12 dma: keep the bus for a whole chunk
27acd30 2026-03-11 periph: deepen the async FIFO to 16 entries
026f8bf 2026-03-10 csr: collect register reset values in dbglab_pkgAnswer: the regression had been red since commit 9, six days earlier. Commit 15 turned 6 failing runs into 9, and the script reports only a count; 9 looks like 6 to someone who has stopped reading it. Be honest about the gap: the history cannot tell us whether the author ran the regression. It does not need to. Either way nothing compared today's failures with yesterday's, and that you can change.
A cause earns an action only if the action would have caught this bug. So test each one against the history before the meeting agrees to it. The escape-chain cause is the easiest to test: compare the regression output before and after the commit.
$ diff ../runs/regress-b357bd8.txt ../runs/regress-122e88a.txt
16,18c16,18
< periph_power seed 1 PASS
< periph_power seed 2 PASS
< periph_power seed 3 PASS
---
> periph_power seed 1 FAIL
> periph_power seed 2 FAIL
> periph_power seed 3 FAIL
22c22
< 6 failing run(s)
---
> 9 failing run(s)That diff is the whole new-failure gate: any line that went from PASS to FAIL blocks the commit. Run the same comparison at every commit of dbglab and it fires three times: commit 9 (0 to 3 failing runs, bug B1), commit 11 (3 to 6, B2) and commit 15 (6 to 9, B5). One process cause, three of the five bugs. That is the sign of a real root cause: it explains more than the bug you started from.
Cause (rung) | Action | Checked how | Would it have caught |
|---|---|---|---|
Missing clamp (2) | Restore | periph_power passes in both simulators; probe silent | this bug only |
Wrong comment (2) | Make line 34 and line 33 agree | review | nothing on its own |
No boundary list, no check (4) | List every port of each switched domain with its reader's supply; generate the isolation check from it | the always-on check in power-intent bugs fires at cycle 76, 355 cycles before the bench | B5 and any future unclamped port |
Red regression is normal (5) | Block a commit that turns any PASS into FAIL; keep known failures in a list, each with its ticket | the diff above, at every commit | B1, B2 and B5 on the day they landed |
Notice what is missing: "be more careful". It has no check and catches nothing. Where the action is a new assertion, the assertion that would have caught it shows how to write and prove one; why the other two bugs, B3 and B4, got through a regression that was run is a different question, answered by escape analysis.
It is blame the tool, stopped at the symptom. Ask for the evidence that the X is pessimistic rather than real: does a two-state run pass? For B5 it does not, it hangs, and the clamp fix makes both simulators pass, so the X was a correct warning. Then ask the next why: what drove the X, and why was it allowed to reach the FIFO?
Partly. It can be checked and has an owner, but would it have caught B5? The power lead would have seen the same one-line diff and convincing message, and the reader's supply is not in it. A boundary list with a generated isolation check catches the next missing clamp whoever reviews it, so it is the stronger action; the review rule can sit beside it.
dbglab at HEAD 49c5f04 rebuilt with dbglab/build-history.sh, 2026-10-18. tb/run.sh periph_power 1 (Icarus Verilog 13.0) and the same with verilator (Verilator 5.048) as shown. first_x.sv compiled next to dbglab with iverilog -g2012 -Wno-timescale -I tb <rtl/files.f> tb/bus_rules.sv tb/dbglab_tb.sv first_x.sv, run with vvp -n ... +TEST=periph_power +SEED=1: first X on rinc at cycle 76, rbin at 87, out_valid at 427.
git 2.54.0: git show 122e88a, git blame -L 33,34 rtl/periph_pd.sv, git log --date=short 2e63466..HEAD. tb/regress.sh icarus 1 2 3 at every commit: 0 failing runs to commit 8, 3 at 9 and 10, 6 at 11 to 14, 9 at 15 and 16; the diff of the outputs at b357bd8 and 122e88a as shown. With assign rinc = iso_en ? 1'b0 : rinc_p; on a scratch clone, periph_power seed 1 passes in both simulators (cycles=469) and the probe prints nothing.
Limits: dbglab's history is a teaching history with one author. It cannot show whether the regression was run before commit 15, or what a reviewer saw, so the lesson says so where it matters. The meeting practice in the lens is experience, not measured here.