The question, as it is asked

You have just described a bug you found and the one-line fix. The interviewer nods and asks: "How do you know your fix works?"

It sounds like a closing pleasantry. It is often the question the whole conversation was building to. What do you say?

What the interviewer is listening for

1. Fail before, pass after, on the same command. The repro is the proof's anchor; without it a pass means nothing (the debug loop).

2. Evidence at the cause, not only at the symptom. A check on the rule the bug broke, which fired before and is silent after (assertions as always-on probes).

3. A wider net that varies what the bug needs. Seeds, configurations or a directed sweep, chosen from the mechanism, with a stated bound if it is rare (statistics of rare failures).

4. Nothing else moved. The regression's other results are unchanged, so the fix did not break or hide anything.

5. Something stays behind. A regression test and an assertion that would catch it again, and ideally a reason it escaped.

A weak answer, and why it fails

A test passes after a change for three reasons: the bug is fixed, the bug is hidden (the change moved the symptom out of the test's view), or the test no longer exercises the bug. "It passed" cannot tell them apart. "The regression was green" is weaker still if the regression never failed on this bug before, and it says nothing about the simulators or stimulus that did. The interviewer hears a candidate who will be surprised by the bug again.

A strong answer, out loud

1. The repro. "The command that failed now passes, in both simulators that showed it, and it fails again if I take the fix out."

2. The cause. "An assertion on the rule the bug broke fired at the first wrong cycle before the fix. After it, the assertion is silent in every test, not just the one that failed."

3. The net. "I swept what the bug depends on, not just the seed number. If it was rare, I sized the sweep to rule out the old rate."

4. The rest. "The regression's other results did not change: no new failures, no failures that disappeared without explanation."

5. What stays. "The failing test is in the regression and the assertion is in the bench, so the next change that breaks the rule fails in minutes."

Do it: one fix, before and after

dbglab's periph_power test powers the stream sink down and up with words waiting. At HEAD it fails, and the two simulators fail differently, which is a clue in itself: Icarus models an unpowered output as X, Verilator as noise. The cause is that the clamp was removed from the sink's read-increment, rinc, on the argument that it stays inside the peripheral; it feeds the FIFO, which is not in the switched domain. The fix puts the clamp back:

// snippet: the fix, rtl/periph_pd.sv line 34 (HEAD has: assign rinc = rinc_p;)
  assign rinc      = iso_en ? 1'b0  : rinc_p;

1. The repro, in both simulators

$ tb/run.sh periph_power 1                 # HEAD
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=431: STREAM: pstream_valid is unknown
$ tb/run.sh periph_power 1 verilator       # HEAD
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008
$ cd ../b5fix                              # the same tree with the fix
$ tb/run.sh periph_power 1
dbglab: test=periph_power seed=1
PASS test=periph_power seed=1 cycles=469 time=4686 ns
$ tb/run.sh periph_power 1 verilator
dbglab: test=periph_power seed=1
PASS test=periph_power seed=1 cycles=469 time=4686 ns

Two signatures before, one pass after. A fix proven only in Icarus would have left you unsure about the Verilator hang; with both, you know the two symptoms had one cause.

2. A check at the cause

The symptoms appear at cycles 431 and 2421. The rule was broken much earlier. This check states the rule (every output of the switched domain is 0 while iso_en is high) and compiles next to dbglab without changing it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// not-run: needs dbglab's sources and bench; compile it next to them (command in the Evidence box)
`timescale 1ns/1ps
// iso_check: every output of the switched domain must be 0 while iso_en is high.
// Compiled next to dbglab; reads the design by hierarchical names and changes nothing.
module iso_check;
  always @(posedge dbglab_tb.pclk) begin
    #0.5;
    if (dbglab_tb.dut.u_per.u_pd.iso_en === 1'b1
        && (dbglab_tb.dut.u_per.u_pd.rinc !== 1'b0 || dbglab_tb.dut.u_per.u_pd.out_valid !== 1'b0)) begin
      $display("FAIL cycle=%0d: ISO: iso_en=1 but rinc=%b out_valid=%b (pwr_on=%b)", dbglab_tb.cycle,
               dbglab_tb.dut.u_per.u_pd.rinc, dbglab_tb.dut.u_per.u_pd.out_valid, dbglab_tb.dut.u_per.u_pd.pwr_on);
      $fatal(1, "isolation clamp missing");
    end
  end
endmodule

$ probes/run-probe.sh dbglab probes/iso_check.sv periph_power 1
dbglab: test=periph_power seed=1
FAIL cycle=76: ISO: iso_en=1 but rinc=x out_valid=0 (pwr_on=0)
$ probes/run-probe.sh b5fix probes/iso_check.sv periph_power 1
dbglab: test=periph_power seed=1
PASS test=periph_power seed=1 cycles=469 time=4686 ns
$ ./iso-all.sh dbglab           # iso_check on all seven tests, seeds 1 to 3
ISO fired: periph_power seed 1
ISO fired: periph_power seed 2
ISO fired: periph_power seed 3
dbglab: ISO fired in 3 of 21 runs
$ ./iso-all.sh b5fix
b5fix: ISO fired in 0 of 21 runs

Before the fix the check fires at cycle 76, 355 cycles before Icarus's symptom and about 2,300 before Verilator's. After it, the rule holds in every run of every test. That is evidence the cause is gone, not just the symptom: a fix that only masked X on pstream_valid would have left this check firing.

3. A seed sweep, and what it does not prove

$ for s in $(seq 1 50); do tb/run.sh periph_power $s | tail -1; done | sed -E 's/seed=[0-9]+ //' | sort | uniq -c
  50 FAIL test=periph_power cycle=431: STREAM: pstream_valid is unknown
$ cd ../b5fix; for s in $(seq 1 50); do tb/run.sh periph_power $s | tail -1; done | sed -E 's/seed=[0-9]+ //' | sort | uniq -c
  50 PASS test=periph_power cycles=469 time=4686 ns

Fifty seeds before, fifty after, the same in Verilator. It looks like strong evidence. Look again at the cycle numbers: all fifty runs fail at cycle 431 and pass in 469 cycles. In periph_power the seed only picks the data words; the power sequence and its timing never change. This was one run, fifty times. A sweep that varies what this bug needs would vary how many words are in flight when power goes off and how long it stays off. The bench cannot do that without a change, so say so in the review rather than quoting "50 seeds clean". The supply-model sweep in the missing isolation is one sweep that does vary what the bug needs: the value of an unpowered output.

4. Nothing else moved

$ tb/regress.sh icarus 1 2 3 | grep -E "FAIL|failing"      # HEAD
irq_w1c                  seed 1      FAIL
...
dma_burst_backpressure   seed 3      FAIL
periph_power             seed 1      FAIL
periph_power             seed 2      FAIL
periph_power             seed 3      FAIL
9 failing run(s)
$ cd ../b5fix; tb/regress.sh icarus 1 2 3 | grep -E "FAIL|failing"
irq_w1c                  seed 1      FAIL
...
dma_burst_backpressure   seed 3      FAIL
6 failing run(s)

The three periph_power runs moved from FAIL to PASS and nothing else changed: the other six failures are two known bugs with their own signatures, and random seeds 1 to 60 fail 19 times in both trees. A fix that turned an unrelated test green would be as suspicious as one that turned one red.

5. What stays behind

periph_power already fails on this bug in both simulators, so it is the regression test. The clamp rule belongs in the bench as an assertion on every power domain's outputs, which fires at cycle 76 in any test that powers down, including tests not yet written. The escape question is the review one: the commit removed a clamp because the signal "stays inside the peripheral"; the question that catches it is which domain is the receiver in? Part 9 takes regression tests, assertions and escape analysis further.

What the evidence says

The strongest line is the check at the cause; the weakest is the one that looks biggest. Saying which is which is what makes the answer senior.

Follow-up questions

• "The bug failed once in 500 runs. How many clean runs do you need?" Decide the rate to rule out first; about 3/p clean runs for 95%. Better, prove on a stressed repro that fails every run. A test fails once in 500 runs rehearses it.

• "Could you prove it formally?" If the rule is local, yes: an assertion like the clamp rule on one module is a small formal job, and a proof covers every sequence, not just the bench's. See formal counterexamples as traces.

• "How do you know you fixed the cause and not the symptom?" The check at the cause is silent, and the hypothesis predicted something before the fix that you then saw, as in the debug loop's Test step.

• "What if the fix is in silicon, as a metal change?" The same evidence in simulation on the ECO netlist, an equivalence check of the change, and the failing lab test passing on the fixed part (equivalence failures).

Traps

Related topics

• Part 12 · Interview Prep

• Part 9: fix, prove, prevent

• Power-intent bugs

• Case study: the missing isolation, proved under every supply model

• Statistics of rare failures

• SystemVerilog: assertion debug