"You get a call from the lab. The new chip boots, runs traffic for a few minutes and then stops responding. Nothing on the console. What do you do?"
The interviewer has left out almost everything on purpose: how often, which traffic, whether the clocks still run. Part of the test is whether you notice. Where do you begin?
Keep the evidence. Do not reset or power-cycle until the state has been read: status registers, error logs, counters, a scan dump if the lab can take one. The first failing test on silicon lesson covers the order.
Questions about the report. Is the whole chip dead or one function? Do clocks toggle? Does the debug port answer? How often, and what was running?
A classification. Deadlock, livelock or starvation, from evidence, as in hang, livelock or starvation.
Who is waiting for whom. Every agent waits for something; write it down and look for a cycle (who is waiting for whom, finding the cycle).
Move it somewhere cheaper. Reproduce the conditions in simulation or on an FPGA, where every signal is visible (moving the failure to the cheapest stage).
A fix that breaks the cycle, and a detector. Not a longer timeout.
Three problems. The reset throws away the only snapshot of the bug, and a hang that takes minutes may take days to come back. "The waveform around where it hangs" does not exist on silicon, and in simulation it shows flat lines: a hang has no first wrong value to walk back from. And a timeout that "recovers" turns a deadlock into dropped data and hides the cause, as the watchdog lesson shows. The interviewer hears a candidate who would make the bug harder to find.
Say it as steps, and name each step. This is the shape of a two-minute answer:
Report. "Before anyone resets it: is the whole chip stuck or one interface? Are the clocks running and does the debug port answer? How long after boot, and with what traffic? Has it happened before?"
Observe without disturbing. "Through the debug port I read every status and error register, the interrupt state, any progress counters and hang detectors, and each bus master's outstanding requests. If the lab can scan-dump the stopped chip, I take one."
Classify. "I read the counters twice, a second apart. If nothing changed anywhere it is a deadlock. If attempts grow but nothing completes it is a livelock. If one agent waits while others progress it is starvation."
Localise with a wait-for table. "For each agent: what is it waiting for, and what is the evidence? If following the waits comes back to the start, there is the cycle. I look for the one arrow that is a design choice rather than a protocol rule, usually something holding a shared resource while it waits."
Reproduce cheaply. "I rebuild those conditions in simulation, starting from the scan dump or from the traffic and configuration, so the cycle forms where I can see every signal. If it used to work, I bisect the history."
Fix, prove, prevent. "The fix removes an arrow from the cycle. I predict what it changes, prove it on the repro and a sweep, and leave an assertion on the broken rule so the next one fires in tens of cycles, not when a watchdog gives up."
Steps 3 to 5 are easiest to show where everything is visible. dbglab's dma_burst_backpressure test hangs at HEAD; the case study hunts it in full. Here is how much of the answer each command gives you, in the order you would say it.
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
$ tb/run.sh dma_burst_backpressure 1 verilator
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
$ for s in $(seq 1 20); do tb/run.sh dma_burst_backpressure $s | tail -1; done | sed -E 's/seed=[0-9]+ //' | sort | uniq -c
20 FAIL test=dma_burst_backpressure cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008Twenty seeds, one line, one cycle: the seed does not matter, and two simulators agree. That rules out a race and a rare ordering before any waveform is opened. The watchdog line already names one waiting agent: the CPU, on a write to 0x3008 that never gets ready.
$ probes/run-probe.sh dbglab probes/hang_probe.sv dma_burst_backpressure 1 +FROM=2160 +TO=2164
dbglab: test=dma_burst_backpressure seed=1
cycle cpu v r addr grant lock dma dreq full sink_en
2160 1 0 00003008 10 10 WAIT 0 1 0
2161 1 0 00003008 10 10 WAIT 0 1 0
2162 1 0 00003008 10 10 WAIT 0 1 0
2163 1 0 00003008 10 10 WAIT 0 1 0
2164 1 0 00003008 10 10 WAIT 0 1 0
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008Nothing changes from cycle to cycle: no retries, no state moving. That is a deadlock, not a livelock. On silicon you would get the same answer by reading the progress counters twice.
The snapshot at cycle 2165 gives one row per agent: the CPU waits for the bus (valid=1 ready=0 on the write that turns the sink on), the fabric for its locked owner, the DMA (grant=10 lock=10), the DMA for FIFO space (state=WAIT dreq=0), and the FIFO and sink for the CPU's write (full=1 sink_en=0). The case study builds that wait-for table line by line. Read down the waits and you are back at the CPU: a cycle. Three arrows are protocol (a paced DMA waits for space, a stopped sink does not drain, a CPU waits for the bus). One is a choice: the fabric keeps the grant for an owner that is not using it. That is the sentence the interviewer wants.
$ probes/run-probe.sh dbglab probes/hang_probe.sv dma_burst_backpressure 1 +FROM=164 +TO=168
dbglab: test=dma_burst_backpressure seed=1
cycle cpu v r addr grant lock dma dreq full sink_en
164 1 0 00003008 10 10 RD_RSP 0 1 0
165 1 0 00003008 10 10 POP 0 1 0
166 1 0 00003008 10 10 LOAD 0 1 0
167 1 0 00003008 10 10 WAIT 0 1 0
168 1 0 00003008 10 10 WAIT 0 1 0
...
$ git bisect start HEAD v0.1 && git bisect run tb/run.sh dma_burst_backpressure 1
...
c4a71b585e287d527b9c2a827a3969bca0334445 is the first bad commit
...
bisect found first bad commit
$ git log -1 --oneline c4a71b5
c4a71b5 dma: keep the bus for a whole chunkThe DMA enters WAIT at cycle 167 still holding lock and the grant; every line after is the same. Four bisect runs name a performance change that taught the fabric to keep the grant while the owner holds lock. The case study shows why the obvious fix (drop lock in WAIT) still hangs, and the fix that works (re-arbitrate after every response).
Each command answered one part of the spoken answer: deterministic or not, deadlock or livelock, which agents and which arrow, when, and which commit. None of it needed a waveform viewer, and none of it needed a guess. On silicon the commands change (register reads through the debug port, a scan dump, a lab repro) but the questions are the same, which is why the interviewer cares about the questions.
"It never hangs in simulation." List what the failure needs (clock ratio, traffic mix, configuration, power state, temperature) and which stage has each ingredient. Rebuild the missing one in simulation, or start simulation from a scan dump of the stuck chip. See silicon vs simulation correlation.
"How do you tell a deadlock from a livelock without a waveform?" Two reads of the counters, some time apart: attempts and completions per agent. No change anywhere is a deadlock; attempts rising with no completions is a livelock.
"Why not just add a timeout?" A timeout ends the wait without removing the cause: a DMA that gives up drops a word, a bus that times out returns an error software did not expect. A timeout is a detector to report the hang, not a fix.
"The hang goes away when you connect the debugger." The probe changed the timing. Use passive observation (counters, a trace buffer frozen by the hang) and see heisenbugs.
"What would you want in the next chip?" Per-port progress timers, a sticky record of the first hang with a snapshot of who waits for what, readable without a reset (watchdogs and hang detectors).
They lost the only snapshot of the bug: the state of every agent at the hang, which a reset clears. If the hang takes days to recur, they also lost days. First come the questions (whole chip or one function, clocks, debug port) and the reads: status and error registers, counters twice, outstanding requests, a scan dump if possible. Only then reset and try to reproduce.
No. Holding valid high while waiting for ready is one attempt that never completes, not a repeated attempt. A livelock needs agents that act and undo, such as retries that are each refused. The probe lines are identical from cycle 167 on, so nothing acts: a deadlock. Count attempts as distinct tries, not cycles spent waiting.
dbglab at HEAD 49c5f04 (build-history.sh), 2026-10-18. tb/run.sh dma_burst_backpressure 1 (Icarus Verilog 13.0) and the same with verilator (5.048) print the same WATCHDOG line at cycle 2165; seeds 1 to 20 all fail with that line at that cycle.
hang_probe.sv (the case study's probe, compiled next to dbglab by probes/run-probe.sh: iverilog -g2012 -Wno-timescale -I tb ... hang_probe.sv, then vvp -n build/probe.vvp +TEST=dma_burst_backpressure +SEED=1) with +AT=2165, +FROM=164 +TO=168 and +FROM=2160 +TO=2164. Bisect in a scratch copy: git bisect start HEAD v0.1, git bisect run tb/run.sh dma_burst_backpressure 1 (git 2.54.0), four test runs, first bad commit c4a71b5.
Limits: a simulated hang in one small design; no silicon, FPGA or lab instrument was used, so the lab steps of the answer are described, not run. The interview wording and the weak answer are illustrative.