Part 10 answers: What does the whole loop look like on real bugs?
Twelve real hunts on dbglab and small designs, each told as a mystery: the report first, then the hunt step by step with real output, then the fix and what it teaches. Try each one before reading the answer.
Run the whole debug loop on unfamiliar bugs.
Recognise which step won or lost each hunt.
Explain a hunt in an interview in two minutes.
12 lessons, about 152 min
The part on one page, taken from each lesson's Remember box.
The Off-by-One That Fails at Wrap
Wrong by exactly a power of two is a width signature. Compute requested minus done for every case before you read any RTL.
A stale value read back usually means a write never happened. Watch the address on the bus before you suspect the memory.
Two simulators, two failing lists: split by signature first. A two-state run can hide one bug and show the other cleanly.
Rule of thumb: count what the hardware did against what was asked, for every run, before reading the code.
The Reset Value That Changed
A regression from a known-good release is a history question first. Check both ends, then bisect: eleven commits take four runs.
A signature is the message, not the cycle. Strip numbers that move between commits, or bisect calls a bad commit good.
A commit message that says "no functional change" is a claim, not evidence. Test it against its parent.
Rule of thumb: if it used to pass, ask history first, and make the bisect script test this bug's message, nothing more and nothing less.
The X That Hid for Thousands of Cycles
A pass in a two-state simulator says nothing about X. Rerun with random initial values before you call a four-state failure a quirk.
Read the scoreboard's message for who last wrote the word. The write, not the read, is where the hunt starts.
An X can sit in memory for any length of time, or be overwritten unseen. A clean run is not proof that no X was written.
Rule of thumb: check for X where a value enters the datapath, not where it is finally read.
The Race Two Simulators Disagree On
A result that changes with the simulator, with the source order, or from one edge to the next is a race until proven otherwise.
Look for one process writing a variable with a blocking assignment on an edge while another reads it on the same edge.
A print inside a racing process is not evidence of what the other statements saw. Remove the race, then look again.
Rule of thumb: never let one process read on a clock edge what another process writes with a blocking assignment on the same edge.
The Bus Value That Was Never Sent
A multi-bit value that crosses bit by bit can be captured as a mix of old and new bits: a value the sender never had.
Plain RTL simulation changes every bit at once. To see this bug, model the difference in arrival, and say what the model does not prove.
Gray code works because one bit changes per step. It still needs the skew between bits bounded to less than one source clock.
Rule of thumb: never let more than one bit of a crossing value change at a time, and bound the skew between its bits.
The Response to the Wrong Master
A response is routed by its tag, so a wrong destination means the tag's owner was wrong when it came back. Find when the owner was written.
A rolling tag counter is safe only if responses return in the order requests left. An out-of-order slave breaks that silently.
Check the rule at the moment it is broken: never hand out a tag that is still in flight. That fires cycles before any scoreboard.
Rule of thumb: a tag is free when its response is back, not when a counter says it is its turn.
The Deadlock Under Back-Pressure
A hang has no first wrong value. Snapshot every agent at the hang and write down what each one is waiting for.
A deadlock is a cycle of waits. Break it where an agent holds a shared resource while waiting for something that needs it.
A fix must break the cycle in every case. Predict what a candidate fix changes, then run it; this one bug has a fix that looks right and still hangs.
Rule of thumb: never hold a grant while waiting for an event that may need the bus.
The Credit Leak That Hangs After an Hour
A rate that sags and then stops is a leak, not a deadlock. Count what is conserved; do not wait for the hang.
Turn up the rate of each rare event in turn. The one that brings the hang closer is in the cause; the hang time scales with it.
Every event that throws work away must give back what that work held. Ask which words a flush, drop or retry covers in its own cycle.
Rule of thumb: for every event that discards work, count what it discards in its own cycle, and check the credit law every cycle.
The 2x Throughput Regression
A clean factor such as 2x points at a structure, often one idle cycle per transfer. Count the idle cycles; if they equal the transfers, you know the shape.
Bisection needs a test that fails. For a performance bug, write one that fails on cycles, then git bisect works as it does for a wrong value.
Registering a control signal moves its decision one cycle later. Ask what the circuit does in that cycle; often it is idle.
Rule of thumb: every timing fix that adds a register gets a cycles-per-transfer check before it merges.
The Missing Isolation
One bug, two simulators, two symptoms is normal for power and X bugs. Put both runs on one timeline before splitting the ticket.
A watchdog reports late by its limit. The last progress was at cycle 420, 2,001 cycles before the WATCHDOG at 2421: the moment Icarus also fails.
The supply model decides what an unclamped output does. Prove a power fix under X, noise, 0 and 1; a model that drives 0 hides the bug.
Rule of thumb: before you split one failing test into two bugs, put both symptoms on one timeline and look for the same moment.
The Test-Mode Leak
A register that changes with its enable off is being clocked. On a gated clock, look at the gate's test enable before the data path.
When a failure needs one boot step, compare the run with and without it. What that step leaves behind is the suspect list.
Drive test controls from what the test logic is doing (busy), not from what software asked for (enable). Check them in every functional test.
Rule of thumb: after any test feature runs, every test control must return to its functional value: check it, do not assume it.
The Testbench That Was Wrong
A mismatch says two things disagree, not which one is wrong. Find a referee that shares no code with either side.
The spec often gives a check cheaper than a model: a property every output must satisfy. Run it as a second, independent checker.
After fixing a bench, break the design on purpose and make sure the bench still fails.
Rule of thumb: before you debug the design, prove the checker is right on the failing case.
Practice question: Counter with wrap and saturate modes
Practice question: Design a simple DMA controller
Practice question: Debugging rare seed failures in regression
Code lab: DMA control FSM
Code lab: 8-bit up counter
Interview: run a mock interview on this topic