A validation board running a network soak test starts failing after about six hours: a few packets an hour fail their end-to-end checksums, and DCRouter raises no alarm that anyone is watching. Two other boards ran the same test overnight without a failure. The team has STATUS and counter registers, a trace buffer on DCRouter's outputs, scan dump capability, and the RTL. This lesson walks through the failure from symptom to cause.
Walk me through debugging a silicon failure.
The obvious instinct is to stare at the failing board and guess. With almost no visibility, guesses multiply and each experiment takes hours. Silicon debug works as a disciplined narrowing: make it fail reliably, then reduce where, when and under what conditions it fails, until the remaining space is small enough to see or to simulate.
Step | What the team does | What it learns |
|---|---|---|
1. Capture | Read STATUS, ERR_CNT, PKT_CNT and the soak test's log of failing packets before resetting | Every failing packet went to output 2 and arrived split: its first flits, then the rest as a separate packet with a nonsense header, not necessarily on output 2 |
2. Reproduce | Run traffic only to output 2, at maximum rate, with long idle gaps | Corruption now within about 20 minutes on every board |
3. Functional or electrical | Repeat at high and low voltage, hot and cold, half clock speed | Same failure everywhere: functional, not a margin |
4. Localise | Vary which inputs send to output 2; change idle gap length | Needs idle gaps longer than 8 cycles in the middle of a packet: the sleep logic |
5. Capture | Trigger the trace on sleeping rising while output 2 is locked | The router entered sleep with output 2 locked mid-packet; the lock was dropped, so the rest of the packet was routed as a new one |
6. Reproduce in RTL | Replay the captured window in simulation | The idle detector ignores locks: Bug Zoo BUG_20's class, escaped in this story |
In the course's actual DCRouter, the R13 assertions and the back-pressure tests catch BUG_20 long before silicon. The walk-through shows how the same bug would be found if they had not existed: in hours of narrowing instead of seconds of simulation.
Capture before reset: the failing state is the most valuable evidence and disappears at reset.
Reproduction first: a failure that happens every 20 minutes can be debugged; one that first appears after six hours, on one board in three, barely can.
Separate functional from electrical early: vary voltage, temperature and frequency (lesson 11.5).
Change one thing at a time, and keep a log of every experiment.
End in RTL: a root cause is confirmed when the RTL reproduces the failure and the fix removes it.
Practice | Purpose |
|---|---|
Failure capture scripts run before any reset | State preserved for analysis |
Reproduction suites shrinking failing tests | Fast, reliable failures to work on |
Shmoo and margin experiments on every new failure | Functional and electrical causes separated |
Replay of trace windows and scan dumps in simulation | Root cause confirmed in RTL |
A team (illustrative) reset a hung board immediately each time to restart the test, hoping to catch the failure under a logic analyser. Each reset discarded the frozen state that a scan dump could have read. After two weeks they changed procedure: capture first, then reset. The next failure's scan dump showed a state machine in an illegal state in its first hour of analysis.
It failed identically at high and low voltage, hot and cold, and at half clock speed; electrical margin failures move with those conditions.
It confirms the root cause and lets the fix be verified with full visibility before any change to silicon or software.
Silicon debug is a disciplined narrowing from symptom to root cause.
Capture before reset, reproduce reliably, separate functional from electrical, localise, capture the window.
Finish by reproducing in RTL and adding the check that would have caught it.