During an OS boot on the emulator, network traffic stops after about 3 billion cycles. The software log shows a driver timeout. Somewhere, perhaps billions of cycles earlier, something went wrong in DCRouter, a DMA engine or the driver. Recording every signal of the chip for 3 billion cycles is impossible, and re-running the boot to look takes about 50 minutes each time. Debug at this scale is a search problem.
How do you debug a failure that appears after billions of cycles?
The obvious approach is simulation-style debug: dump the waves and look. At emulator scale, full waveforms for billions of cycles are far beyond any storage, and the failure's cause may be long before its symptom. The strategy has to be: detect as close to the cause as possible, capture only a window around it, and move the window to a platform where full visibility is affordable.
Technique | What it does | DCRouter hang example |
|---|---|---|
Assertions in hardware | Synthesised checkers fire in the cycle a rule breaks | DCRouter's R13 wake assertion (lesson 5.3) fires the moment an input fails to wake the router |
Triggers and trace buffers | Record a window of chosen signals around a trigger condition | Trigger on the assertion; capture 10,000 cycles of DCRouter's pins before it |
Monitors and transaction logs | Record transactions, not signals, for the whole run | A log of every packet in and out of DCRouter shows the last ones before traffic stopped |
Checkpoints (save and restore) | Save the full state periodically; restore near the failure | Restore a checkpoint taken 50 million cycles before the hang instead of rebooting |
Replay on a slower platform | Move the state or the stimulus window to simulation for full visibility | Run the last few thousand cycles in simulation with all signals dumped |
Bisection | Narrow the failing window by re-running from checkpoints | Find the first checkpoint after which the hang is inevitable |
Every checker carried up from block level (lesson 8.3) turns a vague software timeout into a precise event. In the opening story, with DCRouter's assertions synthesised onto the emulator, the failure would have been reported as "R13: in_vld 1000 did not wake the router" at the cycle it happened, perhaps billions of cycles before the driver noticed. Without them, the team starts from the timeout and searches backwards.
Decide what to capture: interface transactions everywhere, signals only around triggers.
Synthesise the assertions and route their failures to triggers.
Checkpoint regularly on long runs, so any failure can be revisited from nearby.
Keep runs reproducible: the same software, seeds and stimulus, so a failure can be re-run.
Practice | Purpose |
|---|---|
Synthesisable assertions and monitors on emulators | Failures detected at the cause |
Deep trace buffers with complex triggers | Windows of full visibility exactly where needed |
Save and restore with replay in simulation | Full waveforms for the critical cycles only |
Transaction-level logging of every interface | A complete, compact history to search |
A team (illustrative) spent two weeks on an emulator hang by rebooting the OS each time with a different set of signals captured, hoping to catch the cause. Each attempt cost over an hour and saw a few hundred signals. A synthesised assertion on the deadlocked interface, plus checkpoints every 100 million cycles, would have pinned the failing cycle on the first run and allowed a full replay in simulation from the nearest checkpoint.
It reports the exact rule and cycle when the router fails to wake, instead of the driver's timeout, possibly billions of cycles later.
Restoring it avoids re-running the first 3 billion cycles of the boot for each debug attempt.
At billions of cycles, debug is a search: detect early, capture a window, replay slowly.
Synthesised assertions, trace triggers, transaction logs and checkpoints make it possible.
Block-level checkers carried to the emulator turn software symptoms into precise causes.