A processor reads a register in a device that has crashed. With no timeout, the read waits for ever, the core waits behind it, and the operating system freezes with nothing in its logs. With a timeout, the read completes with an error after a few milliseconds, the driver logs "device not responding", resets the device, and the system carries on. The device failed the same way in both cases; only one system survived it.
How do you detect a request or a system that will never make progress, and what should happen next?
The obvious answer is to design so that hangs cannot happen: deadlock-free fabrics, compliant endpoints, every request answered. That is necessary, but devices crash, links fail, power domains switch off, and bugs escape verification. Something must turn "waiting for ever" into an error that software can handle. And the tool for that, a timeout, has two hard parts: choosing the value, and leaving the system in a clean state afterwards.
Mechanism | Watches | Fires when | Typical action |
|---|---|---|---|
Request (completion) timeout | One outstanding request | Its response has not arrived within a limit | Complete the request with an error; free its tag and buffers; log it |
Watchdog | A whole agent or system | It has not shown progress (a "kick") within a limit | Interrupt, then reset the agent or the system |
Progress monitor | A channel or queue | No handshake for N cycles while something is waiting | Raise an error with a snapshot of state for debug (lesson 9.10) |
Longer than the slowest legitimate response, including the worst case: power-state wake-ups, retries, congestion, device firmware that is slow on purpose. A timeout that fires on a healthy but slow device causes the failure it was meant to catch.
Shorter than what the system can tolerate: the requester and everything waiting behind it are stuck until it fires.
Different per class of target: on-chip registers answer in nanoseconds; a device behind a link that may retrain or wake from a low-power state may legitimately take milliseconds.
Illustrative device behind a link:
normal read latency ~1 us
link retraining after an error (worst case) ~10 ms
device wake from a deep power state ~20 ms
Timeout must exceed the worst legitimate case: > 30 ms; choose ~50 ms
An on-chip peripheral answering in < 1 us could use ~1 msThe late-response step is where recovery usually breaks: if the tag was freed and reused, a stale response can be matched to a new request and deliver wrong data silently (lesson 6.4). Designs mark timed-out tags as quarantined until the target is reset, or include a sequence number that late responses fail to match.
Protocol | Timeouts |
|---|---|
PCI Express | Completion timeouts are programmable in ranges from 50 µs to 64 s (the Device Control 2 register); functions must support 50 µs to 50 ms, and a timeout can be disabled. A timed-out request completes with an error and is reported through error logging. |
Interconnects and SoCs | Many provide per-target timeout monitors that answer a stuck request with an error response, so one dead peripheral cannot hang a core. |
Processors and systems | Hardware watchdog timers that reset the system if software stops kicking them. |
A design (illustrative) set a 1 ms completion timeout for a device that, in a deep power state, took 5 ms to wake. Every first access after idle timed out, the driver reset the device, and the device went back to sleep: a loop that looked like a hardware failure. The timeout was below the slowest legitimate response. Raising it to 50 ms, and keeping a shorter one for other targets, fixed it.
After every link error, requests time out even though the target is healthy; the system treats a slow recovery as a failure. The timeout should exceed 15 ms.
A late response for the old request could arrive and be matched to the new one, delivering wrong data with no error.
Timeouts turn requests that will never complete into errors; watchdogs reset systems that have stopped.
Choose a timeout above the target's worst legitimate latency and below what the system can tolerate.
On firing: error status, free everything, log, and discard late responses safely.