This scenario is fictional and illustrative: the design, dates and numbers are invented. The method is general. An escalation is a bug with a deadline attached. The technical work is the hang triage of Part 12; the staff engineer's job is to make sure the right people do it in parallel, that everyone knows what is known, and that a decision is made on time whatever the root cause turns out to be.
The nightly eviction-stress regression on the new RTL tag shows 3 watchdog hangs in 30,000 seeds (illustrative). The previous tag had none in the same profile. All three hangs involve a contended line X between C0 and C1 while C0 also misses on another line Y. Tape-out is in 14 days; the last RTL drop is on day 8, because a full regression takes three days and freeze needs a clean one.
Step | Owner | Output, and by when |
|---|---|---|
Reproduce: rerun the 3 seeds, save full traces and dumps at the hang | Verification engineer A | Seeds that hang every time, day 1 |
Freeze and dump MSHRs, H's pending entries, queues and credits; draw who waits for what | Home-controller designer B | Classified hang (lost message, deadlock, livelock, starvation), day 2 |
Diff the two RTL tags for changes on the paths in the chain | Fabric designer C | Candidate change list, day 2 |
Build a small-model check of the suspected mechanism | Formal engineer D | Stuck state found or ruled out, day 4 |
Hold the schedule options: revert, fix, de-feature | Staff engineer (incident lead) | Decision on day 5 |
One owner per row, one date per output. Steps 1 to 3 run in parallel. The method for step 2 is Hang Triage; reproducing on demand is Reproducing a Race.
The dump matches worked case B of Hang Triage, with its roles: C1's GetM for X holds H's only pending entry P; C0's GetS for Y holds an ingress slot Q while waiting for P; C0's egress slot holds Data(1) for X and waits for space in Q. The chain closes on itself and nothing is lost: the newest value 1 is safe in C0's egress slot. The diff found why it is new: a late performance change let responses use request-queue slots when the response queue was full. That undoes the rule of A Protocol Deadlock: responses must always sink, because each pending entry reserves its response slot when it is allocated.
Option | Evidence needed by day 11 | Risk |
|---|---|---|
Revert the buffer-sharing change | Full regression on the reverted tag, which matches the last known-good behaviour | Lose the change's illustrative 3% bandwidth gain |
Fix it: borrow only free, unreserved slots | Full regression, new directed test, small-model check of the new rule | New logic in a resource path two weeks before tape-out |
Chicken bit to disable sharing | Full regression with the bit set | The bit itself is new RTL |
The lead chose the revert: it restores a design whose evidence already exists, and the performance gain can return in the next project with proper verification. The deadline was fixed on day 0, so the decision did not wait for a perfect fix.
Evidence | Why it is needed |
|---|---|
Mechanism explained: the dump's cycle matches the reverted change | A fix without a mechanism may hide the hang rather than remove it |
Directed test hangs on every run before the revert and passes after | Shows the test reaches the cycle, so its pass means something |
Small-model search with finite buffers finds a stuck state with buffer sharing and none without it (Small Models and Parameters) | A reachable state with pending work and no enabled event is a deadlock; exhaustive search covers timings the seeds missed |
The three original seeds pass | Necessary, never sufficient |
Full regression on the final tag: zero watchdog hangs and zero value-checker failures | Closes the item under Closure Evidence; a cleared hang must not have become a wrong value |
Cover bin 'response queue full while a request holds the only pending entry' hit | Proves the stress still reaches the dangerous condition |
Deadlock freedom is a property of states (no reachable stuck state), so the same breadth-first search can check it; a livelock or starvation claim would need the fairness arguments of Safety Versus Liveness.
Field | Content | Day 2 example (illustrative) |
|---|---|---|
Status | On track, at risk or blocked, in words | At risk: root cause not yet known |
What we know | Facts with evidence | 3 seeds reproduce; the dump shows a closed cycle P, egress slot, Q |
What we do not know | Open questions | Which RTL change created the cycle |
Actions and owners | One owner and date per action | C: tag diff by day 3. D: small-model check by day 4 |
Decision needed | What, from whom, by when | Revert, fix or de-feature, program lead, day 5 |
Impact on tape-out | Dates at risk | None if decided by day 5 |
Next update | When | Tomorrow, same time |
Yes, with care. The cycle needs only one entry in each resource, so one-entry queues are the hardest case for this mechanism, and the small model makes every timing reachable. But the claim must be stated as being about that configuration; deeper queues add states, and an argument (or a run with two entries) should cover why more entries do not create a new cycle.