A colleague circulates a two-page proposal to replace DCLink with a faster version and asks for review comments by tomorrow. It reads well. Every paragraph sounds reasonable. It also contains ten flaws, some of which would show up in the first simulation and some only in silicon after months. Finding them is the principal engineer's core skill.
Can you find the flaws in a plausible protocol proposal, and explain each one's consequence?
DCLink v2 proposal
1. Flow control. Credits per hop. The credit round trip (flit sent to its credit
returned) is 24 cycles. Each receiver has an 8-flit buffer, giving 8 credits.
2. Credit pools. To save area, requests and completions share one credit pool.
3. Link reliability. CRC-16 per packet, 4-bit sequence numbers, go-back-N replay.
Up to 16 packets may be unacknowledged.
4. Replay buffer. Holds 8 packets. The receiver sends one cumulative ACK every 16
packets to save return bandwidth.
5. Replay timer. 16 cycles: if no ACK or NAK arrives in 16 cycles, replay.
6. Tags. 4 tags. Completions may return in any order, including two completions
for the same tag.
7. Fences. A fence completes as soon as it has been sent on the link.
8. Reset. Either side may be reset alone; the other side keeps its credit count
and replay buffer so traffic resumes quickly.
9. Power. The power controller gates the receiver's clock after 4 idle cycles.
10. Start-up. Each side adopts the partner's capabilities from the first training
word it receives.Write your review before opening the answers: for each flaw, the rule it breaks, what goes wrong, and the fix.
8 credits cannot cover a 24-cycle round trip: the sender stalls two thirds of the time, limiting throughput to 8/24 of the link (lesson 4.5, fact T2). Fix: at least 24 flits of buffer per hop, or accept a third of the bandwidth knowingly.
A shared pool lets requests consume every credit, so completions cannot get through, and requests waiting on completions never finish: message-dependent deadlock (lesson 9.4). Fix: separate pools, or reserve credits for completions.
Go-back-N with 4-bit sequence numbers allows at most 2^4 − 1 = 15 unacknowledged packets (fact T1). With 16, a full window wraps onto an old number and a lost ACK makes the receiver accept a duplicate. Fix: window 15, or 5-bit sequence numbers.
With an 8-packet replay buffer, the sender must stop after 8 unacknowledged packets, but the receiver only ACKs every 16: the link stalls for good after 8 (and item 3's window of 16 can never be used). Fix: ACK at least every 8 packets (and when idle), or a buffer that covers the window.
A 16-cycle replay timer is shorter than the 24-cycle round trip: every packet is replayed before its ACK can arrive, wasting bandwidth and risking replay storms (lesson 10.11). Fix: a timeout above the worst-case ACK round trip.
Two outstanding completions for the same tag cannot be told apart, so responses may be matched to the wrong requests (lesson 6.4). Fix: one outstanding request per tag (tags unique while outstanding), or same-tag completions in order.
A fence that completes when sent guarantees nothing about earlier writes having reached their targets: early acknowledgement breaks the ordering software relies on (lessons 7.4, 7.5). Fix: complete the fence only after all earlier writes are acknowledged by their targets.
After a one-sided reset, the reset side's buffers are empty and its sequence numbers restart, but the other side keeps stale credits and replay contents: overflow, wrong counts or rejected packets (lessons 4.8, 11.1). Fix: any reset of either side re-initialises credits and sequence numbers on both, through retraining.
Gating the clock without a handshake can cut off a packet in flight or arriving: lost data and hangs (lesson 12.4). Fix: a request/accept/deny handshake; accept only with nothing in flight, and wake on incoming traffic.
A single corrupted training word makes the two sides adopt different capabilities and the link never comes up (lesson 11.5). Fix: confirm by receiving the same word several times (and take the minimum of both sides, not the partner's value).
Cite the rule: "go-back-N allows 2^n − 1" is a fact, not an opinion.
Show the consequence with numbers: "8 of 24 cycles: a third of the bandwidth".
Propose a fix and its cost: reviewers who only find problems are less useful than those who make the design work.
Rank by severity: deadlocks and silent corruption first (items 2, 3, 6, 7, 8), then hangs (4, 9, 10), then performance (1, 5).
Review question by question; check every number with arithmetic against the rule it must satisfy.
Look for interactions between sections: window, buffer, ACK rate and timer must all agree.
Each comment: the rule, the consequence with numbers, the fix and its cost, ranked by severity.