Three interfaces, three flat lines

A new accelerator is wired into a subsystem through three valid/ready interfaces: a data stream in, a second stream from a converter, and a request/response port to a lookup table. The first simulation after integration hangs at once. Each interface was unit-tested on its own against a well-behaved model, and every unit test passed.

All three waveforms end in flat lines. Are these three bugs, or one? And which side of each interface is wrong?

What people usually do, and why it fails

A hung handshake usually gets a skid buffer, a register slice, or a sink tied to ready = 1 "to see if data comes out". Each changes the timing, and sometimes the hang moves or disappears. None of them names the broken rule, and tying ready high removes back-pressure, which the design needs. The unit tests passed because each block was tested against a model that never waits: a model sink that raises ready before valid hides pattern 1, and a model that always takes responses hides pattern 3.

The rules are in the RTL course (rules that keep flow control safe) and in Protocol First Principles (the valid/ready rules, channel dependencies). This lesson is about recognising which one was broken, from a hang.

The method: read the levels, then the dependency

1. At the hang, print valid and ready of every channel, with how many transfers each has done. A channel that finished is not part of the hang.

2. Both low: each side waits for the other to start. Find the side whose output depends on the other's: a source that waits for ready before raising valid is wrong (pattern 1). If the dependency is combinational in both directions, it is a loop (pattern 2): run lint.

3. Valid high, ready low on one channel: ordinary back-pressure. Follow the sink's wait, as in who is waiting for whom.

4. Valid high, ready low on two channels between the same agents: each channel's ready waits for the other channel (pattern 3). Find the agent that will not take what it is owed until it can send.

5. Name the rule broken, and fix the side that broke it. In pattern 1 the sink's "ready after valid" is legal; the source is wrong.

On the debug loop this is Localise (which side, which rule) and Prevent (which check catches the next one before simulation).

The picture: the arrow that should not exist

Do it: all three in one bench

Three pairs run side by side. BAD=0 is correct everywhere; BAD=1, 2 or 3 breaks that one pattern. The sink in pairs 1 and 2 uses ready = valid, which the rules allow. The responder takes no new request while its one response waits, a legal one-deep design. When no channel has handshaken for 100 cycles, the bench prints every channel's levels.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

// Three valid/ready deadlock patterns. Each pair is correct unless BAD names it.
// vr_src: sends N words. WAIT_RDY=1 (pattern 1) offers valid only after it has seen ready.
// COMB_RDY=1 (pattern 2) gates valid with ready combinationally.
module vr_src #(parameter bit WAIT_RDY = 0, parameter bit COMB_RDY = 0, parameter int N = 50) (
  input  logic       clk, rst_n,
  output logic       valid,
  input  logic       ready,
  output logic [7:0] data
);
  logic       v_q;
  logic [7:0] n_q;
  assign valid = COMB_RDY ? (v_q && ready) : v_q;
  assign data  = n_q;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin v_q <= 1'b0; n_q <= '0; end
    else begin
      if (valid && ready) n_q <= n_q + 8'd1;
      if (!v_q || (valid && ready))
        v_q <= (32'(n_q) + 32'(valid && ready) < N) && (WAIT_RDY ? ready : 1'b1);
    end
endmodule

// vr_sink: takes words; ready follows valid (allowed: ready may wait for valid).
module vr_sink (input logic valid, output logic ready);
  assign ready = valid;
endmodule

// Pattern 3: a requester and a responder with one request and one response channel.
// The responder holds one response and takes no new request until it is taken.
// HOLD_RSP=1: the requester takes a response only when its own request is not stuck.
module requester #(parameter bit HOLD_RSP = 0, parameter int N = 50) (
  input  logic       clk, rst_n,
  output logic       req_valid,
  input  logic       req_ready,
  output logic [7:0] req_data,
  input  logic       rsp_valid,
  output logic       rsp_ready,
  input  logic [7:0] rsp_data,                     // checked by the bench
  output int unsigned got
);
  logic [7:0] n_q;
  assign req_valid = (32'(n_q) < N);
  assign req_data  = n_q;
  assign rsp_ready = HOLD_RSP ? !(req_valid && !req_ready) : 1'b1;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin n_q <= '0; got <= 0; end
    else begin
      if (req_valid && req_ready) n_q <= n_q + 8'd1;
      if (rsp_valid && rsp_ready) got <= got + 1;
    end
endmodule

module responder (
  input  logic       clk, rst_n,
  input  logic       req_valid,
  output logic       req_ready,
  input  logic [7:0] req_data,
  output logic       rsp_valid,
  input  logic       rsp_ready,
  output logic [7:0] rsp_data
);
  assign req_ready = !rsp_valid;                    // one response at a time
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin rsp_valid <= 1'b0; rsp_data <= '0; end
    else if (req_valid && req_ready) begin rsp_valid <= 1'b1; rsp_data <= req_data + 8'd100; end
    else if (rsp_ready) rsp_valid <= 1'b0;
endmodule

$ iverilog -g2012 -Pvr_tb.BAD=0 -o v0.vvp vr_tb.sv && vvp -n v0.vvp
PASS: 50 words on each stream and 50 responses in 100 cycles (BAD=0)
$ vvp -n v1.vvp
HANG at cycle 201
  pattern 1 stream   valid=0 ready=0  0 of 50 words
  pattern 2 stream   valid=0 ready=0  50 of 50 words
  pattern 3 request  valid=0 ready=1  50 of 50 requests
  pattern 3 response valid=0 ready=1  50 of 50 responses
FAIL cycle=201: no handshake on any channel for 100 cycles
...
$ vvp -n v2.vvp
HANG at cycle 201
  pattern 1 stream   valid=0 ready=0  50 of 50 words
  pattern 2 stream   valid=0 ready=0  0 of 50 words
...
$ vvp -n v3.vvp
HANG at cycle 152
  pattern 1 stream   valid=0 ready=0  50 of 50 words
  pattern 2 stream   valid=0 ready=0  50 of 50 words
  pattern 3 request  valid=1 ready=0  1 of 50 requests
  pattern 3 response valid=1 ready=0  0 of 50 responses
FAIL cycle=152: no handshake on any channel for 100 cycles
...

• Read the counts first. Lines with 50 of 50 are finished channels; their levels are just idle. In each run exactly one channel has stopped short.

• BAD=1: pattern 1 has moved 0 words, with both signals low. The source holds its data and waits to see ready; the sink waits to see valid. The hang starts at reset: not one word ever moved.

• BAD=2: pattern 2 looks identical in this simulator: 0 words, both low. The difference is not in the waveform; it is in the structure, below.

• BAD=3: one request went through, then both channels show valid=1 ready=0. The responder holds the first response and takes no new request; the requester has its second request up and refuses the response until that request is accepted. Each channel waits for the other. Note that the first transfer worked, which is why a single-transfer unit test passes.

The loop that lint sees

Pattern 2's valid = v_q && ready and the sink's ready = valid form a combinational loop. Simulation settled it at 0 and called it a hang; the tools that look at structure call it what it is. Verilator reports it while building the BAD=2 bench, and a Yosys structural check reports it on the pair alone:

$ verilator --binary --timing --assert -Wno-fatal -GBAD=2 --top-module vr_tb -Mdir vl2 vr_tb.sv
%Warning-UNOPTFLAT: vr_tb.sv:75:17: Signal unoptimizable: Circular combinational logic: 'vr_tb.v2'
...

1
2
3
4
5
6
7
8

// snippet: pair.sv is vr_src and vr_sink from above (vr_src's "assign valid" is line 9), then this wrapper
// pair: one source and one sink, for a structural check
module pair #(parameter bit COMB = 1) (input logic clk, rst_n, output logic [7:0] data, output logic fire);
  logic valid, ready;
  vr_src #(.COMB_RDY(COMB)) s (.clk, .rst_n, .valid, .ready, .data);
  vr_sink k (.valid, .ready);
  assign fire = valid && ready;
endmodule

$ yosys -q -p "read_verilog -sv pair.sv; hierarchy -top pair; proc; flatten; opt_clean; check -assert"
Warning: found logic loop in module pair:
    cell $flatten\s.$logic_and$pair.sv:9$13 ($logic_and) source: pair.sv:9.30-9.42
      B[0] --> Y[0]
    wire \ready source: pair.sv:24.16-24.21
ERROR: Found 1 problems in 'check -assert'.

Both name the loop through valid and ready. With COMB set to 0 the same check reports 0 problems. Pattern 1 has no loop, because the source's wait goes through a flop, so lint is silent on it: the same mistake, one flop apart, and only one of the two is visible to a tool.

What the evidence says

The levels at the hang separated pattern 3 from the other two at a glance: two channels both offering and both refused. Patterns 1 and 2 looked identical in simulation, and only a structural check split them. In all three the fix belongs to one side, and in patterns 1 and 3 it is not the side that looks guilty: the sinks and the responder followed the rules. Each failed on its first or second transfer, so these are not rare bugs; they were hidden by unit-test models that never waited.

Traps

Related topics

• Part 7 · Hangs and Deadlocks

• RTL: rules that keep flow control safe

• Protocol First Principles: the valid/ready rules

• Protocol First Principles: channel dependencies

• Protocol First Principles: the consumption assumption

• RTL interview: the valid/ready contract