A small interconnect lets two masters read from one memory slave. Each request gets a 2-bit tag, and the slave sends the tag back with the data, so the interconnect knows whose response it is. Until last week the slave answered in order. The new slave has a slow region (addresses with bits 6 to 3 all ones) that answers after 12 cycles while everything else takes 2, so its answers can now overtake each other. The block regression has a scoreboard per master. Master 0 reads addresses 00 to 7f, master 1 reads 80 to ff, and the slave's data carries the address it came from. This is the new failure:
$ iverilog -g2012 -o t.vvp tag_tb.sv && vvp -n t.vvp +SHOW=0
FAIL cycle=20: M0 got data a5ff (address ff), which it never asked for; it waits for nothing
FATAL: tag_tb.sv:153: a response went to the wrong master
...Address ff belongs to master 1. Master 0 got it, and at that moment master 0 was not waiting for anything at all. Every seed from 1 to 50 fails somewhere between cycle 14 and cycle 375.
Stop here if you want to try it yourself. The design and bench are in the hunt below. Where would you start: the arbiter, the slave, or somewhere else?
The interconnect does not decide where a response goes when the response arrives; it looks up who owns the response's tag. So the question is not "why did the response go to M0?" but "when did M0 become the owner of that tag, and was the tag free then?"
The failure repeats at cycle 20 for seed 1 in Icarus Verilog and in Verilator. That is the easy kind: it is not a race, it is a sequence of events. Here is the whole design and bench. tag_xbar has the arbiter and the tag table, slave answers out of order, and sys adds two random masters, their scoreboards and a trace.
`timescale 1ns/1ps
// tag_xbar: two masters share one slave that may answer out of order. Each request gets a 2-bit
// tag; the xbar remembers which master owns each tag and sends the response with that tag back to it.
// FIX = 0: the tag is a rolling counter, and up to four requests may be outstanding.
// FIX = 1: the tag is the lowest tag not in use, and a request waits when all four are in use.
module tag_xbar #(parameter bit FIX = 0) (
input logic clk,
input logic [1:0] m_valid,
input logic [7:0] m_addr [2],
output logic [1:0] m_ready,
output logic [1:0] m_rvalid,
output logic [15:0] m_rdata,
output logic s_valid,
output logic [7:0] s_addr,
output logic [1:0] s_tag,
input logic s_rvalid,
input logic [1:0] s_rtag,
input logic [15:0] s_rdata
);
logic owner [4]; // which master owns each tag
logic [3:0] busy = '0; // tags with a response still to come
logic [1:0] next_tag = '0, tag;
logic [2:0] outstanding = '0;
logic last = 1'b1, pick, can_issue, issue;
initial for (int t = 0; t < 4; t++) owner[t] = 1'b0;
always_comb begin
if (FIX) begin
tag = 2'd0;
for (int t = 3; t >= 0; t--) if (!busy[t]) tag = 2'(t);
can_issue = (busy != 4'hf);
end else begin
tag = next_tag; // assumes tags come back in the order they left
can_issue = (outstanding < 3'd4);
end
end
assign pick = (m_valid == 2'b11) ? !last : m_valid[1]; // round robin
assign issue = (m_valid != 2'b00) && can_issue;
assign m_ready = issue ? (pick ? 2'b10 : 2'b01) : 2'b00;
assign s_valid = issue;
assign s_addr = m_addr[pick];
assign s_tag = tag;
assign m_rvalid = s_rvalid ? (owner[s_rtag] ? 2'b10 : 2'b01) : 2'b00;
assign m_rdata = s_rdata;
always @(posedge clk) begin
if (s_rvalid) busy[s_rtag] <= 1'b0;
if (issue) begin
owner[tag] <= pick;
busy[tag] <= 1'b1;
next_tag <= next_tag + 2'd1;
last <= pick;
end
outstanding <= outstanding + 3'(issue) - 3'(s_rvalid);
end
endmodule
// slave: accepts a read every cycle and answers after FAST cycles, or slow_lat cycles for an
// address in the slow region (addr[6:3] == f). One response per cycle, so answers can come back
// out of order. The data names the address it came from: {addr ^ 5a, addr}.
module slave #(parameter int FAST = 2) (
input logic clk,
input int slow_lat,
input logic s_valid,
input logic [7:0] s_addr,
input logic [1:0] s_tag,
output logic s_rvalid,
output logic [1:0] s_rtag,
output logic [15:0] s_rdata
);
logic pv [8];
logic [1:0] ptag [8];
logic [7:0] paddr [8];
int pdue [8];
int sel;
initial for (int i = 0; i < 8; i++) pv[i] = 1'b0;
always_comb begin
sel = -1;
for (int i = 7; i >= 0; i--) if (pv[i] && pdue[i] <= 0) sel = i;
end
assign s_rvalid = (sel >= 0);
assign s_rtag = (sel >= 0) ? ptag[sel] : 2'd0;
assign s_rdata = (sel >= 0) ? {paddr[sel] ^ 8'h5a, paddr[sel]} : 16'h0;
always @(posedge clk) begin
int fr;
fr = -1;
for (int i = 7; i >= 0; i--) if (!pv[i] || i == sel) fr = i;
for (int i = 0; i < 8; i++) if (pv[i] && pdue[i] > 0) pdue[i] <= pdue[i] - 1;
if (sel >= 0) pv[sel] <= 1'b0;
if (s_valid) begin
pv[fr] <= 1'b1; ptag[fr] <= s_tag; paddr[fr] <= s_addr;
pdue[fr] <= (s_addr[6:3] == 4'hf) ? slow_lat - 1 : FAST - 1;
end
end
endmodule
// sys: two random masters, the xbar and the slave, a scoreboard per master and a tag check.
// Master 0 reads addresses 00-7f, master 1 reads 80-ff, so a response says whose it is.
module sys #(parameter bit FIX = 0, parameter int ID = 0) (
input logic clk, input logic [31:0] seed, input int slow_lat, show, trace_from, trace_to, input bit stop
);
logic [1:0] m_valid = '0, m_ready, m_rvalid;
logic [7:0] m_addr [2];
logic [15:0] m_rdata;
logic s_valid, s_rvalid;
logic [7:0] s_addr;
logic [1:0] s_tag, s_rtag;
logic [15:0] m_rdata_s;
tag_xbar #(.FIX(FIX)) u_x (.clk, .m_valid, .m_addr, .m_ready, .m_rvalid, .m_rdata,
.s_valid, .s_addr, .s_tag, .s_rvalid, .s_rtag, .s_rdata(m_rdata_s));
slave u_s (.clk, .slow_lat, .s_valid, .s_addr, .s_tag, .s_rvalid, .s_rtag, .s_rdata(m_rdata_s));
logic [7:0] wq [2][16]; // each master's reads still to be answered
int wn [2];
int unsigned cyc = 0, errors = 0, reads = 0, rng = 0;
string first = "", reuse = "";
logic [3:0] in_use = '0; // the bench's own record of tags in flight
initial begin m_addr[0] = 8'h00; m_addr[1] = 8'h80; wn[0] = 0; wn[1] = 0; end
function automatic string list(input int m);
string s;
s = "";
for (int i = 0; i < wn[m]; i++) s = {s, $sformatf(" %h", wq[m][i])};
if (s == "") s = " nothing";
return s;
endfunction
always @(posedge clk) begin // values just before the edge
cyc++;
if (trace_to > 0 && show == ID && cyc >= trace_from && cyc <= trace_to) begin
if (s_valid && in_use[s_tag])
$display("%4d issue M%0d addr %h tag %0d <- tag still in flight", cyc, u_x.pick, s_addr, s_tag);
else if (s_valid)
$display("%4d issue M%0d addr %h tag %0d", cyc, u_x.pick, s_addr, s_tag);
if (s_rvalid)
$display("%4d response tag %0d data %h -> M%0d", cyc, s_rtag, m_rdata, m_rvalid[1]);
end
// the tag check: a tag must not be handed out again before its response is back
if (s_valid && in_use[s_tag] && reuse == "")
reuse = $sformatf("cycle %0d: tag %0d issued to M%0d while still in flight", cyc, s_tag, u_x.pick);
// scoreboards
for (int m = 0; m < 2; m++) if (m_rvalid[m]) begin
int k;
k = -1;
for (int i = 0; i < wn[m]; i++) if (wq[m][i] == m_rdata[7:0]) k = i;
reads++;
if (k < 0 || m_rdata[15:8] != (m_rdata[7:0] ^ 8'h5a)) begin
errors++;
if (first == "")
first = $sformatf("cycle %0d: M%0d got data %h (address %h), which it never asked for; it waits for%s",
cyc, m, m_rdata, m_rdata[7:0], list(m));
if (show == ID) begin
$display("FAIL cycle=%0d: M%0d got data %h (address %h), which it never asked for; it waits for%s",
cyc, m, m_rdata, m_rdata[7:0], list(m));
$fatal(1, "a response went to the wrong master");
end
end else begin // answered: remove it from the list
for (int i = k; i < wn[m] - 1; i++) wq[m][i] = wq[m][i + 1];
wn[m]--;
end
end
if (s_rvalid) in_use[s_rtag] <= 1'b0;
if (s_valid) in_use[s_tag] <= 1'b1;
// masters: hold a request until it is taken, then maybe start another
if (rng == 0) rng = seed * 32'h9E37_79B9 + 32'(ID) * 32'h85EB_CA6B + 32'h7F4A_7C16;
for (int m = 0; m < 2; m++) begin
if (m_valid[m] && m_ready[m]) begin wq[m][wn[m]] = m_addr[m]; wn[m]++; end
if (!m_valid[m] || m_ready[m]) begin
rng ^= rng << 13; rng ^= rng >> 17; rng ^= rng << 5;
m_valid[m] <= !stop && wn[m] < 8 && (rng % 4 == 0); // at most 8 reads waiting
m_addr[m] <= {1'(m), 7'((rng >> 8) % 128)};
end
end
end
endmodule
The scoreboard is right that M0 received a response it never asked for. It cannot say when the mistake was made. The response arrived at cycle 20, but the decision to send it to M0 was taken earlier, when someone wrote M0 into the tag table. Scoreboards report late: the cycle of the message is where to start walking back, not where the bug is.
The response is routed by owner[s_rtag], so the next hop on the walk back is the tag table. Print every request and every response, with tags, from the start of the run:
$ vvp -n t.vvp +SHOW=0 +FROM=1 +TO=30
2 issue M1 addr dd tag 0
4 issue M1 addr c4 tag 1
4 response tag 0 data 87dd -> M1
...
8 issue M1 addr ff tag 1
8 response tag 3 data c993 -> M1
9 response tag 0 data 8cd6 -> M1
10 issue M1 addr fc tag 2
13 issue M0 addr 6a tag 3
15 response tag 3 data 306a -> M0
16 issue M1 addr ff tag 0
17 issue M0 addr 54 tag 1 <- tag still in flight
19 response tag 1 data 0e54 -> M0
20 issue M1 addr dd tag 2 <- tag still in flight
20 response tag 1 data a5ff -> M0
FAIL cycle=20: M0 got data a5ff (address ff), which it never asked for; it waits for nothing
...Read it backwards from the failure, one hop at a time, as in the first wrong cycle:
Cycle 20: the response carries tag 1 and the data of address ff, so it is the answer to M1's slow read of ff. It went to M0 because the table says tag 1 belongs to M0.
Cycle 19: another response with tag 1, M0's read of 54, correctly to M0. Two responses with the same tag, one cycle apart: two requests were in flight with tag 1 at once.
Cycle 17, the first wrong cycle: tag 1 is given to M0's read of 54, and the trace marks it: tag still in flight. M1's read of ff has had tag 1 since cycle 8 and is still waiting for its slow answer. Writing M0 into owner[1] here wiped out the only record that the next tag-1 response belongs to M1.
Cycle 8: M1 reads ff, a slow address, with tag 1. Nothing wrong yet.
The tag table at the cycles that matter:
After cycle | Event | owner[1] | Tags in flight | Counter says |
|---|---|---|---|---|
8 | M1 reads ff (slow) with tag 1 | M1 | 1 | next tag 2 |
16 | M1 reads ff (slow) with tag 0; 2 is still out (fc, slow) | M1 | 0, 1, 2 | 3 outstanding, next tag 1 |
17 | M0 reads 54 with tag 1 | M0 | 0, 1, 1, 2 | 4 outstanding: allowed |
19 | the fast tag-1 response (54) comes back first | M0 | 0, 1, 2 | |
20 | the slow tag-1 response (ff) comes back; dd takes tag 2, also still in flight | M0 | 0, 2, 2 | sent to M0: wrong |
Why was tag 1 handed out while still in flight? In tag_xbar with FIX = 0, the tag is next_tag, a counter that goes 0, 1, 2, 3, 0, ..., and the only limit is outstanding < 4. At cycle 17 only three requests were outstanding, so the counter's turn came round to tag 1. The counter assumes that when it gets back to a tag, that tag's response has returned, because the oldest request finishes first. That was true of the old slave. The new one answers fast reads first, so tags 3 and 0 came back while tag 1 was still waiting.
Hypothesis: the rolling tag counter reuses a tag whose response is still coming, which can only happen when responses return out of order. It misroutes a response only when the two requests that share the tag come from different masters; with the same master, the response still reaches the right place. That explains why it needs a particular interleaving: a slow read from one master, three more requests that finish first, then the other master's request landing on the reused tag.
A hypothesis has to predict something you have not seen. This one predicts that if the slow region is made as fast as the rest, so the slave answers in order again, the same design and the same stimulus will never misroute. +SLOW=2 does that:
$ vvp -n t.vvp +SLOW=2
FIX=0: 1954 responses, 0 to the wrong master, 0+0 never answered; first: none
FIX=1: 1928 responses, 0 to the wrong master, 0+0 never answered
tag check, FIX=0: none
tag check, FIX=1: none
END: experiment finished (seed 1, slow latency 2)
...No misrouted response and no tag reused, on seed 1 and on all 50 seeds tried. The prediction could have failed, for example if the arbiter's choice of pick were wrong; it did not. The bug lives exactly where in-order return was assumed. See a hypothesis must predict.
Keep a busy bit per tag, set it when the tag is handed out and clear it when its response comes back. Hand out the lowest tag whose busy bit is clear, and stall the requester when all four are busy. That is FIX = 1 in the bench:
// snippet: tag_xbar with FIX = 1 (busy[] is set at issue and cleared at the response)
tag = 2'd0;
for (int t = 3; t >= 0; t--) if (!busy[t]) tag = 2'(t); // the lowest free tag
can_issue = (busy != 4'hf); // wait when all four are in flight$ vvp -n t.vvp
FIX=0: 1199 responses, 12 to the wrong master, 4+8 never answered; first: cycle 20: M0 got data a5ff (address ff), which it never asked for; it waits for nothing
FIX=1: 1924 responses, 0 to the wrong master, 0+0 never answered
tag check, FIX=0: cycle 17: tag 1 issued to M0 while still in flight
tag check, FIX=1: none
PASS: the rolling tag sends responses to the wrong master; the free-tag version does not
...
$ for s in $(seq 1 50); do vvp -n t.vvp +SEED=$s > o.txt; grep -q "^PASS" o.txt || echo "seed $s FAIL"; done
$Read the first two lines together. The buggy version sent 12 responses to the wrong master and left 12 reads unanswered (4 of M0's, 8 of M1's); each master stops issuing at 8 waiting reads, so without a scoreboard this test would have ended as a hang, a lost token in the sense of credit and token loss. The fixed version answered every read, to the right master. The loop prints nothing: all 50 seeds pass their checks, the bug failing and the fix clean in each. Verilator 5.048 prints the same default lines.
The third line is the check that should have been there from the start. The bench keeps its own record of tags in flight and fails when a tag is issued while its bit is set. It fired at cycle 17, the first wrong cycle, three cycles before the scoreboard, and its message names the tag and the master. In a real bench it is a one-line assertion on the request channel. It does not depend on the data, on which master is involved, or on the response ever coming back, so it also catches the same-master reuse that the scoreboard never sees.
Which step won this hunt? Localise. The data told you what went wrong; only following the tag told you where: to the cycle the table was overwritten, and the line that allowed it.
No, it only makes the failure rarer. With three allowed, a slow request on tag 1 can be followed by requests on tags 2 and 3 that finish, then tag 0 that finishes, and the counter comes back to tag 1 while it is still in flight, with only one or two requests outstanding. Any limit below the number of tags still lets the counter lap a slow request. The tag check would still fire; the fix must free tags by response, not by count.
Yes. A tag was handed out while its response was still coming. If both requests came from the same master, both responses reach that master and this scoreboard, which matches by address, is happy. But the two responses share a tag and may come back in either order, so a master that matches responses by tag would pair them wrongly, and the next time the two requests come from different masters it is this case study's failure. The check fires on the cause; the scoreboard only on some of its effects.
tag_tb.sv as shown, 2026-10-18: iverilog -g2012 -o t.vvp tag_tb.sv && vvp -n t.vvp (Icarus Verilog 13.0) prints the five lines shown and PASS; vvp -n t.vvp +SHOW=0 and +SHOW=0 +FROM=1 +TO=30 give the failure and trace shown; +SLOW=2 gives the in-order run. verilator --binary --timing --assert -Wno-fatal --top-module tag_tb -Mdir vl tag_tb.sv (Verilator 5.048) prints the same result, FAIL and trace lines; only its FATAL line has its own format.
Seeds 1 to 50 (+SEED=<n>): the default run passes its checks for every seed, FIX=0 sending 9 to 15 responses to the wrong master (first at cycles 14 to 375) and FIX=1 none. With +SLOW=2 FIX=0 sends no response to the wrong master and leaves none unanswered on all 50 seeds.
Limits: one small xbar and slave model, simulated only; no real bus protocol, no formal proof. dbglab's fabric allows one request in flight and has no tags, so this bug cannot occur in it.