The nightly regression has one new failure: SB_MISMATCH expected 52c7d685, got ad38297a. The designer's first reply is the one every DV engineer hears: "your model is wrong". Sometimes they are right. A scoreboard mismatch is an accusation with three possible defendants, and the person who can say quickly and with evidence which one is guilty is the person teams trust. How do you get from one wrong word to a bug report nobody can argue with?
If two clocks disagree, either one could be wrong, or you could be reading one of them at the wrong moment. A scoreboard mismatch is the same: the DUT may be wrong, the model (the predictor's idea of the specification) may be wrong, or the bench timing may be wrong, with a monitor sampling on the wrong edge or the comparison pairing the wrong two items. The playbook is to classify the corruption, shrink the failure until only the essential remains, and then let the smallest case tell you whose rule was broken.
Two habits come before everything else. Read only the first mismatch: later ones are often consequences (one lost word shifts every comparison after it). And never change the scoreboard to make a mismatch go away until you can say which defendant is guilty.
The course DUT has a planted bug behind a define. Built with EXTRA_ARGS="+define+DMA_BUG_WORD6", the smoke test fails with exactly one mismatch:
+UVM_TESTNAME=dma_smoke_test with EXTRA_ARGS=+define+DMA_BUG_WORD6
UVM_ERROR dma_tb.sv(396) @ 375000: uvm_test_top.env.sb [SB_MISMATCH] expected 52c7d685, got ad38297a
UVM_INFO dma_tb.sv(411) @ 655000: uvm_test_top.env.sb [SB] 7 words matched, 1 mismatched
UVM_ERROR : 1
[SB] 1
[SB_MISMATCH] 1XOR the two values: 52c7d685 ^ ad38297a = ffffffff. Every bit is flipped. That one number already rules out most causes. A stale or reordered word would give a random-looking difference; a single stuck bit would give one bit; a byte-lane problem would give one non-zero byte. A full inversion in a design that XORs data with a key points at the key being inverted, or the data, on exactly one word.
expected ^ actual | What it suggests | First place to look |
|---|---|---|
| Inversion of the whole word | A |
One bit set, the same bit every time | A stuck or flipped bit | That bit's logic or connection |
Non-zero in one byte only | Byte lane or strobe | Byte enables, lane steering, endianness |
Equal to the key | Key applied twice, or not at all | Key path; when the key was sampled |
Equal to the next word's expected ^ actual | Order: a word lost or duplicated | Monitor sampling, handshake, FIFO |
The bench scoreboard reports values but not positions. A scratch override adds both, plus the XOR and the key in force, without editing the bench: the scoreboard's write_* methods are virtual, so a factory type override can extend it.
// snippet: scratch scoreboard (dbg_tb.sv), installed with a type override of dma_scoreboard
class dma_scoreboard_idx extends dma_scoreboard;
`uvm_component_utils(dma_scoreboard_idx)
int unsigned n_out;
function new(string name, uvm_component parent);
super.new(name, parent);
endfunction
virtual function void write_out(axis_item t);
n_out++;
if (expected.size() != 0 && t.data !== expected[0])
`uvm_info("SB_IDX", $sformatf("output word %0d: expected %08h, got %08h, xor %08h, key %08h",
n_out, expected[0], t.data, expected[0] ^ t.data, key), UVM_LOW)
super.write_out(t);
endfunction
endclassscratch run: +UVM_TESTNAME=dbg_smoke_idx_test with EXTRA_ARGS=+define+DMA_BUG_WORD6
UVM_INFO dbg_tb.sv(102) @ 375000: uvm_test_top.env.sb [SB_IDX] output word 6: expected 52c7d685, got ad38297a, xor ffffffff, key a5a50f0f
UVM_ERROR dma_tb.sv(396) @ 375000: uvm_test_top.env.sb [SB_MISMATCH] expected 52c7d685, got ad38297a
UVM_INFO dma_tb.sv(411) @ 655000: uvm_test_top.env.sb [SB] 7 words matched, 1 mismatchedA minimal reproduction removes everything the bug does not need. The smoke test already uses one job, so the remaining knobs are the number of words and the key. A scratch test reads both from plusargs ($value$plusargs works on every build, unlike the UVM command-line processor here), sets KEY=0 so that output should equal input, and runs one job:
// snippet: scratch test (dbg_tb.sv); also overrides dma_scoreboard with dma_scoreboard_idx
task run_phase(uvm_phase phase);
int unsigned words = 6;
bit [31:0] key = 32'h0;
axis_pkt_seq pkt = axis_pkt_seq::type_id::create("pkt");
void'($value$plusargs("WORDS=%d", words));
void'($value$plusargs("KEY=%h", key));
phase.raise_objection(this);
reg_write(KEY, key);
reg_write(LEN, words);
reg_write(CTRL, 32'h3);
if (!pkt.randomize() with { n == words; }) `uvm_fatal("RAND", "axis_pkt_seq randomize failed")
pkt.start(env.in_agt.sqr);
wait (irq_vif.irq === 1'b1);
reg_write(STATUS, 32'h6);
#100ns;
phase.drop_objection(this);
endtaskscratch runs, EXTRA_ARGS=+define+DMA_BUG_WORD6, +UVM_TESTNAME=dbg_min_test
+WORDS=1
UVM_INFO dma_tb.sv(411) @ 345000: uvm_test_top.env.sb [SB] 1 words matched, 0 mismatched
TEST VERDICT: PASS (UVM_ERROR=0, UVM_FATAL=0)
+WORDS=5
UVM_INFO dma_tb.sv(411) @ 465000: uvm_test_top.env.sb [SB] 5 words matched, 0 mismatched
TEST VERDICT: PASS (UVM_ERROR=0, UVM_FATAL=0)
+WORDS=6
UVM_INFO dbg_tb.sv(102) @ 375000: uvm_test_top.env.sb [SB_IDX] output word 6: expected f762d98a, got 089d2675, xor ffffffff, key 00000000
UVM_ERROR dma_tb.sv(396) @ 375000: uvm_test_top.env.sb [SB_MISMATCH] expected f762d98a, got 089d2675
UVM_INFO dma_tb.sv(411) @ 505000: uvm_test_top.env.sb [SB] 5 words matched, 1 mismatched
TEST VERDICT: FAIL (UVM_ERROR=1, UVM_FATAL=0)
+WORDS=6 +KEY=12345678
UVM_INFO dbg_tb.sv(102) @ 375000: uvm_test_top.env.sb [SB_IDX] output word 6: expected e5568ff2, got 1aa9700d, xor ffffffff, key 12345678
TEST VERDICT: FAIL (UVM_ERROR=1, UVM_FATAL=0)The four runs say everything. One word passes, five words pass, six words fail, and the sixth word is inverted whatever the key and whatever the data. With KEY=0 the actual value is simply the inverted input. The bug depends on position only, so it lives in logic that counts words. The minimal reproduction is not one word, and that is itself a finding: the shortest failing case points at the counter. The design's data path, compiled with the define, confirms it:
// snippet: dma_dut.sv output data path with +define+DMA_BUG_WORD6
`ifdef DMA_BUG_WORD6
m_tdata <= s_tdata ^ (count == 16'd5 ? ~key : key); // planted bug for the mismatch playbook
`else
m_tdata <= s_tdata ^ key;
`endifThe planted bug was a DUT bug because the specification is clear (every word is XORed with KEY) and the DUT broke it. Two of the bench's war stories show the other defendants, and they are the ones that cost teams days.
The model was wrong (war story 9). The DMA ignores KEY writes while it is busy. The predictor and the scoreboard both take a new key at once. Nobody wrote KEY mid-job, so nobody noticed, until a test did and both checkers reported five mismatches. Two independent checkers agreeing did not make the DUT guilty: they shared the same wrong assumption. The deciding evidence was the specification plus a register-level fact the DUT could show directly (the key it actually used, read by backdoor), and the fix went into the model.
The bench timing was wrong (war stories 1 and 3). In one, two clocking blocks sampled on the same edge and a beat was lost: every comparison after it was off by one, which in the XOR table is the "neighbour's difference" row. In the other, an end-of-test check passed with a word missing, because the number of words owed was counted from what the monitor saw instead of from what the DUT was asked to do. In both, the DUT was innocent and the first mismatch was a symptom of the bench.
Defendant | Typical signature | Deciding evidence |
|---|---|---|
DUT | A clean pattern tied to a DUT condition (a count, a state, a register value) | The specification, and a minimal case that breaks it |
Model | Mismatches start right after a stimulus the model treats differently (a write while busy, a corner value) | The specification says the DUT is right; fix the predictor |
Timing | Every word after the first mismatch is also wrong, or a word goes missing | Waveforms at the monitor's sampling edge; counts of beats in and out |
A good report lets the designer reproduce the failure in one command and see the broken rule without reading the testbench. It states the smallest failing case and the largest passing one, the classified corruption, the scope ("independent of data and key" saves the designer an hour), the first bad time, and which checkers agree. This is the report for the planted bug:
Title: DMA output word 6 of every job is bit-inverted (data XOR ~KEY instead of KEY)
Found by: dma_smoke_test, dma_rand_test, dma_check_test (nightly 2026-10-16), all seeds
Reproduce: dbg_min_test +WORDS=6 +KEY=0 (fails) run.sh dbg_tb.sv ...
dbg_min_test +WORDS=5 (passes)
Expected: word 6 = input ^ KEY e.g. f762d98a with KEY=00000000
Actual: word 6 = input ^ ~KEY e.g. 089d2675 (expected ^ actual = ffffffff)
Scope: only the 6th output word of a job; independent of data, KEY and LEN >= 6
First bad: 375 ns, m_tdata on the beat where COUNT goes 5 -> 6
Suspect: dma_dut.sv output data path, COUNT == 5 term
Bench: commit <bench revision hash>, no bench change needed; checkers: dma_scoreboard and
uvm_in_order_comparator agree (13 matched, 1 mismatched each)0000a5a5, got 0000a5a4, and the next twenty words all match. What does the XOR say, and where do you look?0000a5a5 ^ 0000a5a4 = 00000001: a single bit, bit 0, on one word. It is not an ordering problem (the following words match) and not a lane problem. Look for a condition that flips or forces bit 0 on that one transfer: a parity or flag bit sharing the data bus, a reset or enable edge on that cycle, or a rounding rule the model does not implement.
Probably not at first: the output is the expected sequence delayed by one, so a word was duplicated, or the monitor sampled one beat twice or one beat late. Check the input and output beat counts and the monitor's sampling edge in the waves before blaming the DUT. If the DUT really repeated a word, the waves at its port will show it.
It isolates the trigger: the sixth word, independent of everything random. The designer can go straight to the logic that depends on the word count, and can verify a fix with one short directed run. A random seed proves the bug exists but leaves the designer to rediscover the trigger.
Next: a mismatch at least proves the scoreboard is awake. False-Pass Playbook deals with the green runs that never compared anything.