The DMA is going into a car. The safety manager sends one question: if a flip-flop inside this block breaks in the field, will anything notice? Your regression is green, coverage is close to closed, and you realise you have never measured that. Every test you wrote assumes the hardware works. So you break it on purpose, one fault at a time, and count how often the bench complains. On the course bench the answer is 5 out of 8. What do the three misses mean, and what does a safety standard want from you?

The idea: assume the silicon breaks, then prove it is caught

Functional verification asks does the design do what the spec says? Functional safety asks a different question: when the hardware fails at random in the field, does the system stay safe? Random hardware failures are real: a transistor wears out and a node sticks at 0 or 1 (a permanent fault), or a particle strike flips one flip-flop or memory bit for a moment (a transient fault, often called a single-event upset, SEU). No amount of functional testing removes them, because the RTL was never wrong.

Think of a smoke detector. You do not test it by checking that the kitchen is not on fire; you test it by holding smoke under it. A fault campaign does the same for a chip: inject a fault, run a workload, and record whether something noticed. The "something" is either the testbench (when you want to measure how strong the bench is) or an on-chip safety mechanism (when you want to prove the chip protects itself).

ISO 26262 in a DV engineer's terms

ISO 26262 is the automotive functional safety standard; part 5 covers hardware. You do not need to recite it, but interviewers for automotive work expect the vocabulary:

Two hardware architectural metrics turn this into numbers. The single-point fault metric (SPFM) is one minus the failure rate of single-point and residual faults, divided by the total failure rate of the safety-related hardware: how well the design is covered against faults that hurt at once. The latent fault metric (LFM) is one minus the latent-fault rate divided by the failure rate that remains after single-point and residual faults are removed: how well the safety mechanisms themselves are checked. ISO 26262-5 gives target values per ASIL:

The failure rates come from the safety analysis (an FMEDA: failure modes, effects and diagnostic analysis), usually done by a safety engineer. The DV engineer's job is the evidence: the fault campaign that shows a claimed DC is real, and the functional tests that show each safety mechanism works (ECC corrects a single-bit error and flags a double, the lockstep comparator fires on a mismatch, the watchdog bites).

The picture: one fault, one run, one verdict

A fault only counts if the workload activates it (the stuck node would otherwise have held the other value) and propagates it to a point someone checks. Those are the same two words you know from manufacturing test, and they explain most misses below.

Build it: a real campaign on the DMA

The DMA has no safety mechanism, so on this bench every detection comes from the testbench itself. That makes the campaign a measure of bench strength: a bench that cannot see a stuck output bit cannot help a safety case either. dma_fault_tb.sv includes the normal bench and replaces its top with one that forces a single stuck-at fault inside dma_dut after reset, chosen by +FAULT=<n>:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

// snippet: the fault list in dma_fault_tb.sv (its top module)
int fault_id;
initial begin
  if ($value$plusargs("FAULT=%d", fault_id) && fault_id != 0) begin
    @(posedge rst_n);
    case (fault_id)
      1: force dut.key[0]      = 1'b0;   // KEY bit 0 stuck at 0
      2: force dut.m_tdata[31] = 1'b1;   // output data bit 31 stuck at 1
      3: force dut.count[3]    = 1'b0;   // COUNT bit 3 stuck at 0
      4: force dut.irq_en      = 1'b0;   // interrupt enable stuck at 0
      5: force dut.err         = 1'b0;   // ERR flag stuck at 0
      6: force dut.done        = 1'b1;   // DONE flag stuck at 1
      7: force dut.m_tlast     = 1'b0;   // output TLAST stuck at 0
      8: force dut.len[4]      = 1'b0;   // LEN bit 4 stuck at 0
      default: $fatal(1, "unknown fault %0d", fault_id);
    endcase
    $display("FAULT %0d injected", fault_id);
  end
end

The top also sets a 50 us UVM timeout, because a fault that hangs the DMA must still end the run. The script builds once, then runs every fault against each test and calls a fault detected if the run ends with any UVM_ERROR or UVM_FATAL:

1
2
3
4
5
6
7
8

# excerpt: scripts/tools/uvm-bench/fault-campaign.sh
for f in $faults; do
  for t in "${tests[@]}"; do
    log=$("$out/obj/Vtop" +UVM_TESTNAME="$t" +FAULT="$f" 2>&1 || true)
    e=$(echo "$log" | sed -n 's/^UVM_ERROR : *//p'); x=$(echo "$log" | sed -n 's/^UVM_FATAL : *//p')
    if [ "${e:-0}" -gt 0 ] || [ "${x:-0}" -gt 0 ]; then r="detected(E${e:-0},F${x:-0})"; else r="MISSED"; fi
  done
done

One build, many runs: the same pattern as a regression. A commercial fault simulator does this far more efficiently (many faults per simulation, on a gate-level netlist), but the verdict logic is the same.

Watch it run: 5 of 8

$ UVM_HOME=... scripts/tools/uvm-bench/fault-campaign.sh
fault  dma_rand_test        dma_truncate_test
1      detected(E11,F0)     MISSED
2      detected(E8,F0)      detected(E2,F0)
3      detected(E1,F0)      MISSED
4      detected(E0,F1)      detected(E0,F1)
5      MISSED               detected(E0,F1)
6      MISSED               MISSED
7      MISSED               MISSED
8      MISSED               MISSED
faults detected by at least one test: 5 of 8

Read the detected rows first, because how a fault was caught matters as much as whether:

• Fault 2 (output bit 31 stuck at 1) is caught by the scoreboard: every word whose expected bit 31 is 0 mismatches.

• Fault 3 (COUNT bit 3 stuck at 0) is caught indirectly. On the 10-word job the counter wraps from 7 back to 0, so the DMA never sees its last word arrive; the input packet's TLAST then looks like an early end, the DMA sets ERR, and the test's STATUS check fails: [STATUS] job 0: STATUS=00000004, expected DONE only. Nobody checks COUNT itself.

• Faults 4 and 5 are caught only as hangs (F1 is the PH_TIMEOUT fatal): no interrupt, so wait (irq) never returns. A hang is a detection, but a slow and uninformative one.

The per-fault logs of the misses end with a clean scoreboard:

--- FAULT 6 rand
UVM_INFO dma_tb.sv(731) @ 0: uvm_test_top [JOB] key=90fdd8fd len=10 irq_en=1
FAULT 6 injected
UVM_INFO dma_tb.sv(731) @ 645000: uvm_test_top [JOB] key=0e564b57 len=1 irq_en=1
UVM_INFO dma_tb.sv(731) @ 885000: uvm_test_top [JOB] key=5a1793f4 len=3 irq_en=1
UVM_INFO dma_tb.sv(411) @ 1305000: uvm_test_top.env.sb [SB] 14 words matched, 0 mismatched
UVM_ERROR :    0
--- FAULT 7 rand        (same jobs, same result)
UVM_INFO dma_tb.sv(411) @ 1305000: uvm_test_top.env.sb [SB] 14 words matched, 0 mismatched
UVM_ERROR :    0
--- FAULT 1 truncate
FAULT 1 injected
UVM_INFO dma_tb.sv(520) @ 275000: uvm_test_top.env.in_agt.drv [TRUNC] TLAST forced on beat 3
UVM_INFO dma_tb.sv(761) @ 385000: uvm_test_top [TRUNC] DMA reported ERR for the short packet, as expected
UVM_INFO dma_tb.sv(411) @ 485000: uvm_test_top.env.sb [SB] 4 words matched, 0 mismatched
UVM_ERROR :    0
UVM_FATAL :    0

What the misses mean, one by one

Fault 8 should feel familiar: dma_check_test's coverage report already lists len=16 as a plan hole (see Closing Coverage: The Holes Random Never Hits). A fault campaign and a coverage model looking at the same bench found the same gap from two directions. That agreement is worth saying in an interview: coverage tells you what you did not stimulate; fault injection tells you what you did not check, which coverage cannot.

Fixing the bench (scratch run)

A scratch copy of the fault bench adds a job checker and a directed test, and leaves the bench files unchanged. The checker follows the APB writes to learn LEN at each GO, expects TLAST on the job's last word (or on an input word that carried an early TLAST), and two clocks after each interrupt rises checks that the job's last word has already left. The env picks it up through a factory type override of dma_env, enabled by +JOB_CHECK:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

// snippet: scratch dma_fault_fix_tb.sv, the checks added to the bench
virtual function void write_in(axis_item t);
  n_in++;
  exp_last.push_back(t.last || n_in == job_len);    // TLAST expected here?
endfunction

virtual function void write_out(axis_item t);
  bit exp;
  n_out++;
  if (exp_last.size() == 0) return;                 // the scoreboard reports unexpected words
  exp = exp_last.pop_front();
  if (t.last !== exp)
    `uvm_error("TLAST", $sformatf("output word %0d of %0d: TLAST=%0b, expected %0b", n_out, job_len, t.last, exp))
  if (exp) job_ended = 1;
endfunction

task run_phase(uvm_phase phase);
  forever begin
    @(posedge irq_vif.irq);
    repeat (2) @(posedge irq_vif.clk);               // the last word leaves one clock after DONE
    if (!job_ended)
      `uvm_error("DONE_EARLY", $sformatf("interrupt raised with %0d of %0d words out", n_out, job_len))
  end
endtask

// and a directed test: dma_rand_test with one job of LEN=16 passed through config_db
class dma_len16_test extends dma_rand_test;
  `uvm_component_utils(dma_len16_test)
  function new(string name, uvm_component parent);
    super.new(name, parent);
  endfunction
  function void build_phase(uvm_phase phase);
    dma_job_cfg job = dma_job_cfg::type_id::create("job");
    if (!job.randomize() with { len == 16; }) `uvm_fatal("RAND", "dma_job_cfg randomize failed")
    uvm_config_db#(dma_job_cfg)::set(this, "", "job", job);
    n_jobs = 1;
    super.build_phase(phase);
  endfunction
endclass

Without faults, all three tests still pass with the checker on (14, 4 and 16 words matched, 0 errors). The campaign, rerun with the checker on and the new test added:

scratch run: dma_fault_fix_tb.sv with +JOB_CHECK
fault  dma_rand_test        dma_truncate_test    dma_len16_test
1      detected(E11,F0)     MISSED               detected(E17,F0)
2      detected(E8,F0)      detected(E2,F0)      detected(E10,F0)
3      detected(E1,F0)      MISSED               detected(E1,F0)
4      detected(E0,F1)      detected(E0,F1)      detected(E0,F1)
5      MISSED               detected(E0,F1)      MISSED
6      detected(E1,F0)      detected(E1,F0)      detected(E1,F0)
7      detected(E3,F0)      detected(E1,F0)      detected(E1,F0)
8      MISSED               MISSED               detected(E1,F1)
faults detected by at least one test: 8 of 8

UVM_ERROR dma_fault_fix_tb.sv(72) @ 175000: uvm_test_top.env.jc [DONE_EARLY] interrupt raised with 0 of 10 words out
UVM_ERROR dma_fault_fix_tb.sv(63) @ 575000: uvm_test_top.env.jc [TLAST] output word 10 of 10: TLAST=0, expected 1
UVM_ERROR dma_fault_fix_tb.sv(72) @ 175000: uvm_test_top.env.jc [DONE_EARLY] interrupt raised with 0 of 16 words out

• Fault 6 now fails at 175 ns, the moment GO is written: the first clear message instead of silence.

• Fault 8 is caught twice in dma_len16_test: LEN=16 becomes 0, the DMA treats it as a zero-length job and finishes at once (DONE_EARLY), then the packet can never be accepted and the run ends in PH_TIMEOUT. The checker turns a 50 us hang into a 175 ns error with a cause in it.

• Faults 1 and 5 are still missed by some tests. Detection is per test, so a safety regression keeps the tests that together catch everything, and the campaign tells you which ones those are.

How a project does this for real

A production campaign differs from the course script in scale and in what counts as detection. The fault list comes from the gate-level netlist (stuck-at 0 and 1 on every node, plus transient flips on every flip-flop and memory), which runs to millions of faults. Tools collapse equivalent faults, use formal analysis to prove some safe or untestable, and sample the rest. Sampling is statistics, not a shortcut: the sample size follows from the margin and confidence you will claim. A common reference (Leveugle et al., 2009) gives n = N / (1 + e²(N−1) / (t² p(1−p))); with the worst case p = 0.5, a 1% margin (e = 0.01) and 99% confidence (t ≈ 2.58), a large fault list needs about 16,600 injected faults, whatever N is. The safety case states the margin and confidence, and the diagnostic coverage is reported with that interval, not as a bare percentage. A commercial fault simulator runs many faults per simulation against a golden run and classifies each at the declared observation and detection points; the results feed the FMEDA. Those tools cannot be run here, but every major vendor's flow follows these steps.

A transient fault is modelled differently from a stuck-at. For a flip-flop, force then release one clock later: after a release, a variable keeps the forced value until the design next writes it, which is exactly a bit flip that lasts until the register is reloaded.

1
2
3
4
5

// snippet: a single-event upset on a register bit, held until the design next writes it
@(posedge clk);
force dut.key[5] = ~dut.key[5];
@(posedge clk);
release dut.key[5];

What breaks: a campaign that measured the wrong thing

A team reports 99% detection on a block with ECC. The safety assessor asks how detection was decided: "any UVM_ERROR". The scoreboard caught most faults by comparing data, so the number measures the testbench, not the ECC, and in the field there is no scoreboard. Rerun with detection defined as "the ECC alarm fired before the corrupted data left the block", the number drops sharply: faults in the encoder's output path corrupted data with no alarm. The fix was a design change. Decide what counts as detection before you run, and keep the two uses of fault injection, bench strength and diagnostic coverage, apart in every report.

Next: a real campaign runs millions of faults; a full-chip test can take weeks in simulation even without faults. Emulation and FPGA Prototyping is what teams do when the simulator is too slow.

Related topics

• Up: Safety, Acceleration and Models

• False-Pass Playbook

• Closing Coverage: The Holes Random Never Hits