Two months before tape-out

The design review for the DMA subsystem is going well until the bring-up lead asks one question: "When this hangs on the bench, what can I read?" The block owner answers honestly: the CSRs, if the bus works, and scan. Area is tight and the schedule tighter.

In simulation the deadlock case study took one probe and a wait-for table. On silicon there is no probe. What should this block carry into silicon, and how do you defend its cost?

What people usually do, and why it fails

The usual plan is "we will catch everything in simulation", plus scan and a JTAG port for the lab. It fails in three ways. Bugs that escape simulation are the ones it did not provoke: rare orderings, long runs, real software. Scan dump is slow and destructive: it stops the clock and gives you one snapshot of every flop, with no history, after the interesting moment has passed. And the CSRs that would explain a hang are often behind the very bus that is hung.

dbglab has that last flaw. Its CSR block is a slave on the shared bus, so in the back-pressure deadlock the CPU could not have read a single status register: its request was the one stuck waiting for the bus. Observability that depends on the thing you are debugging is not observability.

The method: design the questions, then the registers

Start from the failures, not the features. For each way the block can fail, write the question you will ask on the bench and the register that answers it:

• Keep the first failure. A sticky bit set by the tenth error tells you nothing the first did not. Record the first error's source and time, and freeze the trace around it.

• Read without disturbing. A debug read must not pop a FIFO, clear a flag or change arbitration. Clearing is a separate, deliberate write.

• Give it a path that does not need the block. The debug registers sit on a port that still works when the functional bus, clock gating or the block's own FSM is stuck.

• Budget it and test it. Every feature has an area cost and a verification cost. Untested debug logic fails exactly when you need it.

On the debug loop this is Observe, moved to the start of the project: you are deciding now what the chip will be able to show the Observe step later. It also feeds Prevent: each escape analysis should ask which register would have shown the bug sooner.

The picture: debug features around a block

Read it right to left, as the lab will: through the side-band port to the registers, each answering one question about the block.

Do it: a sticky-error register and a freezing trace ring

The block below holds two of the features. Four error sources set sticky bits; the first error after a clear also records which source and the timestamp. Every event goes into an 8-entry ring with its timestamp. The first error arms a freeze: two more events are stored, and then the ring stops, so it holds the history around the first failure. A later error neither overwrites the record nor restarts the trace. Clearing all sticky bits re-arms it. The bench checks each of those promises.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

// dbg_monitor: two debug features a block can carry into silicon.
//  - Sticky errors: err_i pulses set bits in sticky_o that stay set until software writes 1 to
//    clear them (W1C). The first error after a clear also records its code and timestamp.
//  - Trace ring: every event (ev_valid_i, ev_code_i) is stored with a timestamp in a DEPTH-entry
//    ring. The first error arms a freeze: POST more events are stored, then the ring stops, so it
//    holds the history around the first failure. Clearing the sticky bits re-arms it.
// Read side: tr_idx_i = 0 is the oldest entry, DEPTH-1 the newest.
`timescale 1ns/1ps
module dbg_monitor #(parameter int DEPTH = 8, parameter int POST = 2, parameter int CW = 4) (
  input  logic          clk, rst_n,
  input  logic          ev_valid_i,
  input  logic [CW-1:0] ev_code_i,
  input  logic [3:0]    err_i,                       // one bit per error source
  input  logic [3:0]    w1c_i,                       // software clear, one pulse
  output logic [3:0]    sticky_o,
  output logic [3:0]    first_err_o,                 // which source failed first
  output logic [15:0]   first_ts_o,                  // and when
  output logic          frozen_o,
  input  logic [$clog2(DEPTH)-1:0] tr_idx_i,
  output logic [15:0]   tr_ts_o,
  output logic [CW-1:0] tr_code_o
);
  localparam int AW = $clog2(DEPTH);
  logic [15:0]   ts;                                 // free-running timestamp
  logic [15:0]   ts_mem [DEPTH];
  logic [CW-1:0] code_mem [DEPTH];
  logic [AW-1:0] wptr;                               // next slot to write = oldest entry
  logic          armed;
  logic [$clog2(POST+1)-1:0] post_left;

  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin
      ts <= '0; sticky_o <= '0; first_err_o <= '0; first_ts_o <= '0;
      wptr <= '0; armed <= 1'b0; frozen_o <= 1'b0; post_left <= '0;
    end else begin
      ts <= ts + 16'd1;
      // sticky bits: a new error wins over a clear in the same cycle
      sticky_o <= (sticky_o & ~w1c_i) | err_i;
      if (|err_i && sticky_o == 4'b0000) begin
        first_err_o <= err_i;
        first_ts_o  <= ts;
      end
      // trace ring with freeze
      if (ev_valid_i && !frozen_o) begin
        ts_mem[wptr]   <= ts;
        code_mem[wptr] <= ev_code_i;
        wptr <= wptr + 1'b1;
        if (armed) begin
          post_left <= post_left - 1'b1;
          if (post_left == 1) frozen_o <= 1'b1;
        end
      end
      if (|err_i && !armed && !frozen_o) begin
        armed <= 1'b1;
        post_left <= POST[$clog2(POST+1)-1:0];
      end
      if (|w1c_i && (sticky_o & ~w1c_i) == 4'b0000 && err_i == 4'b0000) begin
        armed <= 1'b0; frozen_o <= 1'b0;             // all clear: re-arm
      end
    end

  assign tr_ts_o   = ts_mem[AW'(wptr + tr_idx_i)];
  assign tr_code_o = code_mem[AW'(wptr + tr_idx_i)];
endmodule
// tb_dbg_monitor: 20 events (an idle cycle after every third), an error from source 2 with event 12, a later error from source 0.
// Checks the sticky bits, the first-error record, the frozen trace and the re-arm after W1C.
`timescale 1ns/1ps

$ iverilog -g2012 -o t.vvp dbg_monitor_tb.sv && vvp -n t.vvp
sticky=0101 first_err=0100 first_ts=16 frozen=1
  trace[0] ts=9 code=7
  trace[1] ts=10 code=8
  trace[2] ts=12 code=9
  trace[3] ts=13 code=10
  trace[4] ts=14 code=11
  trace[5] ts=16 code=12
  trace[6] ts=17 code=13
  trace[7] ts=18 code=14
PASS: first error and the 8 events around it kept; later error did not overwrite them
dbg_monitor_tb.sv:121: $finish called at 286000 (1ps)

Read it as the lab would after a failure. sticky=0101: sources 2 and 0 both failed. first_err=0100 at timestamp 16: source 2 came first, and the record survived source 0's later error. The trace holds events 7 to 14 in order: five before the error, event 12 that came with it at timestamp 16, and two after. The gaps in the timestamps (10 to 12, 14 to 16) are the idle cycles, which is why a trace needs timestamps and not just an order. Events 15 to 19 are missing on purpose: they came after the freeze. Verilator 5.048 prints the same lines and PASS.

A bench must be able to fail. Remove the freeze:

$ sed 's/if (ev_valid_i && !frozen_o) begin/if (ev_valid_i) begin/' dbg_monitor_tb.sv > mutant.sv
$ iverilog -g2012 -o m.vvp mutant.sv && vvp -n m.vvp
...
  trace[7] ts=25 code=3
FAIL: trace[7] should be event 14
FATAL: mutant.sv:119: 8 check(s) failed
...

Without the freeze the ring keeps recording, and the events around the failure are gone by the time anyone reads it: all eight trace checks fail.

What it costs

$ yosys -q -p "read_verilog -sv dbg_monitor.sv; chparam -set DEPTH 8 dbg_monitor; synth -top dbg_monitor; tee -o stat8.txt stat"
Warning: Replacing memory \code_mem with list of registers. See dbg_monitor.sv:46
Warning: Replacing memory \ts_mem with list of registers. See dbg_monitor.sv:45
$ yosys -q -p "read_verilog -sv dbg_monitor.sv; chparam -set DEPTH 32 dbg_monitor; synth -top dbg_monitor; tee -o stat32.txt stat"
Warning: Replacing memory \code_mem with list of registers. See dbg_monitor.sv:46
Warning: Replacing memory \ts_mem with list of registers. See dbg_monitor.sv:45
$ grep -E "cells$|DFF" stat8.txt
      614 cells
       27   $_DFFE_PN0P_
      160   $_DFFE_PP_
       20   $_DFF_PN0_
$ grep -E "cells$|DFF" stat32.txt
     2166 cells
       29   $_DFFE_PN0P_
      640   $_DFFE_PP_
       20   $_DFF_PN0_

Eight entries of 20 bits cost 207 flip-flops and 614 generic cells; 32 entries cost 689 and 2,166. The 160 and 640 enable flops are the trace storage itself (8 or 32 entries times 20 bits); the rest, 47 flops at depth 8 and 49 at depth 32, is the sticky register, the first-error record, the timestamp and the pointers, and it barely grows. So the decision that matters is depth and width: how much history, and how many bits per event. Yosys's warnings say the same thing: with no RAM to map to, it turned the trace memory into registers. A deep trace usually goes into a small RAM instead.

The verification cost is as real: thirteen checks and a mutation run for this block, and every shipped feature also needs a test that reads it through the lab's side-band path.

What the evidence says

A few hundred flops buy an answer to "what happened first, and what led up to it", which is the question every silicon hang starts with. The cost scales with trace depth, so spend it where the questions are hardest: interfaces between blocks, arbiters, anything with a FIFO that can fill. For dbglab, the review would add three things: grant, lock and DMA state readable on a side-band port; a hang detector on each master port; and a short trace of bus handshakes that freezes on the detector.

A review checklist

Traps

Related topics

• Part 2 · Observe

• Bring-up: debug interfaces (trace, JTAG, scan dump)

• Counters and performance probes

• Case study: the deadlock under back-pressure

• Part 11: debug as a team