Scoreboard & Checker Strategy: Reports & Metrics

Reports and metrics

Reports & Metrics for Scoreboard & Checker Strategy focuses on false fail rate, escaped bug count, response match latency. The goal is to connect the observable symptom to protocol mechanism, ownership, and regression risk.

The job of a report is to turn false fail rate, escaped bug count, response match latency into a decision. A single average number is almost never enough; you need the distribution, the traffic class breakdown, and a clear gap between legal maximum and product target.

Metric movement

METRIC GRAPH — false fail rate, escaped bug count, response match latency

throughput / success
  ^
  |                         target
  |                       - - - - - - -
  |                  o after bounded fix
  |              o
  |         o baseline
  |    o failing run
  +--------------------------------------> experiment
    config A     isolated root cause     accepted change

Readout:
  - compare identical payload, clock, reset, traffic seed, and firmware setup
  - separate headline bandwidth from useful payload bandwidth
  - explain why the protocol mechanism moved the metric

Latency distribution

LATENCY HISTOGRAM — Scoreboard & Checker Strategy

count
  |               ███
  |             ███████
  |          █████████████
  |        █████████████████        <- long tail = the real complaint
  |      ████████████████████████████
  +------------------------------------> latency
   p50      p90    p95       p99  (watch p99, not the average)

Average hides the tail; product pain lives at p95/p99.

• Track false fail rate, escaped bug count, response match latency by traffic class, payload size, and clock/reset mode.

• Report p50/p95/p99 latency when user-visible stalls matter.

• Include legal maximums and product targets; they are not the same thing.

• Always store the metric next to the artifact that produced it.

Protocol deep dive

Verification closes the gap between 'works in directed test' and 'legal under all stressed traffic'.

Concept diagram

VERIFICATION CLOSURE LOOP

spec clause -> test -> assertion -> coverage -> waiver -> signoff
                  ^                           |
                  +--------- gap found --------+

Metric graph

COVERAGE vs ESCAPE RATE

escapes
  |*
  | *
  |  *
  |   **  <- knee: more random helps
  |     ****
  +----------------> constrained-random depth

Metrics and artifacts to collect

• compliance pass rate

• coverage closure

• scoreboard mismatch rate

• assertion fire count

• waiver log

Mini case study

Block passed VIP compliance but chip failed: system test omitted cross-master ID reuse through a bridge. Scoreboard key did not include upstream port ID.

Debug branches

• If VIP pass but silicon fail, check integration assumptions.

• If scoreboard noisy, fix key (ID+port+addr).

• If coverage plateau, add compliance gap analysis not more repeats.

Senior review question

Ask: what is the first transaction that deviates, and which spec rule does it test?

Key takeaways

• Connect every protocol claim to a transaction identity and measurable metric.

• Store the artifact (waveform, log, counter) next to every signoff decision.

Common pitfalls

• Debugging timeouts without finding the first bad transaction.

• Quoting peak bus width without payload efficiency and retry overhead.

• Treating VIP compliance as a substitute for system integration replay.

How to read the numbers

false fail rate, escaped bug count, response match latency must be split by traffic class, payload size, and reset mode.