Reports and metrics

Reports & Metrics for Compliance & Coverage Plan focuses on spec feature coverage, compliance pass rate, waiver count. The goal is to connect the observable symptom to protocol mechanism, ownership, and regression risk.

The job of a report is to turn spec feature coverage, compliance pass rate, waiver count into a decision. A single average number is almost never enough; you need the distribution, the traffic class breakdown, and a clear gap between legal maximum and product target.

Metric movement

METRIC GRAPH — spec feature coverage, compliance pass rate, waiver count

throughput / success
  ^
  |                         target
  |                       - - - - - - -
  |                  o after bounded fix
  |              o
  |         o baseline
  |    o failing run
  +--------------------------------------> experiment
    config A     isolated root cause     accepted change

Readout:
  - compare identical payload, clock, reset, traffic seed, and firmware setup
  - separate headline bandwidth from useful payload bandwidth
  - explain why the protocol mechanism moved the metric

Latency distribution

LATENCY HISTOGRAM — Compliance & Coverage Plan

count
  |               ███
  |             ███████
  |          █████████████
  |        █████████████████        <- long tail = the real complaint
  |      ████████████████████████████
  +------------------------------------> latency
   p50      p90    p95       p99  (watch p99, not the average)

Average hides the tail; product pain lives at p95/p99.

• Track spec feature coverage, compliance pass rate, waiver count by traffic class, payload size, and clock/reset mode.

• Report p50/p95/p99 latency when user-visible stalls matter.

• Include legal maximums and product targets; they are not the same thing.

• Always store the metric next to the artifact that produced it.

Protocol deep dive

Verification closes the gap between 'works in directed test' and 'legal under all stressed traffic'.

Concept diagram

VERIFICATION CLOSURE LOOP

spec clause -> test -> assertion -> coverage -> waiver -> signoff
                  ^                           |
                  +--------- gap found --------+

Metric graph

COVERAGE vs ESCAPE RATE

escapes
  |*
  | *
  |  *
  |   **  <- knee: more random helps
  |     ****
  +----------------> constrained-random depth

Metrics and artifacts to collect

• compliance pass rate

• coverage closure

• scoreboard mismatch rate

• assertion fire count

• waiver log

Mini case study

Block passed VIP compliance but chip failed: system test omitted cross-master ID reuse through a bridge. Scoreboard key did not include upstream port ID.

Debug branches

• If VIP pass but silicon fail, check integration assumptions.

• If scoreboard noisy, fix key (ID+port+addr).

• If coverage plateau, add compliance gap analysis not more repeats.

Senior review question

Ask: what is the first transaction that deviates, and which spec rule does it test?

Key takeaways

• Connect every protocol claim to a transaction identity and measurable metric.

• Store the artifact (waveform, log, counter) next to every signoff decision.

Common pitfalls

• Debugging timeouts without finding the first bad transaction.

• Quoting peak bus width without payload efficiency and retry overhead.

• Treating VIP compliance as a substitute for system integration replay.

How to read the numbers

spec feature coverage, compliance pass rate, waiver count must be split by traffic class, payload size, and reset mode.