Part 10 · Advanced Topics · Intermediate

Regression Matrix: Scaling Runtime Knobs Systematically

Designing, pruning, and automating regression matrices over tests, seeds, modes, and command-line knobs.

Matrix dimensions and explosion control

CLI configurability makes matrix testing possible, but naive Cartesian products explode quickly. Define dimensions intentionally and classify each as required, sampled, or conditional.

diagram

[REG] common matrix dimensions

  D1 tests             smoke, protocol, stress
  D2 seeds             fixed replay set + random sample
  D3 mode              compat / strict
  D4 verbosity profile low / targeted debug
  D5 custom knobs      NUM_TXN tiers, BURST_MAX tiers
  D6 simulator         vendor/version lanes

  Full product can be huge -> prioritize risk-weighted subsets

diagram

[ADV] risk-weighted selection

  High-risk combinations     : always include
  Medium-risk combinations   : rotating schedule
  Low-risk combinations      : periodic audit only

Tie matrix size to CI budget and expected defect detection yield.
Use fixed seeds for signature stability in required lanes.
Rotate random samples to expand state-space coverage over time.

Cell definition and naming conventions

A matrix cell should be uniquely identifiable from its name. Good naming allows quick search, dashboard grouping, and replay command generation.

diagram

[REG] cell naming pattern

  <sim>-<mode>-<test>-seed<seed>-txn<num>-burst<max>

  Example:
    simA-strict-axi_stress-seed12345-txn1000-burst32

diagram

[ADV] matrix manifest example

  cell_id,sim,mode,test,seed,num_txn,burst_max,verbosity
  c001,simA,strict,smoke,101,100,16,UVM_LOW
  c002,simA,strict,stress,202,1000,32,UVM_LOW
  c003,simB,compat,smoke,101,100,16,UVM_LOW
  c004,simB,strict,stress,202,1000,32,UVM_MEDIUM

Store manifest in version control for auditability.
Avoid hidden matrix generation logic inside shell scripts only.
Generate replay commands directly from manifest rows.

Execution orchestration patterns

Matrix execution should separate scheduling from simulation command construction. This keeps infra independent from UVM-specific argument details.

bash

# Pseudo runner for one cell row
run_cell() {
  local sim=$1 mode=$2 test=$3 seed=$4 num_txn=$5 burst_max=$6 verb=$7
  make sim SIM="$sim" MODE="$mode" \
    TEST="$test" SEED="$seed" \
    EXTRA_ARGS="+NUM_TXN=$num_txn +BURST_MAX=$burst_max +UVM_VERBOSITY=$verb"
}

diagram

[REG] orchestrator responsibilities

  Scheduler:
    - picks cell set for this run window
    - handles retry/backoff policy
    - captures artifacts and metadata

  Command builder:
    - converts cell row -> simulator invocation
    - applies mode-specific flags
    - injects UVM and custom plusargs

Keep mode-specific simulator flags in one mapping layer.
Emit full command line to artifacts for replay and audits.
Tag failures with matrix cell ID for immediate traceability.

Result interpretation and matrix evolution

Matrixes should evolve based on failure yield and coverage gaps. Keep what finds bugs; retire what never contributes signal.

diagram

[ADV] matrix health metrics

  Metric                               Why it matters
  -------------------------------------------------------------------------
  Unique failure signatures per 100 cells  defect yield efficiency
  Replay success rate                      reproducibility quality
  Strict vs compat delta                   migration debt visibility
  Runtime per detected issue               CI cost effectiveness
  Coverage gain per matrix slice           closure contribution

diagram

[REG] weekly matrix review questions

  1) Which cells produced unique high-value failures?
  2) Which cells are redundant (same signatures repeatedly)?
  3) Are strict-mode failures shrinking or flat?
  4) Should any random lanes be promoted to required fixed-seed lanes?
  5) Which knobs need additional value tiers?

Key takeaways

Matrix design is an optimization problem, not full Cartesian brute force.
Manifest-driven cell definitions improve replay and maintainability.
Review matrix yield regularly to keep CI cost aligned with bug detection value.

Common pitfalls

Running oversized matrices with low unique-failure yield.
Hiding matrix generation logic in unversioned scripts.
Ignoring strict/compat delta trends during migration phases.

Practice this lessonQuestions tagged for this topic in the bank