Regression Matrix: Expanding Tests, Seeds, and Env Modes

Matrix dimensions for env-test regressions

A practical matrix sweeps test class , random seed , and env mode knobs — all via plusargs on one compiled image.

[TEST][UVM][ENV] matrix axes

axis A: +UVM_TESTNAME     (scenario selection)
axis B: seed option        (random stability / coverage)
axis C: +MODE= / +SEQ_COUNT (env behavior tuning)
axis D: +UVM_VERBOSITY     (debug vs throughput)

cell = unique tuple replayable from logs

#!/bin/bash
# nightly_matrix.sh — expand testlist x seeds
TESTS=(smoke_sanity_test burst_traffic_test error_injection_test)
SEEDS=(1 42 9001)

for t in "${TESTS[@]}"; do
  for s in "${SEEDS[@]}"; do
    echo "RUN test=$t seed=$s"
    ./simv +UVM_TESTNAME=$t +ntb_random_seed=$s +UVM_VERBOSITY=UVM_LOW \
      || { echo "FAIL test=$t seed=$s"; exit 1; }
  done
done

Key takeaways

• Matrix expansion belongs in scripts/CI — not compile-time forks.

• Every cell should log test name, seed, and custom plusargs.

• Smoke matrices stay small; nightly adds breadth; weekly adds depth.

Common pitfalls

• Omitting seed from failure bundles — unreproducible random failures.

• Expanding MODE knobs without validating parser support.

• Running full matrix at UVM_HIGH — log volume hides real failures.

Structured matrix files and reporting

For larger teams, represent matrix rows as data (JSON/CSV) consumed by a launcher that builds the plusarg string.

JSON row launcher

[TEST] matrix row example

{
  "test": "burst_traffic_test",
  "seed": 42,
  "plusargs": ["+SEQ_COUNT=2000", "+MODE=stress", "+UVM_VERBOSITY=UVM_LOW"]
}

#!/bin/bash
# run_row.sh — args: test seed "plusarg string"
TEST=$1; SEED=$2; EXTRA=$3
./simv +UVM_TESTNAME=$TEST +ntb_random_seed=$SEED $EXTRA

CI stage mapping

• PR gate: 1–3 smoke tests, fixed seeds, UVM_LOW.

• Nightly: full testlist x 3 seeds, default knobs.

• Weekly: stress modes (+MODE=strict, high SEQ_COUNT) on subset.

[UVM][TEST] dashboard fields per run

test_name | seed | plusarg_hash | pass_fail | sim_time | uvm_errors

Common pitfalls

• Launcher scripts that drop custom plusargs on retry.

• Non-deterministic ordering when parallel workers share one log dir.

• Matrix growth without pruning obsolete tests or flags.