A vendor presentation promises that machine learning will "cut regression time by 10x". Your manager forwards it with one line: is this real, and what would we need to try it? You know the regression: 2,000 runs a night, a manifest per run, signatures from the triage script, a coverage database per run. You do not know what the tool does with them, and the slides do not say. What you need is a way to tell what ML can genuinely do with regression data, what it cannot, and how to test a claim on your own runs before anyone signs a purchase order.

The idea: it is statistics on the data you already record

Strip away the branding and most "ML in verification" is applied statistics on regression history: which tests fail, how often, after which changes, at what cost, and which coverage each run adds. Machine learning here means fitting a model to past runs so it can predict something about future ones: which test is likely to fail, which failures belong together, which stimulus settings reach a coverage hole. The model can only learn what the data shows. If your runs do not record their seed, coverage and commit, no tool can learn from them.

Think of a weather forecast. The forecasting model matters, but its quality is set mainly by years of logged observations from many stations. A team that keeps a manifest line, a signature and a coverage record for every run is building its observation network. A team that keeps only today's pass rate has nothing for any model to learn from, however clever.

What is real today

Every row needs data your regression either records or does not. The regression infrastructure lesson's manifest (verdict, test, seed, log) and the triage signatures are the minimum. Add per-run coverage, runtime, build or commit ID and the bug ticket each signature was filed under, and you have the inputs for all four.

What is hype

• "Replaces verification engineers." No tool decides what the design is supposed to do. Test plans, checkers, reference models and waivers encode intent, and that is the job.

• "Closes coverage automatically." Steering can raise coverage of holes the bench can reach. It cannot reach a hole the stimulus cannot generate (the DMA's gap=0: the stream driver always drops TVALID between beats), and coverage is only as good as the model that defines it.

• "Finds all the bugs" or "guarantees quality." A statistical model gives probabilities, not proofs. Only formal gives proofs, and only for the properties written.

• Language models that write complete, correct testbenches. They are useful for drafts of sequences, properties and scripts. Every line still needs a reviewer who knows the protocol; an assertion that compiles is not an assertion that is right.

• Any speed-up quoted without a baseline. "10x" against running everything, in random order, is a claim about the old process, not about the tool.

The picture: from regression data to decisions

Build it: rank tests by what they add

The simplest useful automation needs no ML at all: rank runs by the coverage each one adds. Run the regression, read each run's coverage, then pick greedily: the run that adds the most new plan items, then the run that adds the most of what is left, and so on. This is the classic greedy set cover; finding the smallest covering set is NP-hard, but greedy is provably within a logarithmic factor of optimal and usually much better in practice.

The course bench prints its plan-item tally in every run that uses dma_env_check ("plan items hit: N of 17" and the list of holes), so the logs already hold the data. The script below reads them; cost is the simulated time of the run, a stand-in for the CPU seconds a farm would record.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

#!/usr/bin/env python3
"""Rank regression runs by the plan items each one adds.  Usage: rank_tests.py <test>.<seed>.log ..."""
import re, sys, os

PLAN = ["len=0", "len=1", "len=2-4", "len=5-15", "len=16", "len>16",
        "irq_en=0", "irq_en=1", "outcome=done", "outcome=err",
        "clear=done", "clear=err", "gap=0", "gap=1-3", "gap=4+",
        "tlast=on-time", "tlast=early"]

runs = {}                                   # run name -> (items hit, cost in ns)
for path in sys.argv[1:]:
    text = open(path).read()
    if "UVM_FATAL :    0" not in text or "UVM_ERROR :    0" not in text:
        continue                            # coverage from failing runs does not count
    m = re.search(r"@ (\d+): \S+ \[COV\] plan items hit", text)
    if not m:
        continue                            # no tally: this test has no coverage collector
    holes = re.search(r"\[COV\] plan holes:(.*)", text)
    missed = set(holes.group(1).split()) if holes else set()
    runs[os.path.basename(path)[:-4]] = (set(PLAN) - missed, int(m.group(1)) // 1000)

covered, order = set(), []
while runs:
    # most new items first; among equals, the cheaper run
    name, (items, cost) = max(runs.items(), key=lambda r: (len(r[1][0] - covered), -r[1][1]))
    new = items - covered
    if not new:
        break
    covered |= new
    order.append((name, len(new), cost, len(covered)))
    del runs[name]

print(f"{'rank':<5}{'run':<26}{'new':>4}{'sim ns':>8}{'total':>7}")
for i, (name, new, cost, total) in enumerate(order, 1):
    print(f"{i:<5}{name:<26}{new:>4}{cost:>8}{total:>4}/{len(PLAN)}")
print(f"adds nothing new: {len(runs)} run(s): {' '.join(sorted(runs))}")
print("never hit:", " ".join(p for p in PLAN if p not in covered) or "-")

The data set is a scratch regression of 16 runs from one build: dma_check_test with seeds 1 to 8, dma_cmp_trap_test and dma_zero_len_test, plus two scratch tests written for this lesson. dma_check_trunc_test is dma_truncate_test with the coverage env, and dma_check_long_test is dma_check_test with job lengths of 12 to 16 (seeds 1 to 3 each). Every run passed.

$ python3 rank_tests.py cov/*.log
rank run                        new  sim ns  total
1    dma_check_test.8             8    1185   8/17
2    dma_check_trunc_test.1       3     455  11/17
3    dma_zero_len_test.1          1     325  12/17
4    dma_check_test.4             1    1595  13/17
5    dma_check_test.2             1    2445  14/17
adds nothing new: 11 run(s): dma_check_long_test.1 dma_check_long_test.2 dma_check_long_test.3 dma_check_test.1 dma_check_test.3 dma_check_test.5 dma_check_test.6 dma_check_test.7 dma_check_trunc_test.2 dma_check_trunc_test.3 dma_cmp_trap_test.1
never hit: len>16 irq_en=0 gap=0

• Five runs reach everything the sixteen reach. Fourteen of 17 plan items, from 5 runs costing about 6 us of simulated time out of roughly 24 us for the whole set.

• Eight seeds of the same test are mostly the same test. Five of the eight dma_check_test seeds add nothing, and two more are kept for one plan item each. Random lengths 1 to 16 keep landing in the same few bins.

• The truncation test is cheap and unique. At 455 ns it adds three items no other test reaches (outcome=err, clear=err, tlast=early).

• The never-hit list is a plan, not a ranking problem. len>16 needs a longer packet sequence (axis_pkt_seq caps a packet at 16 words), irq_en=0 needs a polling flow, and gap=0 needs a driver change. No ranking or ML over these runs can produce what none of them generate.

What breaks: coverage-redundant is not check-redundant

The ranking says eleven runs add nothing. It is tempting to delete them from the nightly. Before you do, look at the fault campaign from functional safety: a stuck-at-0 fault on KEY bit 0 was detected by dma_rand_test and missed by dma_truncate_test, only because the truncation test's random KEY happened to have bit 0 = 0. No plan item mentions KEY bit values. A ranking built on this coverage model would happily drop the runs whose random keys happened to exercise that bit, and nothing in the coverage report would change.

A reduced list is a claim that the dropped runs find nothing the kept ones do not. Coverage supports that claim only as far as the coverage model reflects real bug classes. Three habits keep the shortcut honest: keep the full list on a slower schedule (weekly, or before each milestone); validate every reduced list with a fault or mutation campaign, which measures detection instead of coverage; and extend the coverage model when a campaign finds a gap (here, toggle coverage on KEY and data bits, or a cross of key bits against data bits).

How to evaluate a tool claim

Next: with formal, assertion VIP and regression data in your toolkit, the senior section puts them to work on whole projects, starting with Verification Architecture.

Related topics

• Regression Triage by Signature

• Regression Infrastructure and CI

• Coverage Merge and Closure

• Functional Safety (fault campaign)